Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
Authors: Xiong Wang
Yangze Li
Chaoyou Fu
Yike Zhang
Yunhang Shen
Lei Xie
Ke Li
Xing Sun
Long MA
Venue: ICML 2025 poster (2025)
Keywords: Speech to Speech
Duplex Dialogue Model
Multimodal Large Language Models
Primary area: applications->language_speech_and_dialog
TL;DR: A duplex speech-to-speech dialogue system based on a frozen LLM.

Abstract: The GPT-4o's excellent duplex speech interaction ability has given users an impressive experience. Researchers have recently proposed several multimodal LLMs to achieve user-agent speech-to-speech conversations. In this paper, we propose a novel speech-text multimodal LLM architecture called Freeze-Omni, and our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We effectively ensure that the intelligence of the Freeze-Omni in the speech modality is at the same level as that in the text modality of its backbone LLM while achieving low latency in the end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multitask training, giving Freeze-Omni a more natural style of dialogue ability between users and agents. In summary, Freeze-Omni holds great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.

Omnize-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
Xiong Wang" 1
Yangze Li 12
Chaoyou Fu²
Yike Zhang' Long Ma
Yunhang Shen Lei Xie? Ke Li Xing Sun
Abstract
users. The traditional method adopts cascaded approach of ASR + LLM + TTS to achieve interaction with LLM in the speech modality. However, this segnnerrin-centeree pipeline approach often leads to considerable interaction latency. Nevertheless, GPT-4o (OpenAI, 2024) has changed this situation it provides end-to-end speech interaction mode which has significantly improved user experience, triggering a research boom regarding multimodal LLMs for speech-to speechh interaction.
The GPT-4o's 's excellent duplex speech interaction ability has given users an impressive experience. Researchers have recently proposed several mul- timodal LLMs achieve user-agent speech-to- speech conversations. In this paper, we propose novel speech-text multimodal LLM architec- ture called Freeze-Omni, and our main contribu- tion is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We effectively ensure that the intelligence of the Freeze-Omi in the speech modality is at the same level as that in the text modality of its backbone I.I while achieving low latency in the end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multitask train- ing, giving Freeze-Omnt a more natural style of dialogue ability between users and agents, In summary, Freeze-Omni holds great potential to conduct speech-to-speed, dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting prob- lem caused by limited data and training resources.
In the field of general LLMs, many public models such as Llama 3.2 (Dubey et al., 2024), Qwen2.5 (Team, 2024), Mixtral (Jiang et al., 2024). etc., have provided good oppor- tunities for researchers to develop downstream tasks. There- fore, in the field of multimodal LLMs targeting speech-to- speech conversation, works such as Mini-Omni2 (Xie & Wu, 2024b), LLaMA-Omni (Fang et al., 2024), Moshi (Défossez et al., 2024) and GLM-4-Voice (Zeng et al., 2024) have provided excellent references for researchers. These works adopt different strategies to align the speech modality with an LLM and design some tricks to achieve a duplex dialogue mode.
In this research context, we found that in the process of align- ing the LLM with the speech modality in existing public speech-text multimodal LLMs (Chu et al., 2024; Défossez et al., 2024; Fang et al., 2024; Fu et al., 2024; Zhang et al., 2023; Xie & Wu, 2024a; Zeng et al., 2024), the param- eters of the LLM
more less fine-tuned. However, in most cases, it is challenging for researchers to easily collect spoken Q&A data at the million-hout level (the cor- responding text content can be comparable to the amount of data for training text-modal LLM). Fine-tuning the LLM inevitably brings the catastrophic forgetting problem to the LLM, negatively impacting its original "intelligence" ability, In addition, only few works have evaluated the perfor- mance of spoken question-answerinn tasks for speech-to- speech multimodal LLMs, showing an obvious gap in the performance between spoken Q&A and text-modality Q&A. Therefore, in this paper, we propose a speech-to-speed dialogue LLM called Freeze-Omni, which achieves effec- tive speech-text modality alignment while keeping the LLM parameters frozen, obtaining low-latency speech dialogue
1. Introduction
Recent years have witnessed rapid development of large language models (L..Ms). The LLM family represented by the GPT series (Floridi & Chiriatti, 2020; Achiam et al., 2023) of OpenAl has demonstrated extraordinary capabili- ties. As speech interaction is one of the most natural forms of human-compute interaction, combining speech input and output with an LLM can bring natural experience to
Equal contribution 'Tencent, China PASLP@NPP China e "aanjnng University, China. Correspondence to: Long Ma <mal- memaattencent.omm
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
Streaming Speech Output
Streaming Codec Decoder
Speech Token FIFO
kv-cache
NAR Prefix Speech Decoder
NAR Speech Decoder
AR Speech Decoder
Chunk Segment
Text Tokens
Hidden States
Large Language Model
Frame-wise
Features
Adapter
Freeze
Chunk-wise
Features
Speech Encoder
Streaming
Speech Input
Figure 1. Overview of proposed Frecze-Omni The streaming specch input forms chunk-wise features through the speech encoder, and then is connected to the LLM through the adapter. The LLM generates hidden states and text tokens. which are sent to the NAR prefix speech decoder and NAR speech decoder the decoder sends the generated tokens into the speech token FIFO, and the streaming codec decoder generates streaming speech output from FIFO according to fixed speech token chunk size.
chunks. respectively, after chunk segmentation. Finally, the speech
capabilities while maintaining the original intelligence of the backbone LLM. Specifically, Freeze-Ooe Omzz is mainly implemented in the following steps:
small amount Q&A data.
Modeling of speech output Second, we use sizable amount of text-speech paired data train the AR-based speech decoder, which can generate speech tokens from text, and single-codebook based codec model is used to decode the speech token into waveform. Then, we design prefix kv-cache fine-tune strategy, using the hidden state vector output by the LLM to transfer the speech decoder into the output text space of LLM, achieving the ability of
Modeling of speech input We first use large amount of ASR data align the speech encoder and the LLM, enabling the L.M to understand the semantic information from the speech. Then, with the LLM frozen, training strategy of prompt embedding is used to let the model have the ability to possess speech input to text output, training on only a
2

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
text input to speech output while keeping the LLM frozen.
representation into the embedding space of the backbone LLM. The speech encoder module here consists of sev- eral down-sampling convolution layers and several Trans- former (Vaswani et al., 2017) blocks, while the adapter only comprises several down-sampling convolution layers. The reason for using down-sampling is to reduce the frame rate of the speech features, increase the speed of the LLM the prefill stage, and decrease the latency.
Design for duplex dialogue Finally, we connect the speech encoder and speech decoder from the above parts to the backbone LLM. Then, task of chunk-wise state prediction is used to determine whether or not the user inter- rupts the dialogue, achieving the duplex speech-to-speech dialogue ability.
In conclusion, the main contributions of Freeze-Omní are as follows:
2.2.2. TRAINING STRATEGY
1) The parameters of the LLM are completely frozen throughout the training process of Freeze-Omni, maintain- ing the original intelligence of the LLM and achieving low- latency speech-to speech-to-speed to-speech dialogue at the same time.
A 3-stage training strategy shown in Fig. 2 is used for the speech encoder, allowing Freeze- -oizi to acquire the ability to understand the streaming input speech while keeping the LLM frozen.
2) The paired text-speech Q&A training data small scale and consumes fewer computing resources in the build- ing of Freeze-Omni.
1) The first stage shown in Fig. 2(a) is the same as the training process of common speech recognition model. The input is speech features, and the label is the transcript corresponding to the speech. CTC (Graves et al., 2006) is used as the loss function.
3) Freeze-Omni can support any (multimodal) LLM with a text modality and retains the abilities of the LLM, such as prompt following and role-playing. Moreover, if it is nec- essary change the style of the LLM's response, it is only required fine-tune it with text data in the corresponding style.
2) In the second stage shown in Fig. 2(b), we use the speech encoder trained in the first stage as the initialization param- eter and connect it with the LLM utilizing an adapter. The output of the LLM still uses the transcript corresponding to the input speech as the label. Several trainable special tokens are added to the input part to guide the LLM in com- pleting the training process at this stage. In this stage, except for the frozen LLM,, the parameters of other networks are trainable.
2. Model
2.1. Overview
Freeze-Omni is a speech-to-speech dialogue model, and its architecture is shown in Fig. I, exhibiting the characteris- tic of being "smart" as it constructed upon "frozen" text-modality LLM. This enables it to keep the original in- telligence of the LLM backbone, without being affected by the forgetting problem induced by the fine-tuning process for integration of the speech modality. Specifically, Freeze- Omni contains a speech encoder that supports streaming speech input and a speech decoder that generates streaming output speech. During the training process, Freeze- Omin first achieves alignment between speech input and text out- put, then between text input and speech output. Finally, by connecting these two components to the LLM, the ability of speech input to speech output is obtained. This section will provide a detailed introduction to the architecture, training strategy, and duplex dialogue design Freeze-Omni.
3) In the last stage shown in Fig. 2(c), we first construct a dataset of multi-round questions and use the LLM backbone relied on in the training to generate multi-roum answers. The dataset constructed in this way will be completely com- patible with the LLM backbone. Subsequently, we use a multi-speaker TTS system to generate data in the speech modality for the questions part and add trainable prompt embedding before each question in the multi-round to guide the LLM to achieve the ability of speech input to text output. In this stage, the trainable special tokens in stage 2 will be dropped, only the prompt embedding part is trainable and they use the same parameters for each question, the speech encoder is frozen to maintain the acoustic robustness obtained from stage 2, and the LLM is also frozen to ensure that its intelligence is not affected.
2.2. Modeling of speech input
2.3. Modeling of speech output
2.2.1. CHUNK-WISE STREAMING SPEECH ENCODER
1. ARCHITECTURE
To allow Freeze-Omni to support speech input and achieve a rapid and latency response to input speech, it utilizes a chunk-wise streaming speech encoder to transform the input speech features into nigh-dimensional representa- tion. Then, an adapter module maps the high-dimensional
Inspired by VALL-E (Chen et al., 2024), Freeze-Omni uses token-based speech decoder that contains NAR prefill and AR generation stage achieve speech output capabilities. The speech decoder mainly consists of the NAR decoder, the AR decoder, and the decoder of codec model. Both the
3

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
Transcript
LLM
Transcript
Adapter
Special
Speech Encoder
Speech Encoder
(a)
(b)
Text of Al
Text of A2
LLM
Adapter
Adapter
Prompt Embedding
Prompt Embedding
Speech Encoder
Speech Encoder
Q1
Q2
Wav
Figure 2.
The 3-stage training method for modeling of speech input, the speech encoder in (c) used in Freeze-Omni finally and LLM frozen in all 3-stage.
NAR decoder and AR decoder are built upon transformer blocks. The NAR decoder is used to model the semantic features from the output of LLM, and then the AR decoder generates speech tokens based on the output of the NAR decoder. Finally, a decoder of the codec model converts the speech tokens into speech stream.
system as much possible.
2) In the second stage shown in Fig. 3(b), we first construct large amount of text-speech paired data and pass the text through the tokenizer of the backbone LLM to convert text into text tokens. Then, we pass the text tokens through the embedding layer of the LLM to convert them into bedding vectors semantic features and send them to the NAR speech decoder. The AR speech decoder predicts the output speech tokens in the form of teacher force. The labels here are extracted using the codec model trained in stage 1. The NAR and AR speech decoders use same parameters, and the embedding layer of the L.M frozen.
2.3.2. TRAINING STRATEGY
For the modeling of speech output, we still use a 3-stage training method as shown in Fig. 3, enabling Freeze-One Omei to obtain the ability of generate speech from the output of LLM while keeping the LLM frozen.
As shown in Fig. (a), we first train single-codebook based codec model using only speech data. Since single codebook is sufficient for extracting speech tokens from the speech signal of a limited number of speakers, using single codebook here can reduce the complexity and latency of the
3) In the last stage, and answers data set in stage of Sec. 2.2.2 and use the text tokens and hidden state sequence generated by the back- bone LLM. As shown in Fig. 3(c), an additional NAR prefix speech decoder is added to model the hidden state of the
the same multi-round questions

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
Speech
-|
Upsample Decoder
NAR Speech Decoder
AR Speech Decoder
Speech Token
Embedding layer
Subsample Encoder
Tokenizer of LLM
Input Text
(b)
Speech Token
kv-coche
NAR Prefix Speech Decoder
NAR Speech Decoder
AR Speech Decoder
Embedding Layer of LLM
Hidden State of LLM
Text Token
(c)
Figure 3. The 3-stage training method for modeling of speech output, the speech decoder (c) is finally used in Freeze-Omni, and LLM frozen in all three stages.
State Label
Predict State
LLM
Input Chunk
Chunk
Chunk
Chunk N
Figure 4. Method of chunk-level state prediction used in the prefill stage of the LLM. An additional classification layer is added to the output hidden state the LLM corresponding to the last frame each chunk output by the speech encoder to predict the state.
LLM and pass its output kv-cache to the NAR speech de- coder. Then the text token will be fed to the NAR speech decoder trained stage 2. The text token label for the AR speech decoder the speech data produced by the output text of the LLM using TTS system and converted into speech tokens by the codec model in stage 1. In this stage, the NAR prefix speech decoder uses different parameters from the NAR and AR speech decoders, and only the rameters of the NAR prefix speech decoder are trainable, while the parameters of other networks are frozen. Because the style of the text tokens produced by the LLM differ- ent from that of the text in the large amount of text-speech paired data obtainable in stage 2, the significance of the third stage lies in closely coupling the speech decoder with the output of the LLM to reduce the occurrence of bad cases.
2.4. Design for duplex dialogue
After the above training process, Freeze-Omni has the ability to convert speech input to speech output. However, to better approximate the natural form of speech-to-spectt dialogue, we use multi-task for chunk-level state prediction as shown in Fig 4. We first use acoustic VAD module detect the starting point of the streaming speech. When the VAD is triggered, the speech stream will be sent into Freeze-Omni chunk by chunk, and an additional classification layer will be added after the last layer of the LLM to predict different states. Three states are defined here: state indicates that the current LLM can continue receive speech, and states 1 and 2 indicate that the current chunk is the end of the speech.
com/ snakers4 wiieerrrs/sieeeeeeeeeeeeeeeeeeee
5

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
State means that the user will interrupt the dialogue and the LLM will perform new generation stage; State 2 means that there is no need to interrupt the conversation. Both states will stop sending speech streams to Freeze-Omni and reset the VAD module. The training process of this part is completed in stage 3 of Sec. 2.2.2, using a multi-task method to optimize the cross-entropy loss of both the state classification layer and the LLM. It should be noted that the state labels here are only valid on the last frame of each chunk.
our method. Besides, Freeze-Omni can use any LLM as backbone because its training process does not update any of the LLM's parameters.
Speech Encoder We used a multi-layer convolution with 4-times downsampling and 24 layers of transformers with a hidden size of 1024. The adapter consists of a multi- convolution layer with 2 times downsampling. The num- ber of parameters for the speech encoder is approximately 350M, with an output frame rate of 12.5 Hz. The input of the speech encoder is the mel-filter bank feature with a 25ms window size and 10ms shift.
In addition, we used a "model as server" strategy to im- plement the speech-to-speech dialogue system. First, we started several models simultaneously and regarded them as a server. Then, when a user's VAD was triggered, the speech would be sent to the server in the form of chunks, and the server would be responsible for scheduling which idle model should respond to the current chunk. Since we separated all the kv-cache and CNN cache of the speech encoder and LLM during the inference process, the server only needs to save the inference cache for each user. In this way, any model on the server could respond to any chunk of any user, and there was no need to specify which model was used as a monitor or a generator.
Speech Decoder We used TiCodec* (Ren et al., 2023) as the codec model, and we customized the configuration so that the size of the codebook 1024 with single-codebook and the frequency of the speech token 40Hz. For the speech decoder part, the NAR (Prefix) speech decoder and the AR speech decoder are 4-layer Llama decoder layers with a hidden size of 896. The number of parameters for the speech decoder is approximately 120M, and the output sample rate of the codec model is 24000Hz.
3.1.3. TRAINING
In the training process, used the Adamw (Loshchilov & Hutter, 2017) optimizer with a warm-up learning rate scheduler, and different learning rates were used in differ- ent stages. The learning rates used in the three stages of the modeling of speech input are 2e-4, le-4, and 6e-4 re- spectively. The learning rates used in stages and of the modeling of speech output are both 5e-5, and the training hyperparameters used in stage I are the same as those in TiCodec. All the experiments were completed on 8 GPUs.
3. Experiments
3.1. Setups
3. DATASETS
In this paper, we only randomly selected 60,000 multi-round Q&A data from mosss-00-sst-datd 2 and used the backbone LLM to generate new answers to replace its original one. We used a zero-shot TTS system to synthesize its text into speech. For the modeling of speech input of Freeze- Ommni we used 110,000h of internal speech-text paired ASR data, including both Chinese and English, in stage 1 and stage 2. In stage 3, we used the pairing of speech input and text output of the multi-round Q&A data mentioned above. For the modeling of the speech output of Freeze-Omnt we used about 3,000h of text-speech paired data generated by zero- shot TTS system in stages I and 2. In stage 3, we used the pairing of text input and speech output of the multi-round Q&& data mentioned above.
3.2. Results on speech input
To measure the understanding ability of Freeze-Omni for input speech, shown in Tab. 1, we verified the accuracy of ASR on different evaluation sets for the model in stage 2 of the modeling of speech input. Since the parameters of the speech encoder and adapter used in stage 3 are unchanged compared to those in stage 2, it can be considered that these results can represent the input speech understanding ability of Freeze- Omni. In the training of stage 2. we used a dy- namic chunk training method (Yao et al., 2021) to enhance the robustness of the model so that different chunk sizes can be used in stage 3. From the results, it can be seen that in the case of dynamic chunk training, decoding with chunk = do shows better performance compared to chunk = 4. If dy- namic chunk training not used but chunk = 4 decoding is used, better results can be obtained, but this also means that the chunk size cannot be changed in stage 3. In this paper, to pursue the best performance, all experiments are
3.1.2. MODEL CONFIGURATION
LLM backend For experiments in this paper, we used Qwen2-7B-Instruet² as our backbone LLM. As an outstand- ing 7B-level public LLM, it is beneficial for us to verify
moss-003-sft-data 'https:/hhugginngfacss ctt.co/////oo /
qwen2-7B-Instrrcct
Phttps://githa htwpchhbb.coom.m/. renll 6/ 6/TiCodec
6

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
Table The ASR performance model corresponding stage in the modeling of speech input, where (aishell-1 (Bu et al., 2017), test).t (Zhang al., 2022), test.meeting (Zhang et al., 2022)} are Mandarin evaluation sets, measured in CER (%), while ddev-clean.dev-other..e. wlt.rrttsttcccccttttttttttte (Panayotov et al., 2015) are English evaluation sets, measured in WER (%).
Model
aishell-1
test_net
test_meeting dev-clean dev-other
test-clean test-other
Wav2vec2-base (Baevski al., 2020)
6.0
13,4
Mini-Omni2 (Xic & Wu, 2024b)
4.8
9.8
4.7
9.4
VITA-1.5 (Fu et al., 2025)
2.16
8.4
10.0
3.3
7.2
3.4
7.5
Freeze-Omní
chunk = 00
2.15
8.57
10.09
3.29
7.4
3.24
7.68
chunk = 4
2.79
12.6
14.2
4.16
10.21
4.05
10.48
w/o dynamic
2.48
11.8
13.46
4.03
9.45
3.82
9.79
Table 2. The CER(%) of the speech decoder on 1,000 evaluation utterances under different top-k.
top-k
Method
2
3
4
Speech Decoder w/o Prefix
5.27
4.64
4.76
4.66
5.03
pre-network
3.11
2.75
2.77
2.84
2.94
Speech Decoder
3.9
3.65
3.53
3.62
3.71
pre-neet network
2.19
1.69
1.85
1.9
1.99
completed the model with this configuration of the last row in Tab. 1.
Therefore, to verify whether the generation quality would be improved if the NAR speech decoder had additional pa- rameters for modeling the outputs of the LLM embedding layer compared to the AR speech decoder, we added an extra pre-network between the NAR speech decoder and the LLM embedding layer. This pre-network consists of two Llama decoder layers with the same configuration as the NAR speech decoder. As shown in Tab 2, this method can significantly improve the speech quality generated by the speech decoder.
3.3. Results speech output
Because we investigated the speech-out performance of Freeze-Omni in single-speaker case in this paper, we ran- domly selected 1,000 utterances of text tokens and hidden states output by the LLM as the input of the speech de- coder and compared the ASR accuracy of the synthesized speech with the label text. As shown Tab the perfor- mance of the model in stage of the modeling of speech output (Speech Decoder w/o Prefix) and the model in stage 3 (Speech Decoder) under different AR decoding parameters top-k are presented respectively, and CER (%) is evaluated using paraformer-zht (Gao et al., 2022). From the results, it can be concluded that after introducing the hidden state of the LLM as the input of the NAR prefix speech decoder, the speech decoder can be more completely aligned with the LLM, reducing the occurrence of bad cases and get lower CER (%). In addition, the increasing top-k shows better robustness of the speech decoder with prefix fine-tune be- cause a larger top-k means a higher quality requirement on the posterior probability distribution output by the model.
3.4. Results on spoken question answering
To demonstrate the intelligence of Freeze-Omni, we verified the accuracy of spoken question answering on three sets: LlaMA MAMA--Qesssooss (Nachmani et al., 2023), Web Ques- tions? (Berant et al., 2013), and Trivia QA* (Joshi al., 2017). Since Web Questions and Trivia QA only have text, used the edge-tts tool with voice en-US-Briann BriinnNeuual to synthesize them into spoken modality. Tab. 3 shows the accuracy of Freeze Omni and its used backbone LLM Qwen2-7B IB-Inruuttt on these three sets. From the results, can observed that Freeze-Ommt exhibits excellent perfor- mance compared other models because the accuracy gap between it and the backbone LLM is smaller than that of
In addition, the NAR and AR speech decoders need model the L.M embedding outputs and speech tokens simultane- ously, but the spaces represented by these two are differentt
"nttps://github.com rroggieerrsserrch--daaaaaee. LLAMAA- rest-Set "hcrps://hoggingaaaa...ooraattt
Stantord/web__ueetti
Shtpps: http:///uggiiiie waagggiinfaaee aag.ace.ee..ee. co/ funasr/ paraformer-
htttps//giithbb..ommra////////

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
Table The accuracy (%) of different models in question answering on three sets. The models in the first four all speech input, while the models in the last two rows use text as input. The backbone LLM of Freeze- Oaezie is Qwen2-7B Beeen2--B-Instreee Instrucctt and the backbone LLM Moshi Helium. Both PrezeeOmmi and Qwen2-7B-Insh -n-nstruct use greedy search in the generation stage with zero-shot, and the accuracy is calculated using the output text. Except for Freeze-Omni and Qwen2-7B-Instrue previous evaluation results are derived from corresponding references.
Model
Modality
Web Q.
LJaMA Q.
Audio Trivia QA
SpeechGPT (7B) (Zhang et al., 2023)
Audio&Ter
6.5
21.6
14.8
Spectron (IB) (Nachmani et al., 2023)
Audio&Text
6.1
22.9
Moshi (7B) (Défossez et al., 2024)
Audio&
26.6
62.3
22.8
GLM-4-Voice (9B) (Zeng et al., 2024)
Audio&Tex.
32.2
64.7
39.1
Freeze-Omni (7B)
Audio&Tes.
44.73
72
53.88
Helium (Défossez et al., 2024)
Text Only
32.3
75
56.4
Qwen2-7B-Instruct
Text Only
45.13
77.67
63.93
Table 4. Detailed information of statistical latency. Among them, 50% represents the median, and 90% represents the percentile at 90. The unit of the results the table is (ms). results are completed using pytorch with 6íloat16 inference.
Latency description
Avg.
50%
90% %
LLM interrupted LLM generate first token chunk
478
468
750
First text token chunk - Prefill of speech decoder Prefill speech decoder Generate first speech token chunk
15
15
17
237
235
252
First speech token Chunk - Decode first PCM chunk
11
11
13
Total
745
753
1020
Moshi, which also verifies that Preeze-Omni has the same level of intelligence text and speech modalities. It is worth mentioning that thanks to the freezing of LLM dur- ing the training process, Treeze-Omni's performance the spoken question answering task even surpasses that of GLM-4- which uses speech training data far more than Freeze-Omni.
network latency (about 200 to 300ms), the average latency of Freeze-Omin used in real scenarios will be controlled at about 1.2 seconds. This result means Freeze-Omnt can deliver low-latency speech-to-speech dialogue experience for users.
4. Conclusion and future work
3.5. Analysis on end-to-end latency
In this paper, we propose Freeze-Omni, a text-audio multi- modal LLM capable of low-latency speech-to-speede dia- logue, which does not need fine-tuning the LLM backbone, showing excellent performance in various tasks, especially in the spoken question answering. In the future, to explore more speech dialogue capabilities, we plan to do the follow- ing:
To verify the latency of Freeze- -mmni for speech-to-speecl dialogue, we defined two parts of latency, namely statistical latency and non-statistical latency. The statistical latency refers to the time from the LLM being interrupted to the first PCM chunk of speech generated. Specifically, it can be divided into four parts as shown in Fig 4, these results based speech token chunk size of 40 and the use of text token chunk segmentation based on the sentence- split strategy. The non-statistiaal latency refers to the time from the real endpoint of speech to the L.LM outputting the interrupt state. This part needs to be measured manually and cannot be counted automatically. According our case analysis conclusion, the non-statistical latency is about one to speech encoder chunk sizes, and according to the experiment configuration above, this time is about 160ms to 320ms. In summary, if we consider the influence of
We will upgrade the speech encoder to general audio encoder to complete tasks like emotion understanding and audio captioning.
Under the condition of a frozen LLM, we will add more tasks to make the LLM complete more downstream tasks in speech dialogue, like the state prediction ability.
We plan to support multiple voices and instruction- following ability in the speech decoder part so that can obtain more instruction information from the hidden state of the LLM and provide richer speaking styles.
8

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
Acknowledgements
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2024.
Supported by National Natural Science Foundation of China under Grant No. 62441234. Supported by the Al & AI for Science Project of Nanjing University under Grant No.
2024300529.
Fang, Q. Guo, S., Zhou, Y., Ma, Z.., Zhang, S., and Feng, Y. Llama-omni: Seamless speech interaction with large lan- guage models. arXiv preprint arXiv:2409 240062409.06666,, 2024.
Impact Statement
Floridi, I.. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fu, C., Lin, H., Long, Z., Shen, Y., Zhao, M., Zhang, Y., Wang, X., Yin, D., Ma, L.,, Zheng, X., et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint arXiv:2408.0555 05211, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Fu, C.,, Lin, H., Wang, X., Zhang, Y.-F., Shen, Y., Liu, X., Li, Y., Long, Z., Gao, H., Li, K., et al. Vita-1.5: Towards gpt- 4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957. 2025.
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information process- ing systems, 2444-12469, 2020.
Gao, Z., Zhang, S., McLoughlin, I., and Yan, Z. Paraformer: Fast and accurate parallel Transformer for non-autoregressive end-to-end speech recognition. In Proc. INTERSPEECH, pp. 5079-5083. ISCA, 2022.
Berant, J., Chou, A., Frostig, R., and Liang, P. Semantic parsing on Freebase from question-answer pairs. Pro- ceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA. October 2013. Association for Com- putational Linguistics. URL https: // http///wllll.oooccccccc///c/..w. acllwwew /D13-1100 1160.
Graves, A., Fernández, S., Gomez, F. J., and Schmidhuber, J. Connectionist temporal classification: labelling unseg- mented sequence data with recurrent neural networks. In Cohen, W. W. and Moore, A. W. (eds.), International Con- ference on Machine Learning, ICML 2006, volume 148 of ACM International Conference Proceeding Series, pp. 369-376 ACM, 2006. doi: 10.1 14.11 00555/1148444 143844 1433911.
Bu, H., Du, J., Na, X., Wu, B., and Zheng, H. AISHELI-11: an open-source mandarin speech corpus and speech recognition baseline. In Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and As- sessment, O-COCOSDA 2017, pp. 1-5. IEEE, 2017. doi: 10.1009/CSDA.22117..388888
Jiang. A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, S., Casas, D. d. I., Hanna, E. B., Bressand, F,, al. Mixtral of experts. arXiv preprint arXiv:2401.044 2024.
Joshi, M., Choi, E., Weld, S., and Zettlemoyer, L. Trivi- aqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Chen, S., Liu, S., Zhou, L., Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. Vall-e 2: Neural codec lan- guage models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.053777 05440000577, 2024.
Loshchilov, I. and Hutter, F. Fixing weight de- cay regularization in adam. ArXiv, abs/1711 2017. URL https: happs://app /api...... sewanticscho...o.. org/ /ts//CoopuTTD::31244
Chu, Y., Xu, J., Yang, Q. Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., al. Qwen2-audio technical report. arXiv preprini arXiv:2407. 10759. 2024.
Nachmani, E., Levkovitch, A., Hirsch, R., Salazar, J., Asawaroengchai, C., Mariooryad, S., Rivlin, E., Skerry- Ryan, R., and Ramanovich, M. T. Spoken question answering and speech continuation using spectrogram- powered Im, arXiv preprint arXiv:2305.1 15255. 2023.
Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint orXiv:2410.000: 2024.

---

Freeze-Omni: A Speech-to-speech Dialogue Model with Frozen LLM
OpenAl.
https://opmmaa.cco/miimm cttpeiieooo/////x/////e/// hello-gpt-40/, 2024.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: An asr corpus based on public domain audio books. IEEE, 2015.
Ren, Y., Wang, T. Yi, J., Xu, L., Tao, J., Zhang, C., and Zhou, Fewer-token neural speech codec with time- invariant codes. arXiv preprint arXiv:2310.000// 2023.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https::/ /twpp:////wemmm.////////m/ qwenlm. www.ggtthh..... blog/qwen2.5/.
http://wwwwn....gu.u.bt.e /
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U.. Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, V. N., and Garnett, R. (eds.), Annual Conference on Neural Information Processing Systems, NeurlPS 2017, pp. 5998- 6008, 2017.
Xie, Z. and Wu, C. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. 16725, 2024a.
Xie, Z. and Wu, C. Mini-omni2: Towards open-source gpt- 4o model with vision, speech and duplex. arXiv preprint arXiv:2410.11190, 2024b.
Yao, 7., Wu, D., Wang, X., Zhang, B., Yu, F., Yang, C., Peng, Z., Chen, X., Xie, L., and Lei, X. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. arXiv preprint arXiv:2102.01547, 2021.
Zeng, A., Du, Z., Liu, M., Wang, K., Jiang, S., Zhao, L.., Dong, Y., and Tang, J. Glm-4-voice: Towards intelli- gent and human- end-to-end spoken chatbot. arXiv preprint XrXi:24122.02611 2024.
Zhang, B., L.v, H., Guo, P., Shao, Q., Yang, C., Xie, I.., Xu, X., Bu, H., Chen, X., Zeng, C., Wu, D., and Peng, Z. WENETSPEECH: A 10000+ hours multi-domain mandarin corpus speech recognition. In International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022. pp- 6182-6186. IEEE, 2022. doi: 10.1 ICASSP44922.2022..74665
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint 10000 2023.
10


Reviews:
[{"review_id": "UynMneE2pX", "reviewer": "Reviewer_WoBP", "summary": "This paper proposes a framework to provide a frozen text LLM with spoken dialogue abilities, by integrating it with a speech encoder and a speech generation system. When orchestrated by an auxiliary turn prediction module, this allows for the model to interact along multi-turn conversations with a latency lower than one second. While easily turning text LLMs into spoken dialogue systems is of wide interest, this paper is inappropriate for publication at ICML in my opinion. First, combining a text LLM with a speech encoder and a speech decoder has been done several times (Spectron, GLM-4Voice, Llama-Omni, etc.) and while this does not make the topic outdated, this paper does not present contributions in a way that convinces me that it does something fundamentally better than others: the model section is too vague (e.g. see below the relation between the NAR and AR modules) and the relation to previous work is not discussed enough to highlight the novelty of the framework. Second, the experiments are limited to ASR (as a proxy for speech encoder quality) and single-turn QA which are not sufficient to properly evaluate what claims to be a multi-turn spoken dialogue system. Finally, comparing Freeze-Omni to end-to-end models such as Moshi is a bit unfair: as Freeze-Omni uses a frozen LLM, it can be adapted easily, but the counterpart is that its text backbone acts as an information bottleneck that loses all non-linguistic information. Avoiding this limitation is the exact motivation behind end-to-end speech models! Overall, while quick adaptation of text LLMs to speech is of crucial interest both in research and in industry, the paper in its current state does not provide a convincing and replicable method and I strongly encourage the authors to submit a more detailed paper, with a more precise focus on the model and more extensive experiments in particular regarding multi-turn performance.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "See \"Summary\".", "methods_and_evaluation_criteria": "The writing of the method section could be considerably improved. In particular, I found the mechanism of the NAR and AR models to remain mysterious after carefully reading this section several times. On the other hand, some elementary descriptions such as Figure a are not useful to improve the understanding of the method.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "Overall, the experimental setup poses several problems:\n1) The word error rates reported in Table 1 are much worse than sota in some cases. This may mislead the reader thinking that the proposed method is e.g. sota on Librispeech test-clean while sota on this dataset is below 1.5% WER.\n2) Evaluating on question answering and emphasizing that Freeze-Omni gets a performance that is close to its LLM backbone is questionable: as the LLM is frozen, it is expected that answers will be identical to the textual topline as long as a) the speech encoder transcribes input audio properly b) the speech decoder produces speech that is intelligible enough to Paraformer for the transcribed answer to be verified against the ground-truth. This score here thus characterizes the performance of ASR and TTS (and probably to some extent the ability of the LLM backbone to correct ASR errors) which defeats the purpose of this metric, which is intended to measure the knowledge of end-to-end spoken LLMs.\n3) The Q&A evaluation only evaluates single-turn interactions. As the authors perform training on multi-round sequences, they should provide metrics for multi-turn behaviour beyond the latency metrics reported in Table 4. Otherwise, it is not possible to favor their multi-turn setting rather than resetting the state of a single-turn dialogue model between every turns.\n4) There are almost no ablation studies whatsoever. This does not allow identifying the key findings of authors through their experiments, and does not facilitate replication as it is unclear which components should be the most precisely reproduced.", "relation_to_broader_scientific_literature": "See \"Summary\"", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The presentation of the paper is below ICML standards. As an example, the style of Figure 1 which is mostly empty space and with a large logo is unusual. Moreover, most acronyms are never defined. This is particularly problematic for not-so-standard acronyms such as NAR which afaik was introduced in Vall-E for \"non-autoregressive\". Moreover, the writing style can be improved, with many typos and inconsistent tense (e.g. Section 2.4).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Section 2.2.2. mentions that the second stage of training---which connects the pretrained speech encoder to the frozen LLM--- involves adding several trainable special tokens. What are those? How many of them, what separates them? How are they presented to the model and for which precise purpose? \n- Section 2.3.1 explains that the speech generator combines a non-autoregressive (NAR) and an autoregressive (AR) model similarly to Vall-E. However, Vall-E first predicts the first codebook of the neural codec, while the NAR model generates the other levels. This is unlike Freeze-Omni which is claimed to first apply a NAR model, followed by an AR one. This would benefit from clarifications.\n- Section 3.1.1 mentions that an LLM and a TTS systems are used to generate synthetic data. Which ones?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q7xWaXUCij", "reviewer": "Reviewer_BFAM", "summary": "The paper introduces Freeze-Omni, a novel speech-text multimodal large language model (LLM) designed for speech-to-speech interaction while keeping the backbone LLM’s parameters frozen throughout the training process. This architecture enables low-latency, end-to-end spoken response while preserving the intelligence of the original LLM, addressing key challenges such as catastrophic forgetting and high computational costs associated with fine-tuning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes. The claims are all supported.", "methods_and_evaluation_criteria": "Yes. The methods and evaluation make sense for the problem.", "theoretical_claims": "None", "experimental_designs_or_analyses": "More Comprehensive Evaluation on Speech Input and Output：The paper mainly evaluates the accuracy of speech recognition (ASR) for speech input and the character error rate (CER) for speech output. However, additional metrics such as Word Error Rate (WER) for output speech and intelligibility scores (e.g., MOS – Mean Opinion Score) could provide a more comprehensive assessment of speech quality.\n\nComparative Analysis with More Baselines：While the paper compares Freeze-Omni with a few existing models, more state-of-the-art speech-to-speech systems (e.g., recent versions of GPT-based multimodal models) could be included for a broader comparison.\n\nLatency Optimization：The paper provides latency analysis but does not discuss potential optimizations. Exploring methods to reduce the response time further, such as improving the efficiency of the speech encoder or speech decoder, could be beneficial.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "-Strengths:\n1. Innovative Frozen-LLM Architecture: The proposed Freeze-Omni model maintains the parameters of the backbone LLM completely frozen throughout training. This prevents catastrophic forgetting and ensures that the model retains the intelligence of the original LLM while integrating speech modalities.\n\n2. Three-Stage Training Strategy: The proposed three-stage training process efficiently models both speech input (ASR) and speech output (TTS) while keeping the LLM frozen.\n\n3. Low-Latency Speech Interaction: The model is optimized for real-time speech-to-speech dialogue, achieving an average response time of 1.2 seconds in real-world scenarios. This is significantly lower than traditional ASR + LLM + TTS pipelines.\n\n4. High Accuracy in Spoken Question Answering: The model achieves competitive performance in spoken Q&A tasks, with an accuracy gap between Freeze-Omni and its text-only backbone LLM smaller than other speech-enabled LLMs like Moshi.\n\nFreeze-Omni presents a well-designed, efficient, and practical approach to integrating speech-to-speech interaction into LLMs while preserving intelligence, reducing computational overhead, and maintaining low latency. The combination of freezing the LLM, modular training strategies, and real-time duplex dialogue capabilities makes it a notable advancement in the development of multimodal conversational AI systems.\n\n-Weaknesses:\n1. More Comprehensive Evaluation on Speech Input and Output：The paper mainly evaluates the accuracy of speech recognition (ASR) for speech input and the character error rate (CER) for speech output. However, additional metrics such as Word Error Rate (WER) for output speech and intelligibility scores (e.g., MOS – Mean Opinion Score) could provide a more comprehensive assessment of speech quality.\n\n2. Comparative Analysis with More Baselines：While the paper compares Freeze-Omni with a few existing models, more state-of-the-art speech-to-speech systems (e.g., recent versions of GPT-based multimodal models) could be included for a broader comparison.\n\n3. Latency Optimization：The paper provides latency analysis but does not discuss potential optimizations. Exploring methods to reduce the response time further, such as improving the efficiency of the speech encoder or speech decoder, could be beneficial.\n\n4. Scalability and Adaptation to Different LLMs Not Fully Explored: While the paper claims Freeze-Omni can work with any LLM, it is only tested on Qwen2-7B-Instruct. There is no empirical evaluation of how the approach generalizes to larger models or smaller, more efficient models that might be deployed on edge devices.\n\n5. Training Efficiency vs. Larger-Scale Training: The paper highlights the efficiency of training on 8 GPUs with only 60K Q&A data, but it does not explore whether performance would scale with larger datasets.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p1GQ4Vwnbv", "reviewer": "Reviewer_RJrj", "summary": "The paper proposed Freeze-omni, which can enable speech input and output capabilities for any LLM backbone without tuning its parameters. In such a case, the model can achieve end-to-end chat experience without losing the intelligence behind the LLM backbone. The framework consists of multiple speech encoder and decoder modules to be aligned with LLM, training with different stages, e.g. speech input modules for understanding, speech output modules for generation, and chunk-level state prediction for interruption. The highlight of the performance is the high accuracy on several speech synthesized language benchmark compared to other speech LLM models, which benefits from the freezing LLM.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The core idea of the framework is to maintain the intelligence of LLM by freezing its parameters. The results of high accuracy on language benchmark provided the evidence for such claims. \n\nA potential issue is the claim of omni for the proposed framework. It seems the LLM is expanded with speech modality only. How about the vision modality? The omni is over-claimed considering the model capabilities. I have also noticed that there are many references in this paper using omni but with audio modality only. This can be misleading to the entire community.", "methods_and_evaluation_criteria": "The proposed method is intuitive, and the evaluation criteria is pretty standard in speech/audio LLM research.", "theoretical_claims": "This is not a theoretical paper, but more based on empirical study.\n\nThere is a very fundamental issue that the paper didn't discuss, that is, how would the framework compare to the ASR+LLM+TTS cascades system? There is no such comparison in the experimental section.", "experimental_designs_or_analyses": "The paper lacks some important comparisons, especially when this paper is more based on empirical study.\n\n1. No comparison to cascades systems. The proposed system is very similar to train an ASR model as LLM input and a TTS model with LLM output. The comparison would provide the community insight and better evident the effectiveness of the method.\n\n2. In terms of the ASR performance in Table 1, pls compare with the model using continuous features as input, e.g. Qwen2-audio, since Freeze-omni is using the similar. \n\n3. It seems that Table 3 presents the speech-to-text QA performance. How about speech in and speech out QA? One of the contributions of the paper compared to previous work is the speech output. The paper should discuss about speech-to-speech QA performance.\n\n4. What is the performance of speech-to-text QA after aligning the speech input module? Is it the same as what is shown in Table 3 after aligning the speech out modules? The question is regarding to whether adding speech output modules can affect the performance for speech-to-text QA.", "relation_to_broader_scientific_literature": "Freeze LLM for enabling speech functions is explored previously in literature. See \"Essential References Not Discussed\". This paper expands the core idea with speech output.", "essential_references_not_discussed": "There are several papers discussing how to enable speech functions for LLM with its parameters freezing. However, these papers do not expand the model with speech output, but I think it is worthing noting these works:\n\nWang, Chen, et al. \"Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing.\" arXiv preprint arXiv:2309.00916 (2023).\n\nFathullah, Yassir, et al. \"Audiochatllama: Towards general-purpose speech abilities for llms.\" arXiv preprint arXiv:2311.06753 (2023).\n\nLu, Ke-Han, et al. \"Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data.\" arXiv preprint arXiv:2409.20007 (2024).\n\nKang, Wonjune, et al. \"Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech.\" arXiv preprint arXiv:2410.01162 (2024).\n\nFan, Ruchao, et al. \"AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM.\" arXiv preprint arXiv:2412.01145 (2024).", "other_strengths_and_weaknesses": "Strengths: This work is a great study to expand LLM with speech functions without destroying the language capabilities. \n\nWeakness: \n\n1. I don't like the omni as the model does not include the vision modality. Shouldn't it be Freeze-audio or something?\n2. The paper lacks comparison and ablation studies.\n3. Reference missing.", "other_comments_or_suggestions": "Overall, the paper is more like a technical report and does not provide enough ablation studies to provide insights to the community.\n\nThe proposed method is without strong theoretical claims. More experimental study is very necessary to strengthen the paper. \n\nThe proposed method involves many stages of training. People might be very interested in how each part affect the performance.", "questions_for_authors": "I have read the paper very carefully and do not have further question.\n\nThe authors, please correct my comments if you feel I don't understand it in the right place.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has got mixed reviews. The reviewer who gave a score 1 mainly complained about the presentation instead of technical aspects. The other weak score of 2 is mentions integrationg computer vision, which is out of scope for this paper. I am mainly agreeing with the reviewer who gave 4 and I think this paper must be presented in ICML.
Decision: Accept (poster)