OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis
Authors: Run Luo
Ting-En Lin
Haonan Zhang
Yuchuan Wu
Xiong Liu
Yongbin Li
Longze Chen
Jiaming Li
Lei Zhang
Xiaobo Xia
Hamid Alinejad-Rokny
Fei Huang
Min Yang
Venue: NeurIPS 2025 poster ()
Keywords: omnimodal large language models
modality alignment
real-time inference
Primary area: deep_learning

Abstract: Recent advancements in omnimodal learning have significantly improved understanding and generation across images, text, and speech, yet these developments remain predominantly confined to proprietary models. The lack of high-quality omnimodal datasets and the challenges of real-time emotional speech synthesis have notably hindered progress in open-source research. To address these limitations, we introduce OpenOmni, a two-stage training framework that integrates omnimodal alignment and speech generation to develop a state-of-the-art omnimodal large language model. In the alignment phase, a pretrained speech model undergoes further training on image-text tasks, enabling (near) zero-shot generalization from vision to speech, outperforming models trained on tri-modal datasets. In the speech generation phase, a lightweight decoder is trained on speech tasks with direct preference optimization, which enables real-time emotional speech synthesis with high fidelity. Extensive experiments demonstrate that OpenOmni surpasses state-of-the-art models across omnimodal, vision-language, and speech-language benchmarks. It achieves a 4-point absolute improvement on OmniBench over the leading open-source model VITA, despite using 5$\times$ fewer training examples and a smaller model size (7B vs. 7$\times$8B). Besides, OpenOmni achieves real-time speech generation with less than 1 second latency at non-autoregressive mode, reducing inference time by 5$\times$ compared to autoregressive methods, and improves emotion classification accuracy by 7.7\%. The codebase is available at https://github.com/RainBowLuoCS/OpenOmni.

OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis
Run 1:23; Ting-En Lin", Haonan Zhang', Yuchuan Wu', Xiong Liu', Yongbin Li Longze Chen Jiaming Li Lei Zhang' Xiaobo Xia
Hamid Alinejad-Rokny" Fei Huang", Min Yang
SShenzhen Key Laboratory for High Performance Data Mining,
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
Shenzhen University of Advanced Technology 'Tongyi Lab, Alibaba Group
à National University of Singapore *MoE Key Laboratory of Brain-insppred Intelligent Perception and Cognition, University of Science and Technology of China UUniversity of New South Wales, Sydney, New South Wales, Australia (ting-en. 1te, shuide. @al halibaba-innc.oo com
{r. luo, min. yang @siat ac
Abstract
Recent advancements in omnimodal learning have significantly improved under- standing and generation across images, text, and speech, yet these developments remain predominantly confined to proprietary models. The lack of high-quality omnimodal datasets and the challenges of real-time emotional speech synthesis have notably hindered progress in open-source research. To address these limi- tations, introduce OpenOmni, two-stage training framework that integrates omnimodal alignment and speech generation to develop state-of-the-an omni- modal large language model. the alignment phase, pretrained speech model undergoes further training on image-text tasks, enabling (near) zero-shot general- ization from vision to speech, outperforming models trained on tri-modal datasets. In the speech generation phase, lightweight decoder is trained on speech tasks with direct preference optimization, which enables real-time emotional speech synthesis with high fidelity. Extensive experiments demonstrate that OpenOmni surpasses state-of-the-ar models across omnimodal, lision-language, and speech- language benchmarks. It achieves 4- 4-point absolute improvement on OmniBench over the leading open-source model VITA, despite using 5x fewer training ex- amples and smaller model size (7B vs. 7x8B). Besides, OpenOmni achieves real-time speech generation with less than 1 second latency at non-autoregressive mode, reducing inference time by 5x compared to autoregressive methods, and improves emotion classification accuracy by 7.7%. The codebase is available at https:/ / //gi gitt.u......... wet.ha...o//aaaaaaaee www..//aainoowwooo///eeeeeeeeeee
Introduction
The success of large language models (LLMs) [1, 2, 3, 5, 6, 7, 8, 9, 10] has driven rapid ad- vancements in multimodal large language models (MLLMs) [11, 12, 13, 14, 15, 16, 17], partic-
"Work done while interning at Tongyi Lab.
Minn Yang and Yongbin Li are corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

borbot
dieect castoval cpferracooptniiiio Yo (acáag car
Spock
louge
Sperch Decodker
Wend Fribeiiing
happy
mni Language Model
Omni Language Model
Alggaand
Cho Kil..
Canyos Ki...
Avent
Figure 1: Overview of the motivation and architecture of OpenOmni. (a) OpenOmni adopts a progressive alignment strategy to generalize from vasion-language to speech-language tasks, avoid- ing the need for costly tri-modal datasets and resources. (b) OpenOmni integrates a lightweight end-to-end speech decoder, enabling parallel text and speech generation while effectively reducing inference latency. (c) By utilizing DPO, OpenOmni generates emotionally coherent and context- aware speech without relying additional control modules or handcrafted prompts. For simplicity, our core architecture is presented without the connectors between modules.
ularly in vision-language models (VLMs) [18, 19, 20, 21, 22, 23] and speech-language models (SLMs) [24, 25, 26, 27, 28]. These innovations mark a paradigm shift in machine understanding and human-computer interaction, fueling interest in omnimodal large language models (OLLMs)) which are models that integrate vision, language, and speech into unified system. The emergence of GPT-4o underscores the potential of holistic multimodal AI, yet open-source alternatives remain significantly behind.
Despite their promise, existing open-source OLLMs [29, 30, 31, 32] face three fundamental chal- lenges, limiting their performance in real-world applications. First, training fully end-to-end OLLMs requires high-quality tri-modal datasets (images, speech, and text), which are scarce, expensive, and difficult to curate at scale. Most open-source models rely on true tri-modal corpora and ignore pair- wise datasets (e.g., image-text or speech-speede resulting in suboptimal cross-modal alignment and weaker generalization. Without effective zero-shot alignment strategies, these models struggle to transfer learned representations across modalities, reducing their robustness in realistic multimodal tasks. Second, existing models predominantly rely autoregressive (AR) architectures, which gen- crate outputs sequentially, introducing high inference latency that hinders real-time multimodal in- teraction. Speech generation, in particular, is slow, as most models integrate external text-to-speech (TTS) modules [33], resulting in latency overhead and preventing end-to-end optimization. Achiev- ing low-latency multimodal synthesis is essential for applications such conversational AI, assis- tive technologies, and real-time interactive agents, where response time directly affects usability. Finally, emotionally expressive speech is critical for natural and engaging human-computer inter- actions. However, current OLLMs fail to generate emotionally consistent responses. Most models lack self-awareness, producing flat and robotic speech that does not modulate prosody, tone, or sen- timent based on conversational context. Without direct preference optimization (DPO) [34, 35] for emotional speech, existing models struggle to align speech intonation with user emotions, leading to inauthentic and disconnected interactions. To summarize, these challenges significantly constrain the real-world applicability of open-source OLLMs, leaving commercial models far ahead in omni- modal reasoning, real-time interaction, and expressive speech synthesis.
To bridge this gap, we propose OpenOmni in this paper, which is a fully open-source two-stage training framework that enables efficient omnimodal learning while addressing the key limitations of existing models. As illustrated Figure 1, OpenOmni introduces a progressive alignment strategy that enables cross-modal generalization from Dision-language tasks to inating the need for expensive tri-modal datasets and computing resources. It further incorporates a lightweight and end-to-end speech decoder that facilitates parallel text and speech generation, which drastically reduces inference latency compared to autoregressive models. Moreover, by leveraging direct preference optimization (DPO), our model generates emotionally coherent and context-aware speech without requiring additional control modules or handcrafted prompts.
language tasks, elim-
Extensive experiments confirm that OpenOmni achieves state-of-the-arr performance in omnimodal alignment, real-time speech synthesis, and emotional speech generation. Compared to VITA [31], the leading fully open-source OLLM, which employs a 7 8B language model trained on 5M sam-

---

ples, OpenOmni attains superior results with smaller model size (7B vs. 7xBB) and 3x fewer train- ing samples (1.6M vs. 5M) while outperforming VITA by four absolute points on the OmniBench benchmark [36]. Additionally, OpenOmni reduces speech generation latency by 5x, achieving real- time inference (<1 second) and improving emotion classification accuracy by 7.7%. Our main contributions are summarized as follows:
High-quality speech datasets. We construct O2S-300K and FO2S-9K, comprising 8,000 hours of bilingual exxtsyytteesized speech, which enables efficient speech generation and emotional preference optimization.
Effective zero-shot omnimodal alignment. We introduce a scalable and model-agnostic framework that enables low-resource and rapid omnimodal alignment using text as a pivot, followed by speech generation and emotional preference training. This approach allows the rapid development of an advanced aol-m assistant akin to GPT-4o.
An end-to-end omnimodal LLM. We train OpenOmni with integrated text, image, and speech understanding progressively. After speech generation training and emotional pref- erence optimization, OpenOmni can naturally produce real-time emotional speech.
2
Related Work
Vision-language models. The rapid advancement of vision lainu-la models (VLMs) has been largely driven by the remarkable success of large language models (LLMs) [37, 38, 39, 40, 41] and the increasing availability of diverse image-text instruction data [18, 42, 43, 44, 45, 46, 47, 48). LLaVA [18] and MiniGPT-4 [49] demonstrate strong cross-task generalization by integrating vi- sual encoders with LLMs through lightweight connector modules trained on instruction datasets. To further enhance visual perception, LLaVA-NeXT [19] employs dynamic resolution techniques, which improve the adaptability to images of varying sizes and complexities. Expanding beyond conventional methods, DreamLLM [50] explores interleaved generation, enabling the simultane- production of images and text within shared multimodal context. Meanwhile, DEEM [21] enhances model robustness by employing diffusion models to extract visual features, which re- places traditional visual encoders and simplifies the overall architecture. These innovations collec- tively contribute to advancing vision-language reasoning in multimodal systems. Readers can refer to [51, 52, 53, 54, 55) for more details and recent advances in VLMs.
Speech-language models. Recent advancements in speech-language models (SLMs) [56, 57, 58, 59, 60] have significantly improved human-computer interactions by enabling direct speech process- ing without relying on intermediate text transcription. For example, SpeechGPT [11] and LLaMA- Omni [25] eliminate the need for explicit text-based transcriptions, reducing latency in multimodal content generation. For full-duplex dialogue systems, Moshi [61] and OmniFlatten [62] introduce mechanisms for handling simultaneous speech and text streams, adeptly managing challenges such as overlapping speech and interruptions [63]. Meanwhile, Freeze-Omni [64] introduces an innova- tive training approach that preserves the core capabilities of the original LLM, allowing low-latency speech-to-speed dialogue without requiring modifications to the pre-trained architecture. Focusing on emotional speech synthesis, Emo-DPO [65] applies direct preference optimization to generate expressive and controllable emotional speech, which addresses the emotional coherence gap in ex- isting ayeech-language models. These developments mark a significant shift towards more natural and real-time speech interactions in multimodal AI systems. Readers can refer to [66] for more details of SLMS.
Omnimodal language models. With the development of multimodal research, models are in- creasingly shifting towards unified frameworks that seamlessly integrate diverse input and out- put modalities [67]. By tokenizing different data types into shared representation, models like AnyGPT [29] and Unified-IO 2 [68] achieve seamless cross-modal task adaptability, allowing them to process audio, text, and images without significant architectural modifications. More recently, Mini-Omni2 [69] extends multimodal capabilities by integrating visual and auditory encoders, en- abling real-time multimodal responses while incorporating mechanisms for detecting and interpret- ing semantic interruptions. Meanwhile, Video-SALMMONN [30] enhances video understanding by incorporating fine-grained temporal modeling, improving the model's ability to interpret speech
3

---

Speech-Tee Text A ligument
Image-Text Aligmment
Text-guided Speech G Generation
A cute dog with hat.
where today?
cute dog with hat.
Speech Decoder
TCM
Omni Language Model
Omni Language Model
Omni Language Model
Speech
espeactoon
Image
<imageria Can you cive me docciipioo of the Image?
Imagee
<imagen 'y Please arooaar quasition input speech.
inputs speadh
conrespooding
where go today?
Figure 2: Overview of the training process of OpenOmni. To enable zero-shot omnimodal learn- ing and real-time emotional speech generation, OpenOmni undergoes a progressive three-stage train- ing process: (1) Speech-text alignment. A speech encoder extracts continuous speech and text fea- tures for alignment learning, equipping the large language model with speech understanding capa- bilities. (2) Image-text alignment. An image encoder extracts continuous image and text features, facilitating alignment learning that enhances OpenOmni's 's image comprehension and instruction- following abilities. This process also establishes implicit omnimodal alignment, which enables muni-uuderstanding. (3) Text-guided speech generation. A lightweight speech decoder trained using high-quality synthesized speech dialogue data, with focus on direct preference optimization for emotional speech. This final stage allows OpenOmni to generate real-time and self-aware emo- tional speech. A text-guided module (TGM) is utilized to accelerate the training convergence.
and actions within videos. To enhance human- compttc interaction, VITA [31] introduces du- plex communication schemes, enabling fluid and intuitive exchanges between users and AI models. EMOVA [32] further extends the expressive capabilities of multimodal systems by integrating con- trollable emotional speech synthesis, which provides more natural and engaging user interactions.
Building upon these advancements, OpenOmni introduces novel method for nearly zero-shot om- nimodal alignment across language, vision, and speech, which incorporates self-aware emotional speech synthesis to enhance expressiveness and realism. By optimizing for speed, data efficiency, and generalization, OpenOmni achieves state-of-the-arr performance in omnimodal tasks, surpassing previous models in real-time speech generation, multimodal alignment, and emotion-aware synthe- sis. Note that compared to Qwen-Omni [70], OpenOmni a fully open-source solution focused on achieving advanced OLLMs under limited training and data resources, which helps researchers to easily conduct their studies and accelerate innovation in the field.
Method
In this section, we first formulate the omnimodal learning problem and provide overview of the training procedure OpenOmni as demonstrated in Figure 2. Afterward, we describe the specific training procedures for omnimodal alignment and real-time speech generation step by step.
3.1 Problem Setup and OpenOmni Overview
Problem setup. Omnimodal learning aims to model the relationships between images (x"), speech (x*), and text (x'). The speech-to-text alignment task, which generates relevant text responses given input speech encoded by speech encoder h,,-), is formulated as learning P Pxxx'] (x'))x)))), pa- rameterized by Φ. Similarly, the image-to-teds alignment task, which involves generating textual descriptions for input images encoded by an image encoder hy (-). is modeled as learning the condi- tional distribution po "xxx (x")), parameterized by 0. Lastly, the omnimodal-to-specch generation task, which synthesizes speech responses based on input text, speech, and images, is represented as learning Pa Pxxx (x (x*)), hy (x)))), parameterized by ub where fiIM represents the large language model.

---

In the setting of standard omnimodal learning, training typically relies sengee-specc--eer triples D. {(x), x x Nevertheless, high-quality image-hext-speech datasets are scarce. To mit- igate this limitation, we introduce text as pivot, which leverages large-scale speech-text dataset D,,=((x; x
M and image-text dataset D.. [(x), x())=== where M >> K and > K. In- spired by human learning mechanisms, where individuals naturally align visual concepts with speech across languages, OpenOmni transfers visual concepts learned from image-texi tasks to speech un- derstanding. Technically, OpenOmni decomposes the omnimodal alignment process into two con- secutive stages: speech-text alignment and image-text alignment. The speech-text alignment stage establishes tross-mddal alignment between speech x* and text x'. This is achieved by training speech LLM on texc-hpeech pairs Dz1 with the objective p& (x' (xxx), which also ensures that the hidden representations of semantically similar speech-text pairs are close. In the image- align- ment stage, OpenOmni utilizes image-text pairs D., to optimize the objective po (x")). Note that OpenOmni is architecture-agnoste which allows flexible integration with existing advanced model architectures. Below, we detail OpenOmni.
3.2 Speech- Text Alignment
We incorporate speech encoder hg extract audio features from input speech x'. These audio features hyg (xx))) are then replaced with corresponding text as input into the LLM. Following recent work train speech conversation models [25, 24, 111, we pretrain OpenOmni large scale of text-speech pairs using the language modeling objective:
M
L Don) = log Pe (x) (x))).
(I)
3.3 Image-Texx Text Alignment
We incorporate image encoder h, to provide visual features h,(x"). These visual features then concatenated with the text embedding as input into the speech LLM. Following prior work to train image-text conversation models [18, 42], OpenOmni's 's training process for image-text align- ment consists of two sub-stages: image-text pretraining and image-text instruction tuning.
Image-text pretraining. In this sub-stage, we pretrain the visual module to align it with the LLM on a large scale image-text pairs using the language modeling objective:
N
(0, (r.))
=
Σ
log po Poggo(())///) (x))
Here we fix the parameters of the LLM to prevent short texts in the image-text pairs from influencing the general capabilities.
Image-text instruction tuning. To enhance models' capabilities in following human instructions, we conduct instruction tuning on elaborately curated multimodal instruction tuning datasets built by blending existing image-text instruction tuning datasets. We denote this image- instruction tun- ing dataset as D ((x), x!_ x! { where x!" denotes the instruction and x!² is the response. Both the visual module and the speech LLM are then fine-tuned by maximizing the probability of the response:
C,, ((,) = Σ
log po (x (Lx))x))
(3)
5

---

Remark. We observe a quasi-zero-shot transfer capability in OpenOmni within this scenario. When instruction tuning is performed exclusively the image-text dataset, the model demon- strates the ability to respond accurately to an image x" and either a text-based question x or an instruction provided in speech x However, its responses can be attributed to the inherent similarity between the hidden representations of textual and spoken instructions learned by the I.IM, Le., (x
predominantly in text. This behavior
fLLM(h, (x59)) Consequently, the model sat- isfies the following approximation: po (x!* (x"), f(LM(x(9)) = po po (xta h,, (x"), fLLM (LLM(,, ())(x))))) OpenOmni completes the progressive omnimodal alignment, enabling the LLM to achieve com- prehensive understanding across image, text, and speech modalities.
3.4 Text-Guided Speech Generation
For speech generation, we incorporate speech decoder h to generate speech based the out- put of the LLM fULM. The speech generation training process in OpenOmmi consists of two sub-stages: speech decoder training and emotional speech direct preference optimization (DPO).
Speech decoder training. To equip OpenOmni with real-time speech generation for enhanc- ing interactive experiences, we adopt a stream- ing speech decoder, which supports both au- toregressive (AR) and non-autoregressive (NAR) speech decoding modes. Besides, curate dataset, termed OpenOmmi-300K, consisting of 300K single-round image-text instructions from MMEvol [42] and UltraChat [71] with corre- sponding speech responses for speech decoder training. We denote this dataset as D.
bae 1l bank outpr, zucch
Speech Demder (TOH
XM
Up Sample
Dmni Languae Moddl
Spuich Earoke
Gate
{(x", x x xx"",x/"))))) x>"")<=>>> where x²* is the speech response.
The structure of our speech de- coder. The speech decoder consists mix- ture of expert modules and multiple transformer layers, which achieves end-to-end speech unit learning through the connectionist temporal classification (CTC) loss.
Figure 3:
To process the speech response x we fol- low [11, 25] to discretize speech into discrete units. Specifically, we use pretrained speech to- kenizer [11] to extract continuous speech repre- sentations and then convert these representations into a single unit, resulting in the final discrete unit sequence x"a = [x] x2,, ....... J.. where x E {0, I...,, V 1} with V is the speech vocabulary size. The discrete units can then be converted back into a waveform using an additional unit-based vocoder [72], trained on English and Chinese datasets. As shown in Figure 3, we integrate the streaming speech decoder after the LLM to generate speech responses. The decoder consists a mixture of expert (MoE) layer and a tiny standard decoder-only language model. The MoE layer stabilizes training and accelerates conver- gence-withour without this layer, the speech decoder fails to train effectively. Similar to [73, 74, 25], the speech decoder takes the output hidden states from the L.M as input and generates the discrete unit sequence corresponding to the speech response in real-time.
Given the output hidden states of the text response, denoted by fLLM(X", x'1),, first pass these hidden states through the text-guiee -uiied module (TGM) to obtain the transformed hidden state c. Then, is fed into the speech decoder layers, leading the final hidden state sequence o. We con- nectionist temporal classification (CTC) [75] to align o with the discrete unit sequence x During training, CTC marginalizes over all possible alignments follows:
Lereltb, D(a) log Pas 4.2 ()) = log
Σ AEA(X!)
(A log Σ IP lo), AEA(X!
(4) where A(xua) denotes all possible alignments that collapse to xua During inference, the best alignment is selected as A* = arg max p(A(o). The corresponding discrete unit sequence is fed into the vocoder to synthesize the waveform.
Emotional speech DPO. To enable OpenOmni to generate self-aware, emotionally coherent, and expressive speech based on contextual history without additional control modules, we introduce

---

the CTC-DPO algorithm. This method enhances smooth and natural dialogue interactions and is formulated
LCTE-DPO = -xxx....) (3 (gg Tx (H(lx))
Tx (x1(2)) Teet (y/(x)
,3 log
)}
(5)
Ted (ywl))
where is a constant, a is the sigmoid function, and (yw, is the preference pair. Besides, the reference model Tref is the pretrained model from the real-time speech generation stage and remains fixed during DPO training. Only the policy model я. is updated. Compared to traditional reinforcement learning with human feedback (RLHF) [76], the DPO paradigm is simpler, more efficient, and more stable for aligning OpenOmni with self-aware emotional speech generation.
Following the Plutchik model of emotions [77], we construct a multi-turn dialogue preference dataset incorporating nine distinct emotions. Each preference pair consists of am emotionally con- gruent speech response unit sequence Yw = x which aligns with the conversational history. and an emotionally neutral sequence yi x which inconsistent with the context. The policy model t, during training is optimized as: log m (y|x) = log Σ A P (x;" o). After train- ing, OpenOmni is capable of generating real-time and emotionally expressive multi-turn dialogues.
Experiments
4.1 Implementation Details
In this subsection, introduce data construction and the models used. More details about the data and optimization strategy of OpenOmni can be found in Appendix A and Appendix D respectively.
Omnimodal alignment data. During the speech-text alignment phase, in addition to WeNet- Speech [78]. LibriSpeech [79], and AIShell-4 [80]. we exploit portions of shorter responses from O2S-300K, totaling 8,000 hours of data, for bilingual speech-text alignment training. For image- text alignment, we train OpenOmni on the LLaVA- AAVA--Perrann-599K [18]. Besides, the image-text instruction tuning stage, we fine-tune OpenOmni the compact high-quality dataset MMEvol [42] for efficient optimization.
Speech decoder training data. To support real-time speech generation, curate dataset of 300K instructions from MMEvol [42] and UltraChat [71] that include long responses for training the speech decoder. Specifically, answer pairs, rank the responses based on their length, and select 100K question-answer pairs with relatively long responses. To support bilingual output in Chinese and English, we translate 50K question-answer pairs into their corresponding Chinese versions using GPT-4o-mini API, and then convert the answers into the corresponding speech using Cosy Voice [33]. We employ the same method for text-conditioned speech synthesis on 200K randomly selected data from UltraChat, As result, we obtain 8,000 hours of high-quality bilingual speech generation data, termed O2S-300K.
decompose multi-turn dialogues into single-turn question-
Emotional speech DPO data. Based on the Plutchik model of emotions [77], which categorizes emotions into eight distinct types, we curate multi-turn speech preference dataset, EO2S-9K 9K., for self-awareness emotion evaluation. In more detail, we randomly select 200K samples from MMEvol and employ Qwen2-72B [2] to categorize responses into nine predefined emotions per round. From this, we extract 1K bilingual dialogues labeled with emotion categories, reserving an additional 100 samples as an emotional test set for evaluating self-aware speech generation. Since certain emotions, such as anger and sadness, are underrepresented in the MMEvol dataset, we augment the dataset using the GPT-4o-min API to ensure sufficient data for these categories. The final dataset maintains an equal representation of Chinese and English samples. To further enhance emotional preference training, we use Cosy Voice [33] to generate unconditional speech as negative samples and emotion- conditioned speech positive samples, constructing preference pairs for training direct preference optimization in emotional speech generation.
Models. We design the architecture following LLaVA series [18, 19), where the omnimodal large language model consists of four key components: an LLM (Qwen2.5-7B-Instton) [2]) for next token prediction, an image encoder (CLIP-VIT-L [81]) for extracting visual features, speech encoder
7

---

Table 1: Overall omni-understanding results OmniBench. In each case, the best result is indicated in bold, and the second-best result is marked with an underline.
Method
Action Activity
Story Description
Plot Inference
& Description
Identification
Contextual & Envirommental
Identity Relationship
Text 8ymbols
Count Quantity
Overall
5.67
Video SALMONN (13B) [30]
28.69
25.65
24.47
23:22
29.08
21.83
$2.00
26.63
26.53
Uniftedi222 (1.18) [68]
28,
22.17
32.49
30.81
28.37
21,83
16.00
13.35
27.76
Nafhedl02-XLargg (3.2B)
30.28
26.52
30.38
31.75
28.37
18.75
28.00
26.3
29.16
UmliaelO2-XXXLara (6.88) [68]
27.49
23.04
28.69
59
26.95
12.50
12.00
46.67
25.92
Baichuan- Omnii (7B) [81]
33.25
VITA (7xSB) [51]
33.47
34.35
27.00
36.02
43.97
31.25
24.00
6.67
33.45
[31]
OpenOmnni (78)
36.65
45.65
32,91
44.08
48.23
34.38
24.00
33.33
37.40
Table 2: Comparison with state-of-the-art methods on visual-language benchmarks. This in- cludes an indication of audio input/output support. The best performance among fully open-source models highlighted in bold.
w/ Andio
PT
RWQA
Proprietary Models
GPT-4
83.4
82.1
55.0
63.8
69.1
75,4
GPT-4e-miii
60.0
Weight Open-Source
MiniCPM-VP (8B) (85]
570M
73.5
45.8
Qwen2-VL-Char VL-Chat 120]
1.4B
60.7
36.4
81.9
50.6
58.2
52.7
69.7
ReichunnCunn (78) [84]
8M
76.2
74.9
47.8
51.9
47.3
62.6
EMOVA
7.4M
Fully Open- Source
Cambenin- (8B) [86]
2,5M
7M
50.7
34,3
47.0
41.8
23.1
64.2
MMEvol (7B) (42]
51.6
VITA (78) [31]
5M
74.7
71.4
39.7
44.9
45.3
74.3
59.0
Openthini (7B)
0.6M
1.7M
82.3
76.2
76.4
44.2
$2.7
46.7
74.8
64.3
(Whisper-aagge-vs [82]) for extracting audio features and streaming speech decoder (Qwen2.5. InS--nnttuut [2]) for generating vivid speech in real-time. Moreover, an image-text projector and a speech-text projector are adopted to align the image-text and speech-text modalities, respectively. The MoE module and the text-guided module are designed to align the omnimodal embedding and speech decoder efficiently and stably. For the autoregressive mode, we use the speech tokenizer from GLM4-Voice [83] with a vocabulary size of 16K, which leads to better speech quality. For non-autoregressive models, we use the Cosveice [33] speech tokenizer with a smaller vocabulary size of 6K, which facilitates faster convergence during CTC-based optimization. All experiments are conducted on 8XNVIDIA A100-80G GPUs.
4.2 Main Results and Discussions
OmniBench [36] is pioneering benchmark designed to evaluate nimodal large language models (OLLMs) by assessing their ability to integrate and interpret simul- taneous inputs from images, audio, and text. This evaluation framework consists of 1,142 question- answer pairs categorized into tasks that focus on cognitive and reasoning abilities, which poses sig- nificant challenges entity recognition, causal inference, and abstract concept comprehension. We compare our OpenOmni with other OLLMs on OmniBench, with results summarized in Table 1. No- tably, our model achieves excellent zero-shot omnimodal alignment using only two training phases: speech-text alignment and image-text alignment. Compared to the fully open-source state-of-the-an OLLM, e.g., VITA [31], which trained on tri-modal data (image- weagge-speec--ee.. triplets). OpenOmni achieves superior overall results on OmniBench (37.40 vs. 33.45) despite using significantly fewer training parameters (7B vs. 7 7BBB) and less image-text training data (1.6M vs. 5M). Furthermore, by leveraging text pivot, our method completes omnimodal alignment implicitly, which demon- strates enhanced scalability in scenarios with limited tri-modal data. addition OmniBench, we provide empirical results on AV-Odyssey Bench in Appendix B.
Omni-language evaluation.
Vision-Ianguage evaluation. To assess the effectiveness OpenOmni in aligning image-text modalities, we compare its performance against previous vision-language models (VLLMs) across eight representative benchmarks: MMBench-EN [87], MMBench-CN [87]. MMStar [88], Re- alWorldQA [89], MMMU [90], Math [91], AI2D [92], and HallusionBench [93]. To en-

---

Table 3: Comparison with state-of-the-art methods on speech-language benchmarks. In each
case, the best result among Omnimodal LLMs is indicated in bold.
AIShell-2 (ZH-CER)
Librispeech (EN-WER)
Model
Dev
Test
Test clean
Test other
S2T
T2S
S2T
T28
S2T
T2S
S2T
T2S
Speech LLM
SpeechT5 [95]
2.4
5.8
SALMONN [96]
2.1
4.9
Mini-Omni (7B) [69]
4.7
9.4
Freeze-Ommn (7B) [64]
3.2
7.7
Qwen2-Audio (7B) [24]
3.1
3.3
2.0
4.5
Omnimodal LLM
AnyGPT (13B) [29]
8.5
VTTA (7x)B) [31]
8.1
18.4
EMOVA (7B) [32]
10.3
7.9
4.0
VITA 1.5 (7B) [31]
3.4
7.5
OpenOmni (7B)
6.8
13.1
3.1
4.1
5.6
Table 4: Overall self-aware emotional speech generation results on the bilingual EO2S-9K test set. In each case, the best result is indicated in bold.
Model
Lang
Angry Disgusted
Fearful
Happy
Neutral
Other
Sad
Surprised
Overall
OpenOmni
ZH
89.7
54.8
33.3
92.3
23.7
57.9
w/ DPO
ZH
96.6
78.4
37.7
97.1
62.8
90.7
29.8
70.4
OpenOmni
EN
89.2
68.7
57.5
91.9
48.0
75.6
7.5
62.6
w/ DPO
EN
91.3
70.4
60.6
94.6
49.6
77.3
13,9
65.4
sure reproducibility and maintain consistency across all models and benchmarks, employ VLMEvalKit [94] for zero-shot evaluation. As shown Table 2, OpenOmni achieves superior results compared to the fully open-source state-of sfate-of-the-art OLLM, VITA [31], despite being trained on significantly less data. Notably, our model outperforms VITA with gains of 7.0% on MMBeneh- Chinese and 11.3% BellusionBennh. We also observe that the use of additional speech modals can further enhance the vision-language capabilities the model. Furthermore, compared to other fully open-source VLMs, OpenOmni maintains competitive performance despite reduced training data, which demonstrates the effectiveness of our image-text alignment strategy.
Speech-language evaluation. To evaluate the speech understanding and generation capabilities of our OpenOmni, we measure word error rate (WER) on the Alshell-2 [80] and Librispeech [79] benchmarks for two tasks: speech-to-text recognition (S2T) and text-to-speech generation (T2S). Specifically, for T2S evaluation, we use Whisper-large-V3 to transcribe OpenOmni's synthesized speech and compute WER against ground-truth text labels. As shown in Table 3, OpenOmni achieves the best performance on both S2T and T2S tasks for bilingual (Chinese and English) data, and outperforms other omnimodal models. These results indicate that OpenOmni not only compre- hends speech effectively but also generates fluent and high-quality audio while maintaining strong alignment between speech and modalities. Additionally, compared to VITA [31], which relies on separate text-to-speech (TTS) models, and EMOVA [32], which uses an autoregressive (AR) structure, OpenOmni demonstrates significantly faster speech generation via two-mode support. Owing to its end-to-end, lightweight, and non-auvoreeressive (NAR) decoding mode, OpenOmni can generate up to 30 seconds of speech with less than one second of latency, which achieves real- time speech generation at over 5x speed of autoregressive models.
Emotional speech synthesis evaluation. assess the effectiveness of direct preference optimiza- tion in emotional speech generation, we evaluate OpenOmni's 's self-aware emotional speech synthe- sis on the BO2S-9K test set. Specifically, we use Emotion2Vec [97] to classify the emotions in the generated speech and measure accuracy against ground-truth labels. As shown in Table 4, direct preference optimization for emotional speech effectively enhances OpenOmni's ability to generate emotionally expressive speech. This improvement is particularly evident in bilingual and multi-

---

turn emotional speech generation tasks, demonstrating the model's ability to produce natural and contextually aware speech with accurate emotional intonation.
We also provide ablation studies to investigate the text-guided module (TGM), the number of layers and experts in the speech decoder, training strategy in alignment stages, and the performance of order in progressive alignment. Due to the limited page, experimental results and the following discussions can be checked Appendix C.
5 Conclusion
In this paper, we introduce OpenOmni, a novel ommimodal model that leverages text as a pivot to achieve tri-modal zero-shot aligmment, which addresses the challenge of limited tri-modal data. By integrating a lightweight streaming speech decoder with direct preference optimization for emo- tional speech, OpenOmni enables real-time, self-aware, and high-quality speech interactions. The extensive evaluations demonstrate that OpenOmni achieves sfate-of-the-art performance on multiple benchmarks while using significantly fewer training parameters and less training data than previous advanced models. Comprehensive ablation studies and discussions are also presented to rigorously validate our claims.
Acknowledgements
This work was supported by Alibaba Research Intern Program. Min Yang is supported by Guang- dong Basic and Applied Basic Research Foundation (2025B151502003 and 2024A151503016) Xiaobo Xia is supported by MoE Key Laboratory of Brain-inspired Intelligent Perception and Cog- nition, University of Science and Technology of China (Grant No. 2421002).
References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux. Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 230.:::300.1397/ 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint 1.XIv:2309.16009 2023.
[3] Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou A survey on self-evolution of large language models. arXiv preprint arXiv:2404. 14387, 2024.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-ri. Incentivizing reasoning capability in I1ms via reinforcement learning. arXiv preprint vrXiv:2501.12944 2025.
[5] Yupeng Chang. Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
[6] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 10792, 2023.
[7] Haiyan Zhao, Hanjie Chen, Fan Yang. Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang. Dawei Yin, and Mengnan Du Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1-38, 2024.
[8] Lei Zhang, Yunshui Li, Jiaming Li, Xiaobo Xia, Jiaxi Yang, Run Luo, Minzheng Wang. Longze Chen, Junhao Liu, Qiang Qu, et al. Hierarchical context pruning: Optimizing real- world code completion with repository-level pretrained code lmms. In AAAI, pages 25886- 25894, 2025.
10

---

[9] Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling- Hao Chen, Junhao Liu, Tongliang Liu, al. One-shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312. 10302, 2023.
[10] Xiaohao Liu, Xiaobo Xia, Weixiang Zhao, Manyi Zhang, Xianzhi Yu, Xiu Su, Shuo Yang, See-Kiong Ng, and Tat-Seng Chua. L-mip: Leap multi-token prediction beyond adjacent context for large language models. In NeurlPS, 2025.
[11] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. EMNLP Findings, pages 15757-15773, 2023.
[12] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. Multimodal large language models: A survey. In BigDala, pages 2247-2256. IEEE, 2023.
[13] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever com- prehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint urXiv:245.210755. 2024.
[14] Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh, Qifan Wang, David Yang, Sheng Yun Peng, Hanchao Yu, Shen Yan, Xuewen Zhang, et al. Compcap: Im- proving multimodal large language models with composite captions. arXiv preprint arXiv:2412.05243, 2024.
[15] Qing Jiang, Gen Luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. Chatrex: Taming multimodal Ilm for joint perception and understanding. arXiv preprint arXiv:2411.1888 2024.
[16] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jac Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-ar multimodal models. arXiv preprint 1.Xiv:2499111166 2024.
[17] Jialong Zuo, Jiahao Hong, Feng Zhang, Changqian Yu, Hanyu Zhou, Changxin Gao, Nong Sang, and Jingdong Wang. Plip: Language-iman pre- pre-training for person representation learning. In NeurIPS, pages
2024.
[18] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2024.
[19] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[20] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large langunee model with ver- satile abilities. arXiv preprint arXiv:2308. /2966, 2023.
[21] Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-Fn En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405. /5232, 2024.
[22] Yuchen Zhou, Jiayu Tang, Xiaoyan Xiao, Yueyao Lin, Linkai Liu, Zipeng Guo, Hao Fei, Xi- aobo Xia, and Chao Gou. Where, what, why: Towards explainable driver attention prediction. In ICCV, 2025.
[23] Yuchen Zhou, Jiayu Tang, Shuo Yang, Xiaoyan Xiao, Yuqin Dai, Wenhao Yang, Chao Gou, Xiaobo Xia, and Tat-Seng Chua. Logic unseen: Revealing the logical blindspots of vision- language models. arXiv preprint arXiv:2508. 131 2025.
[24] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large- scale ladio-language models. arXiv preprint arXiv:2311.07919 2023.
11

---

[25] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.0666 2024.
[26] Ke-Han Lu. Zhehuai Chen, Szu- Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu- Yu-Chiang Frank Wang, and Hung-yi Lee. Developing instruction-followwe speech language model without speech instruction-tuning data. ICASSP, pages 1-5, 2025.
[27] Shengpeng Ji. Ziyue Jiang, Hanting Wang, Jialong Zuo, and Zhou Zhao. Mobilespeoch: A fast and high-fidelity framework for mobile zero-shot text-to-speech. In ACL, 2024.
[28] Haonan Zhang, Run Luo, Xiong Liu, Yuchuan Wu, Ting-En Lin, Pengpeng Zeng, Qiang Qu, Feiteng Fang, Min Yang. Lianli Gao, Jingkuan Song, Fei Huang, and Yongbin Li. OmniChar- acter: Towards immersive role-playing agents with seamless speech-language personality interaction. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Compula- lional Linguistics (Volume I: Long Papers), pages 2631 1-3331,,, Vienna, Austria, July 2025. Association for Computational Linguistics.
[29] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, and Xipeng Qiu. AnyGPT: Unified multimodal LLM with discrete sequence modeling. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, ACL, pages 9637-9662, 2024.
[30] Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406. 1o..24406.5774., 2024.
[31] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint arXiv:2408.052/1 2024.
[32] Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409. 18042,, 2024.
[33] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to- speech synthesizer based on supervised semantic tokens. corr, abs/2407.05407,, 2024. doi: 10.48550. arXiv preprint ARXIV.2407.05007
[34] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2024.
[35] Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-Ei En Lin, Fei Huang, Mingli Song, et al. A survey of direct preference optimization. arXiv preprint arXiv:2503.1 2025.
[36] Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, et al. Omnibench: Towards the future of universal omni- language models. arXiv preprint arXi::2409.15272 2024.
[37] Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 3, 2023.
[38] Yufei Wang. Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: survey. arXiv preprint 2307 /2966, 2023.
12

---

[39] Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommenda- tion. In WWW, page 60, 2024.
[40] Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang. Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. In ICLR, 2024.
[41] Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. Long context is not long all: A prospector of dene-deenndncc data for large language models. arXiv preprint arXiv:2405.17911 2024.
[42] Run Luo, Haonan Zhang, Longze Chen, Ting- En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng. Lianli Gao, et al. Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint arXiv:2409.05840 2024.
[43] Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang. Yinxu Pan, Qianyu Chen, Tianyu Yu, Hang- hao Wu, Yue Zhao, Haoye Zhang, et al. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint arXiv:2308. /2038, 2023.
[44] Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vision-language models. In NeurlPS, pages 3 3122-3156, 2024.
[45] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In ICML, pages 12888-12900. 2022.
[46] Run Luo, Renke Shan, Longze Chen, Ziqiang Liu, Lu Wang, Min Yang, and Xiaobo Xia. Vem: Vision concept modeling based on implicit contrastive learning with vision-language instruction finettunig. arXiv preprint arXiv:2504. 19627, 2025.
[47] Xiaobo Xia and Run Luo. Gui-rl: A generalist r1 rt-style vision-language action model for gui agents, arXiv preprint arXiv:2504. 10458, 2025.
[48] Jiayi Ji, Haowei Wang, Changli Wu, Yiwei Ma, Xiaoshuai Sun, and Rongrong Ji. Jm3d & jm3d-llm: Elevating 3d representation with joint multi-modal cues. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[49] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[50] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal compre- hension and creation. arXiv preprint arXiv:2309.
2023.
[51] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision languae models for vision tasks: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[52] Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. Benchmark evalu- ations, applications, and challenges of large vision language models: A survey, arXiv preprint arXiv:2501.02189, 2025.
[53] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of lasiuanlanguage pre- trained models. arXiv preprint 10X9 2022.
[54] Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C Knoll. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions on Intelligent Vehicles, 2024.
[55] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint 440: 00253, 2024.
13

---

[56] Sijing Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Yu Pan, Pengfei Tan, al. Takin: A cohort of superior quality zero-shot speech generation models. arXiv preprint arXiv:2409. 12139, 2024.
[57] Helin Wang, Meng Yu, Jiarui Hai, Chen Chen, Yuchen Hu, Rilin Chen, Najm Dehak, and Dong Yu. Ssr-speech: Towards stable, safe and robust zero-shot text-based speech editing and synthesis. In ICASSP, pages 1-5, 2025.
[58] Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to- speech with masked generative codec transformer. arXiv preprint arXiv:2409.0077 2024.
[59] Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.0000 0,656, 2024.
[60] Zhisheng Zheng, Puyuan Peng. Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat Learning to reason about spatial sounds with large language models. In ICML, 2024.
[61] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real- time dialogue.
[62] Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang. Siqi Zheng, Jiaqing Liu, Hai Yu, and Chaohong Tan, Omniflatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint arXiv:2410. 17799, 2024.
[63] Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. Duplex conversa- tion: Towards human-like interaction in spoken dialogue systems. In ACM SIGKDD, pages 3299-3308, 2022.
[64] Xiong Wang. Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni:
A smart and low latency speech-to-speech dialogue model with frozen Ilm. arXiv preprint WrXiv:2411.00774, 2024.
[65] Xiaoxue Gao. Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F Chen. Emo- dpo: Controllable emotional speech synthesis through direct preference optimization. arXiv preprint vxXi:2449.101577 2024.
[66] Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey. arXiv preprint arXiv:2410.03751, 2024.
[67] Run Luo, Xiaobo Xia, Lu Wang, Longze Chen, Renke Shan, Jing Luo, Min Yang, and Tat- Seng Chua. Next-omni: Towards any any-to-any omnimodal foundation models with discrete flow matching. arXiv preprint 25X01:25110.3721,, 2025.
[68] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, pages 26439-26455, 2024.
[69] Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o model with vision, speech and duplex. arXiv preprint arXiv:2410.11 2024.
[70] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.2022 2025.
[71] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng. Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
[72] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disen- tangled self-supervised representations. arXiv preprint
2021.
14

---

[73] Zhengrui Ma, Qingkai Fang, Shaolei Zhang, Shoutao Guo, Yang Feng, and Min Zhang. A non-autoregreesivv generation framework for end-to-end simultaneous speech-to-amy transla- tion. arXiv preprint arXiv:2406.0692 2024.
[74] Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. Streamspeech: Simultaneous speech-to-specch translation with multi-task learning. arXiv preprini 2rXiv:2406.088 2024.
[75] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369-376, 376, 2006.
[76] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. In NeurlPS, pages 27730-27744, 2022.
[77] 6seconds.org. Plutchik-model-enan -emotioo.o...... wewiikk-wweeeemmmtee
6swconds.cddd..ooo http://wwwc.oo....//////////////
2022.
[78] Binbin Zhang, Hang Lv, Pengeheng Guo, Qijie Shao, Chao Yang. Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP, pages 6182-6188 2022.
[79] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, pages 5206-5210, 2015.
[80] Yihui Fu, Luyao Cheng, Shubo Lv, Yukai Jv, Yuxiang Kong, Zhuo Chen, Yanxin Hu, Lei Xie, Jian Wu, Hui Bu, et al. Aishell-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario. arXiv preprint arXiv:2104.03603, 2021.
[81] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[82] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, pages 28492-28518, 2023.
[83] Aohan Zeng, Zhengxiao Du, Dong, and lie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:24/2.02611, 2024.
D Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao
[84] Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, et al. Baichuan-omni technical report. arXiv preprint arXiv:2410.08565 2024.
[85] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm your phone. arXiv preprint arXiv:2408.0I4.
2024.
[86] Shengbang Tong, Ellis Brown, Penghao WU, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1ms.. arXiv preprint arXiv:2406. 16860, 2024.
[87] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint 230.::2220....111, 2023.
[88] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- language models? arXiv preprint arXiv:2403.20338
2024.
[89] x.ai. Grok- 1.5 vision preview. https: / http//xxxii//l/// wwwpaaagggoooo///aaeeeee 2024.
15

---

[90] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In CVPR, pages 9556-9567, 2024.
[91] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chumyuan Li, Hanmanch Hajishirzi, Hao Cheng. Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math rea- soning in visual contexts with gpt-4v, bard, and other large multimodal models. arXiv e- prints, pages arXiv-2310 2023.
[92] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235-251, 2016.
[93] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models, 2023.
[94] Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong. Yuhang Zang, Pan Zhang, Jiaqi Wang, al. VImevalkit: An open-source toolkit for evaluat- ing large multi- modllityd models. In ACM MM, pages 11198- 11201, 2024.
[95] Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021.
[96] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generie hearing abilities for large language models. arXiv preprint arXiv:2310.13289. 2023.
[97] Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation, arXiv preprint arxiv:2312.15188. 2023.
[98] Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, Hongquan Lin. Jiaming Li, Yuansheng Ni, Haihong Wu, Yaswanth Narsupalli, Zhigang Zheng, Chengming Li, Xiping Hu, Ruifeng Xu, Xiaojun Chen, Min Yang, Jiaheng Liu, Ruibo Liu, Wenhao Huang, Ge Zhang, and Shiwen Ni. li-bench: An image implication understanding benchmark for multimodal large language models, 2024.
[99] Kaixiong Gong, Kaituo Feng. Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang. Yutong Bai, Zhuoran Yang, al. Av-odyssey bench: Can your multi- modal lims really understand audio-visue information? arXiv preprint 1rXiv:2412.02611, 2024.
[100] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In CVPR, pages 26584-26595, 2024.
[101] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model instruction-follow them all. arXiv preprint arXiv:2305. /6355, 2023.
[102] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding, arXiv preprint arXiv:2306.0288 02858, 2023.
[103] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476, 2024.
Videollama 2: Advancing spatial-temporal
[104] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 1lm. In JCML, 2024.
16

---

A Data Construction
We provide details of the data construction for multiple training stages below:
OpenOmni-I-I: In addition to datasets WeNetSpeech, LibriSpeech, and AIShell-4, we randomly select 80K image-text instruction data with shorter responses from MMEvol [42]. We translate 40K of this data into Chinese using Qwen72B and synthesize the responses into speech data with Vosve This results in 1,600 hours of OpenOmni-1-1 data for speech-text alignment pretraining.
OpenOmni-2-11 For rapid image-text alignment pretraining, we the llava pretrain dataset, following previous work [18, 19, 42, 98].
OpenOmmi-2-2: To achieve efficient image-text instruction tuning, employ MMEvol data. Since we later train the speech decoder by freezing the LLM mode, we include O2S- 300K stabilize the training the speech decoder, leading combined dataset 1.7M for OpenOmni-222.
OpenOmmi-3-11 better utilize computational resources, we select 300K data with long response instructions from MMEvol and UltraChat. This includes 100K image-text instruc- tion data, 100K single-round dialogue, and 100K multi-round dialogue. We synthesize the corresponding speech using Cos Voice, resulting in 8,000 hours of O2S-300K.
OpenOmni-3-22 We curate 9K emotion preference data and generate emotional speech preference pairs using Voice's conditional control. This is used for emotional speech direct preference optimization.
B Additional Experiments
B.1 Additional Omni-Language Evaluation
In addition to OmniBench, we conduct experiments on the AV-Odyssey Bench [99]. which involves the four modalities: audio, text, image, and video. For video. we test by averaging 8 sampled frames into a single image. The experimental results are shown in Table below. Compared to other OLLMs, OpenOmni achieves the best average performance using only bi-modal speech-text and image-text data. With 7B model parameters and no audio or video training, it outperforms VITA by 4.4 points, demonstrating the effectiveness and efficiency of OpenOmni.
Table 5: Overall omni-understanding results on AV-Odyssey Bench. In each case, the best result indicated bold. We conduct performance comparison of omni-understanding among various fully open-source Omnimodal Large Language Models (OLLMs) on AV-Odyssey Bench. Compared the state-of-the of-the-art OLLM, VITA [31], which was trained on tri tri-modal data, OpenOmni achieves comparable advanced performance using significantly less training data and smaller model size.
Method
Timbre
Tone
Melody
Space
Time
Hall
Intricacy
Overall
OneLLM (7B) [100]
25.0
25.5
21.5
37.5
29.3
25.5
38.4
27.4
PandaGPT (7B) [101]
23.5
23.2
27.6
45.0
23.8
28.0
26.7
Video-LLaMA (7B) [102]
25.5
22.3
24.4
30.0
26.2
25.0
26.1
Video-LLaMA2(7BB [103]
24.1
25.5
26.4
30.0
27.2
33.0
34.5
26.8
AnyGPT (7B) [29]
24.6
25.0
26.4
27.5
29.2
29.0
25.7
26.1
NexTGPT (7B) [104]
23.3
20.9
27.8
30.0
28.8
28.5
23.6
25.5
VITA 87 [31]
24.1
26.4
27.8
22.5
26.3
31.0
36.8
26.4
OpenOmni (7B)
23.9
27.7
25.9
60.0
25.2
29.5
37.6
32.8
C Additional Ablation Studies
On TGM. To explore the effect of TGM on speech generation in two modes, we plot the change of training loss under the same setting. As shown in Figure 4, we can observe that TGM can signifi- cantly improve the convergence speed of training and the performance of model speech generation,
17

---

Smoothed Loss vs Step (Outliers Replaced)
Files
18
AR Mode w/e TGM
AR Mode w/ TGM
NAR Mode w/o TGM
16
NAR Mode w/ TGM
14
12
Loss
250
750
1250
1750
Step
Figure 4: Ablation study of the text-guided module (TGM), In order to explore the effect of TGM on speech generation under the two modes, we plot the change training loss under the same setting. TGM can significantly improve the convergence speed of training and improve the effect of speech generation of the speech decoder.
Table 6: Ablation study on the number of layers and experts in the speech decoder. Increasing experts the mixture of experts module stabilizes the CTC loss during training and enhances speech generation capacity. Deeper transformer layers improve English and Chinese speech generation, with greater benefits for Chinese.
Wenelspeech(ZH)
Librispeech(EN)
Layers
Experts
Test Net
Test Meeting
Test clean
Test other
113.6
129.7
87.8
96.5
2
16.7
22.3
10.7
14.6
2
4
8.5
8.4
4.2
4.7
4
7.3
7.9
3.8
4.3
6.7
Table 7: Ablation study of the model training image-text alignment and speech-text align- ment stages. The speech and text have clear temporal correspondence, enabling low-cost alignment. In contrast, the image-text gap larger, requiring LLM fine-tuning for better results.
Stage
LLM freeze
GPUxHour
MMStar
MathVista"
MMMU
AI2D
imago-text
76
41.2
42,3
35,5
54.3
image-text
Stage
LLM freeze
GPUxHour
AlShell Dev
AIShell 2estst
LibriSpeech Clean LibriSpeech Test
speech-text
12.7
13.3
image-ter text
192
13.1
11.8
10.1
13.5
speech-text
12.2
11.1
12.8
which verifies the effectiveness of our model design, whether next-token-prediccion (NTP) loss under the stable AR mode or CTC loss under the unstable NAR mode.
On the number layers and experts the speech decoder. explore the impact of the number of layers in the NAR speech decoder and the MoE module on Chinese and English speech genera-
18

---

Table 8: Ablation study of the alignment order and joint training strategy. The order of the alignment strategies has minimal impact on the final performance. Compared to joint training, the stat alignment strategy not only significantly reduces memory requirements during training but also ensures competitive results, making it the most efficient and optimal training strategy under low-resource conditions.
Order
First
Joint Training
MMStar
HallBench
MathVista
MMMU
A12D
KWQA
AlShell-2
Test
Test-Clean
LihriSpeach Test-Other
image text
x8GBx8
44.7
35.9
47.1
40.7
58.6
60.1
13.4
11.3
10.4
13.6
speech
40GB 8
44.4
36.7
47.6
40.2
59.1
55,9
12.7
11.5
9.8
13.3
specch--eer
90GB
44.9
40.6
39.6
12.4
11.1
tion, we conduct ablation experiments on WeNetSpeech [78] and LibriSpeech [79]. As illustrated in Table 6, the instability and fragility associated with training using the CTC loss function present significant challenges. When simply employing a single feed-forward network (i.e., the number of experts is 1), it becomes increasingly difficult to reconcile the conflicting training dynamics in- herent in laxeudlanguage scenarios, particularly when dealing with varying response lengths. As a result, training the speech decoder under these conditions proves to be quite challenging. Our findings demonstrate that incrementally increasing the number of experts significantly enhances the model's s performance in bilingual speech generation, thereby underscoring the effectiveness of our MoE module design. However, we observe inconsistent preferences regarding the optimal number of layers in the speech decoder for generating speech in Chinese and English. Specifically, while four layers yield the best results for English generation, six layers are more suitable for generating Chinese speech.
LLM training in image-text alignment and speech-text alignment stages. To investigate whether training large language models affects modality alignment at different stages, we conduct an ablation study. As shown in Table 7, since speech and text data naturally have temporal alignment relation- ship, freezing the LLM during alignment training still achieves competitive alignment performance. However, the gap between image and text modalities is significantly larger, and better alignment results are only achieved by unfreezing the LLM during training. Furthermore, we find that even after image-text training, there no catastrophic forgetting of knowledge related to speech-text alignment. This validates the effectiveness and efficiency of our progressive alignment method.
Alignment order and joint training strategy. We conduct ablation studies to explore the impact of multi-stage alignment order and joint training strategies. By using 20K speech data and 500K image data, as shown in Table 8, we observe that the relative order of speech-text alignment and image-text alignment has little effect on the final performance, which indicates low correlation between the two stages.
Due to limitations in data and computational resources, we adopt multi-stage progressive multi- modal alignment strategy complete the omnimodal alignment training. At any given stage, only two modalities of data are processed simultaneously. This method not only alleviates the challenges posed by missing tri-modal data but also significantly reduces computational memory requirements. With fewer computational resources and less training data, our method achieves superior omnimodal alignment results compared to existing approaches.
As shown in Table 8, it can be observed that multi-stage training requires only 40GB x of VRAM, which is significantly lower than the memory demands of joint training. At the same time, it achieves comparable results, making it more efficient and practical choice in desource-constrainge scenarios.
D Additional Implementation Details
OpenOmni is trained in five sequential sub-stages. Further details on these training stages are pro- vided in Table 9.
Besides, as shown in Figure 5, we provide more details of the speech decoder design and training here. For the speech decoder, OpenOmni supports both autoregressive (AR) and noneeuvoreeeessive (NAR) methods. Specifically, the AR mode has better generation quality but slower generation
19

---

batch size omrimodal hidcen state
Image speech
CTC Loss
Text- Cuidd-Gudddda Module
omnimodal
Speech Decoder
speech
Speech Decoder
(TGM
FFN
XN
Cross ttentonn
Layer
mni Language Model
KV
AR Mode
Loss
omnimodal
speech
Speech Decoder
Encoder
Speec Encoder
Linea Laver
Figure 5: Overview of text-guided module and speech decoder mode. (Left) Text-guided mod- ule fuses the hidden state and response textual feature via cross-attention, accelerating convergence speed of training without dropping the speed speech decoding and context emotion perception. (Right) OpenOmni supports both autoregressive (AR) and non-autoregressive speech (NAR) gen- eration. The NAR mode uses the CTC loss modeling and 6K speech vocabulary size to enable real-time parallel speech decoding generation. The AR mode speech vocabulary size of 16K to support streaming decoding and higher-qualits speech generation. To make the training of the speech generator more stable, we design text-guided output feature fusion method to ensure the correctness of semantic alignment in speech generation modeling.
the NTP loss modeling and
Table 9:
The detailed training setup for OpenOmni and the hyperparameters across the training stage. All experiments conducted 8XNVIDIA A100-80G GPUs. The dataset index can be checked in Appendix A.
Hyperparameter
II
III
IV
V
256 1 10
128
128
32 x
32 5
1 10 0.3
x 10-6
10 0.3
warmup ratio epoch freeze LLM optimizer cost
0.3
0.3
0.3
1
3
3
AdamW
AdamW 80 GPU Hours 2-1 Ee
Adam W 500 GPU Hours 2-2 C
Adam
AdamW W
40 GPU Hours 1-1
36 GPU Hours 3-1
GPU Hours 3-2 LeTc.RRO
dataset
Cet
Lere
speed, while the NAR mode can achieve real-time speech generation, but the generation quality is slightly worse. At the same time, in order to train the speech generator more efficiently, we also design a text-guided feature fusion module, so that the conditional features used for speech generation have more accurate alignment semantics, which can improve the generation quality and training efficiency of the speech decoder.
NAR mode. In the NAR mode, the conditional features generated by OLLM are fed into the speech decoder by layer of MoE and then upsampled to obtain the predicted speech output, and finally, the end-to-end optimization is carried out by the CTC loss modeling of the speech output. Due to the instability of CTC loss training, the smaller the size of the speech vocabulary, the easier it is to be successfully trained, but the generation quality of the corresponding speech will be affected by smaller vocabulary.
AR mode. The AR mode projects the conditional features generated by OLLM into the speech space through layer of linear layer and feeds them into the speech decoder to obtain the speech prediction output, and finally optimizes the speech output end-to-end by modeling the NTP loss. Due to the stability of NTP loss training, the quality of speech generation will higher than that of NAR generation, but the speed of speech generation will be reduced by AR decoding.
20

---

Note that both AR and NAR modes depend on the quality of the speech generation conditional fea- tures generated by OLLM. Although OpenOmni will let the OLLM fit the text answer corresponding to the speech through multiple rounds of training in advance, there will still be OLLM outputs de- coded into the wrong text answer. this case, the erroneously generated condition features will be incorrectly aligned with the speech during the training process, which will ultimately reduce the per- formance of the speech decoder. To ensure the efficiency of training, OpenOmni fuses the speech generation condition features output by OLLM with the corresponding text features with correct semantics, and then feeds them into the speech decoder for speech generation modeling training. Through the feature fusion module of text prior, OpenOmni avoids the misalignment of speech and corresponding text and ultimately makes the speech decoder training more stable. At the same time, it enjoys more efficient and accurate speech generation quality.
E
Broader Impacts
OpenOmni marks a significant advancement open-source omnimodal large language models (OLLMs), seamlessly integrating vision, language, and speech into unified framework. Its open- source nature fosters transparency, community-driven innovation, and trust in AI technologies. However, challenges remain, including emotional manipulation and privacy concerns. To address these, we emphasize responsible Al practices and secure handling of speech data.
F
Limitation
Due to resource limitations, our method primarily focuses on Chinese and English and not been trained validated on multilingual data. In the future, we plan to utilize multilingual speech data to activate the multilingual capabilities of the large language model, thereby enhancing its applicability across diverse scenarios.
G Prompt Template
We provide the specific prompts used in our tasks, as well as examples of synthesized multi-turn emotional speech data, to help readers better understand this work.
English Prompts used in Text-to-Speech (T2S) and Speech-to-Text (S2T) tasks.
S2T (ASR)
Please translate the user's spoken input into corresponding text, word by word.
T2S
Echo game: Repeat this sentence back to me exactly as I said it!
Mirror challenge: Reflect my words right back me!
Parrot mode: Can you mimic what just said?
Copycat test: Let's see how well you can replicate my sentence!
Playback time: Rewind and repeat my last sentence.
Echo chamber: Say back me what just mentioned.
Simulate my speech: Can you say exactly what just did?
Mimicry game: Echo my words precisely!
Repeat relay: Send my sentence back to me unchanged.
Reflection exercise: Reiterate my last statement word for word.
21

---

English emotional preference data
Emotion: Neutral
user:
"Please engage in a conversation with me."
assistant: "Of course, please go ahead with your question." " [A calm and collected woman, speaking in steady and neutral tone.] user: "I wondering how you manage stay so composed all the time." assistant: "It's something that comes with practice and mindfulness. I focus on observing my surroundings and maintaining a balanced perspective." [A composed woman, speaking in measured and tranquil tone.]
user: "That sounds quite challenging. Do you ever feel overwhelmed?"
assistant: "There are moments when things can be overwhelming, but remind myself to breathe and take one step at time." " [A calm woman, delivering her words an even and serene manner.]
user: "Do you have any techniques to stay calm under pressure?"
assistant: "Yes, I practice meditation and deep breathing exercises. It helps to center my thoughts and maintain clarity." [A composed woman, speaking with a steady and soothing tone.]
user: "I see. Do you think anyone can learn to be as calm as you?"
assistant: "Absolutely. It requires patience and consistent practice, but anyone develop the ability to remain calm and composed." [A neutral-toned woman, expressing herself clear and composed manner.]
user: "What advice would you give to someone starting on this path?
assistant: "Begin with small steps. Start by observing your thoughts and emotions with- out judgment. Practice mindfulness and breathing exercises daily." [A composed woman, offering her advice in gentle and calm tone.]
user: "Thank you for your advice. It has been very insightful."
assistant: "You't re welcome. I'm glad I could help. Remember, the journey to calmness gradual, but rewarding." [A tranquil woman, speaking in a reassuring and composed tone.]
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the contributions and scope in both the abstract and introduction, supported by series of justifications.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]-
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide extensive experimental details the main content (see Section 4) and appendix (see Appendix D).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the full instructions to access publicly available data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nip. wttp:/ccpccccccgiieeeeeeeo Sttpii/guuiieee/eeuubbaaeeeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We specify all the training and test details in Section 4, Appendix A, and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Following previous works, one trial was performed in each experimental case due to time consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide such implementation details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http:///eeipppeeeeeeeeee hwtp///bl.ccttceccccieeeeeeeeeee
Answer: [Yes]
Justification:
We adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impact statement in Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper involves only publicly available datasets and models. No specific safeguards were necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the relevant papers and provide links to existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswit wwwwwwttehoee...eee www dataatts has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
This paper does not introduce new assets. Therefore, no documentation is applicable.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects. Therefore, the inclusion of participant instructions, screenshots, and compensation details is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
28

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper does not involve LLMs for any important, original, standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https:/ htapp:///neeee //tp:rips..p.//eeee cc//on/feeencce/2225////////// for what should or should not be described.
29


Reviews:
[{"review_id": "bNt5aaaEck", "reviewer": "Reviewer_LJos", "summary": "This paper addresses two key issues currently present in the community: the lack of high-quality omnimodal data and the suboptimal performance of real-time emotional speech synthesis. To tackle these challenges, the authors propose OpenOmni, a comprehensive omnimodal large Languagemodel framework equipped with Real-time Emotional Speech Synthesis capabilities. First, this paper inroduces two high-quality speech datasets—O2S-300K and EO2S-9K—to support efficient speech generation and emotional preference optimization. Then, they introduce a modality-progression training framework to achieve efficient tri-modal alignment. Experimentally, OpenOmni achieves state-of-the-art (SOTA) performance across multiple benchmarks, while also demonstrating significant improvements in training efficiency.", "questions": "1. In Stage 2 of the framework, which focuses on Image-Text alignment, the training involves fine-tuning the LLM. A potential concern is whether this fine-tuning could undermine the Speech-Text alignment achieved during Stage 1.\n\n2. Table 4 shows that the improvement from DPO training on Chinese evaluation is significantly more noticeable than on English. Is this primarily due to the data distribution or the model's pre-training?\n\n3. For the choice of speech decoder, the paper states that AR and NAR decoder differ in generation quality and speed. Please provide quantitative experimental results for both and analyze them.", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. High-quality omnimodal data and omnimodal alignment remain key challenges in current research on omnimodal large models. This paper tackles these two critical issues and presents effective solutions, underscoring its considerable significance to the field.\n\n2. The proposed framework is notably efficient and streamlined. By utilizing a single large language model (LLM) and a lightweight speech decoder within a multi-stage training strategy, it delivers substantial improvements in both generation efficiency and model size, outperforming the leading open-source model, VITA.\n\n3. The paper provides a clear and detailed description of the model architecture and training strategy.\n\n**Weaknesses:**\n\n1. The paper considers dataset construction as one of its contributions; however, the description of the process lacks sufficient detail and fails to clearly demonstrate the advantages of the newly constructed datasets.\n\n2. The paper lacks comparison and discussion with other methods. For example, it does not clearly outline the improvements and differences in architecture and training strategy between OpenOmni and VITA.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1**: The paper considers dataset construction as one of its contributions; however, the description of the process lacks sufficient detail and fails to clearly demonstrate the advantages of the newly constructed datasets.\n\n**Response:**\n\nThank you for the reviewer’s comments. In Section 4.1 of the main text, we provided a detailed explanation of our data sources and construction processes, with a brief introduction in Appendix A. Here, we further clarify and highlight the methodologies and advantages of our datasets.\n\nWe sampled **300K multi-turn dialogues** from **UltralChat** (text) and **MMEvol** (multimodal), selecting longer responses to ensure extended audio durations for better model learning. Half of the data was translated into Chinese to support bilingual generation. Using **CosyVoice**, speech synthesis was performed on the text data to create **O2S-300K**.  \nFor **EO2S-9K**, we intensively filtered dialogues with contextual consistency and strong emotional tendencies in multi-turn interactions. Emotional speech synthesis was conducted using **CosyVoice**, resulting in a dataset optimized for DPO-based emotional speech generation. A total of **8,000 A100 GPU hours** were spent synthesizing O2S-300K and EO2S-9K, with filtering, translation, and selection costing approximately **$2,000**. These efforts ensure diverse, high-quality datasets.\n\n**Advantages**:  \n\n1. **O2S-300K**: It is the **bilingual multi-turn dataset** with tri-modal inputs (text, audio, vision). Unlike prior works like **LLaMA-Omni**, which use single-turn English-only voice data, O2S-300K enables building **multi-turn bilingual interaction models**, supporting **arbitrary input combinations** for enhanced contextual consistency and interaction experience.  \n\n2. **EO2S-9K**: It provides **fine-grained emotional speech annotations** with contextual consistency, allowing adaptive emotional speech generation after training for superior multi-turn interactions. Previous datasets rely on pre-defined emotion tags in prompts, often leading to inconsistencies between emotion tags and text (e.g., “I feel happy [sad]”), limiting adaptive learning and natural dialogue engagement.  \n\nThese datasets address key limitations of prior works, advancing bilingual, multi-turn, and emotionally adaptive speech synthesis. If further clarification is needed, we are happy to address additional questions.\n\n> **W2**: The paper lacks comparison and discussion with other methods. For example, it does not clearly outline the improvements and differences in architecture and training strategy between OpenOmni and VITA.\n\n**Response:**\n\nThank you for the thoughtful feedback. We would like to clarify the key architectural and training differences between OpenOmni and VITA.\n\n1. **Architectural Differences**:\n   - **VITA** uses a pipeline-based speech generation method, where text responses are passed to a separate TTS model. To speed up synthesis, it employs a duplex communication strategy that requires two simultaneous model deployments, doubling GPU memory usage. Its non-end-to-end design limits multi-turn dialogue context utilization, making it less effective at capturing emotional consistency.\n   - **OpenOmni** features an **end-to-end speech generation design**, where LLM outputs are directly fed into a CTC-based speech decoder for real-time parallel decoding. This approach supports context-aware emotional speech generation and ensures consistency across multi-turn interactions.\n2. **Training Differences**:\n   - **VITA** employs **large-scale joint training**, needing **three times more tri-modal data** (5M vs. 1.7M), **80GB peak GPU memory**, and nearly **10,000 GPU Hours**, demanding significantly larger computational resources.\n   - **OpenOmni** adopts a **progressive multi-modal alignment strategy**, reducing reliance on tri-modal datasets while enhancing efficiency. It requires only **40GB peak GPU memory** and **1,400 GPU Hours**.\n\n\nIn summary, OpenOmni is **faster, more efficient, and context-aware**, overcoming VITA’s limitations in emotional speech generation and resource consumption, while enabling smoother and more effective multi-turn interactions.\n\n> **Q1**: In Stage 2 of the framework, which focuses on Image-Text alignment, the training involves fine-tuning the LLM. A potential concern is whether this fine-tuning could undermine the Speech-Text alignment achieved during Stage 1.\n\n**Response:**\n\nThank you for the reviewer’s thoughtful feedback. In fact, we have conducted relevant experiments in the ablation study presented in Table 7, and provided detailed discussions in lines 669-676 of the paper. \n\nThe results demonstrate that due to the natural temporal alignment between speech and text, we choose to freeze the LLM during the speech-text alignment stage, while opting to unfreeze the LLM during the image-text alignment stage. This dynamic alignment strategy effectively balances efficiency and performance trade-offs.\n\nAdditionally, we observe that unfreezing the LLM during the image-text alignment stage has only a minimal impact on the speech-text alignment knowledge from the first stage. Specifically, it results in a very minor performance decline on AI-Shell (2.8%, 12.10 vs. 12.45) and LibriSpeech (2.1%, 11.55 vs. 11.8) evaluation datasets, without causing catastrophic forgetting. \n\nBy employing such a progressive alignment strategy, we achieve a nearly 300% improvement in training efficiency, making it a highly practical approach. For these reasons, we adopt this method in our proposed framework.\n\n> **Q2**: Table 4 shows that the improvement from DPO training on Chinese evaluation is significantly more noticeable than on English. Is this primarily due to the data distribution or the model's pre-training?\n\n**Response:**\n\nWe believe this may be related to the varying distribution of emotion-themed dialogue data during pretraining. To investigate, we randomly sampled 10K emotional speech data from the pretraining dataset and annotated it using GPT4O-mini for emotion labeling. The distribution of different emotions in the data is shown below:\n\n| Lang | Angry & Disgusted | Fearful | Happy | Neutral | Other | Sad  | Surprised |\n| ---- | ----------------- | ------- | ----- | ------- | ----- | ---- | --------- |\n| ZH   | 0.08              | 0.10    | 0.03  | 0.63    | 0.05  | 0.09 | 0.02      |\n| EN   | 0.05              | 0.04    | 0.07  | 0.72    | 0.04  | 0.05 | 0.03      |\n\nFrom the table, it is evident that the differing proportions of emotions in the pretraining data contribute to the observed performance. Emotions with a higher proportion in the pretraining dataset are more likely to be effectively triggered during subsequent DPO training, thereby leading to a significant improvement in emotional speech generation capabilities.\n\n> **Q3**:For the choice of speech decoder, the paper states that AR and NAR decoder differ in generation quality and speed. Please provide quantitative experimental results for both and analyze them.\n\n**Response:**\n\nThank you for the reviewer’s valuable comments. Here, we provide more detailed quantitative experimental results comparing AR (Autoregressive) and NAR (Non-Autoregressive) modes, along with their corresponding analysis.  \n\nIn addition to the baseline metrics for semantic consistency of speech content, such as WER (Word Error Rate) and CER (Character Error Rate), we also employed **10 expert evaluators** to assess the MOS (Mean Opinion Score) metric. The MOS metric provides a comprehensive evaluation of speech naturalness and emotional congruence.  \n\nThrough this detailed comparison, we aim to better illustrate the strengths and weaknesses of both modes and their applicability to various use cases.\n\n| mode         | Speed | GPU Memory | AI-Shell2-Dev (CER)$\\downarrow$ | AI-Shell2-Dev (MOS)$\\uparrow$ | LibriSpeech-Test-clean (WER)$\\downarrow$ | LibriSpeech-Test-clean (MOS)$\\uparrow$ |\n| ------------ | ----- | ---------- | ------------------------------- | ----------------------------- | ---------------------------------------- | -------------------------------------- |\n| ground_truth |       |            | 3.4                             | 4.50                          | 1.6                                      | 4.50                                   |\n| AR           | 1.8s  | 18654MB    | 6.9                             | 3.89                          | 2.3                                      | 4.22                                   |\n| NAR (CTC)    | 226ms | 12786MB    | 7.3                             | 3.83                          | 2.6                                      | 4.16                                   |\n\nAs shown in the table, OpenOmni's NAR mode enables real-time speech synthesis with lower GPU memory usage, while still producing high-quality and fluent synthesized speech as confirmed by human evaluations.  \n\nIn terms of generation quality, the AR mode achieves slightly higher performance but comes at the cost of significantly greater computational requirements and GPU memory consumption. Additionally, the autoregressive nature of the AR mode limits its ability to perform real-time speech generation. In contrast, the NAR mode strikes a balance between speech generation quality and efficiency, making it a practical trade-off solution for widespread adoption.  \n\nTo support the open-source community and democratize OmniModal research, OpenOmni adopts the most efficient approach by integrating a progressive omnimodal alignment strategy combined with the NAR-mode speech decoder. This choice ensures that the community can participate in cutting-edge research with minimal computational and data requirements, ultimately advancing the field together."}, {"review_id": "4Yts4YiXFR", "reviewer": "Reviewer_TCBG", "summary": "This paper introduces a novel two-stage training framework to address key limitations in open-source omnimodal large language models, specifically the scarcity of high-quality tri-modal datasets and challenges in real-time emotional speech synthesis. The framework first aligns speech-text and image-text modalities independently using a progressive alignment strategy, enabling near-zero-shot generalization from vision to speech without tri-modal data. It then trains a lightweight non-autoregressive speech decoder optimized via DPO for real-time, emotionally coherent speech generation. Evaluations across omnimodal, vision-language, and speech-language benchmarks show that OpenOmni outperforms SOTA models like VITA while using 5× fewer training examples and a smaller model size. It achieves sub-second latency and a 7.7% improvement in emotion classification accuracy.", "questions": "The two-stage training procedure is well elaborated in the paper and the appendix. However, I'm still curious how the authors dealt with catastrophic forgetting?", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The authors proposed a smaller model with fewer training samples to achieve SOTA performance in omnimodal learning tasks, which is of great value to the academic community.\n2. The authors found a quasi-zero-shot transfer capability in the proposed model, which is insightful and has the value of further research.\n3. The paper is well organized, and the design choices are elaborated in detail with thorough verification. \n\n**Weaknesses**\n1. The authors adopted CTC loss to train the speech decoder instead of the next-token-prediction paradigm, which is more commonly used, such as Kimi-Audio [1] and Moshi [2]. Please clarify the decision. \n2. The authors introduced the training procedures as two stages (line 51), but the elaborations indicate three stages (Figure 2, Appendix A). I understand that the authors intended to show that the omnimodal learning capability is trained with the first two stages, and the third stage is for speech generation. However, the clarification needs improving.\n3. Table 1 shows that the proposed model underperforms significantly in terms of text&symbols, count&quantity. Please analyze and discuss the reasons. \n\n[1] Kimi-Audio Technical Report. https://arxiv.org/abs/2504.18425.\n[2] Moshi: a speech-text foundation model for real-time dialogue. https://arxiv.org/abs/2410.00037", "was_revised": true, "final_justification": "The authors have provided supplementary experiments and explanations regarding the choice of CTC, and I can understand that they chose CTC for lower latency and less computational load. However, I would prefer to optimize AR inference using a mature inference framework rather than choosing CTC, which has poorer performance. The authors have further elaborated on the phenomenon that text&symbols and count&quantity have poorer performance, and also provided a detailed explanation of the training process. I think the overall paper is interesting, despite some unclear writing.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1**: The authors adopted CTC loss to train the speech decoder instead of the next-token-prediction paradigm, which is more commonly used, such as Kimi-Audio [1] and Moshi [2]. Please clarify the decision.\n\n**Response:**\n\nThank you for the reviewer’s detailed comments. To further explain our choice of using the CTC decoding approach, we conducted the following experiments for comparison.  \n\nIn addition to the basic metrics for semantic consistency of speech content, such as WER (Word Error Rate) and CER (Character Error Rate), we also employed 10 expert evaluators to assess the MOS (Mean Opinion Score), which evaluates speech naturalness and emotional congruence.  \n\nThese experiments provide a comprehensive evaluation framework to highlight the advantages of the CTC decoding approach.\n\n| mode         | Speed | GPU Memory | AI-Shell2-Dev (CER)$\\downarrow$ | AI-Shell2-Dev (MOS)$\\uparrow$ | LibriSpeech-Test-clean (WER)$\\downarrow$ | LibriSpeech-Test-clean (MOS)$\\uparrow$ |\n| ------------ | ----- | ---------- | ------------------------------- | ----------------------------- | ---------------------------------------- | -------------------------------------- |\n| ground_truth |       |            | 3.4                             | 4.50                          | 1.6                                      | 4.50                                   |\n| AR           | 1.8s  | 18654MB    | 6.9                             | 3.89                          | 2.3                                      | 4.22                                   |\n| NAR (CTC)    | 226ms | 12786MB    | 7.3                             | 3.83                          | 2.6                                      | 4.16                                   |\n\nAs shown in the table, the CTC speech decoder enhanced by the TGM module maintains real-time generation capabilities while using less GPU memory. Compared to autoregressive speech decoders, it produces competitive, fluent, and natural speech with high MOS scores evaluated by human experts. This supports our decision to adopt the CTC speech decoder to improve real-time interaction experiences.\n\n> **W2**: the clarification needs improving.\n\n**Response:**\n\nThank you for the reviewer’s careful and thoughtful feedback, which has helped improve the readability of our work. We intend to clarify that the omnimodal learning capability is developed through the first two stages, while the third stage is dedicated to speech generation. These clarifications will significantly enhance the clarity of our paper, and we are committed to implementing these changes thoughtfully in the revised version. Thank you again for your constructive suggestions.\n\n> **W3**: Table 1 shows that the proposed model underperforms significantly in terms of text&symbols, count&quantity. Please analyze and discuss the reasons.\n\n**Response:**\n\nThank you for the reviewer’s careful feedback. We hypothesize that this performance discrepancy might stem from OpenOmni’s reliance on the image data distribution of **MMEvol**, which contains relatively few samples for **text & symbol** and **count & quantity** tasks.  \n\nTo validate this hypothesis, we sampled 50K text & symbol data from the Video-SALMONN training dataset and 50K count & quantity data from the Pixmo-Count dataset. These samples were mixed with the original 1.7M instruction data, forming a 1.8M instruction dataset for use in OpenOmni's second-stage alignment training. The experimental results are reported below.\n\n| Data | Action Activity | Story Description | Plot Inference | Identification Description | Contextual  Environmental | Identity Relationship | Text  Symbols | Count  Quantity |\n| ---- | --------------- | ----------------- | -------------- | -------------------------- | ------------------------- | --------------------- | ------------- | --------------- |\n| 1.7M | 36.65           | 45.65             | 32.91          | 44.08                      | 48.23                     | 34.38                 | 24.00         | 33.33           |\n| 1.8M | 37.12           | 45.82             | 33.63          | 44.32                      | 48.47                     | 34.85                 | **49.64**     | **50.89**       |\n\nAs shown in the table, after incorporating the additional training data specifically targeting text & symbol and count & quantity, OpenOmni demonstrates significant performance improvements in these two capabilities.\n\n> **Q1**: The two-stage training procedure is well elaborated in the paper and the appendix. However, I'm still curious how the authors dealt with catastrophic forgetting?\n\n**Response:**\n\nThank you for the reviewer’s valuable comments. We are happy to further elaborate on our strategies for addressing catastrophic forgetting during the multi-stage progressive omnimodal learning process.  \n\nDuring the text-speech alignment stage, we adopt a **shallow alignment strategy** by freezing the LLM during training. This approach ensures that catastrophic forgetting does not occur.  \n\nIn contrast, during the image-text alignment stage, we adopt a **deep alignment strategy** by unfreezing the LLM for training. While this improves image-text alignment capabilities, it can influence the original knowledge of the LLM and potentially disrupt the text-speech alignment knowledge established in the first stage, leading to catastrophic forgetting. To mitigate catastrophic forgetting for the LLM, we employ a mature approach inspired by MMEvol and LLaVA, mixing 143K high-quality textual instruction data from WizardLM into the training process. This strategy effectively alleviates catastrophic forgetting in the LLM.  \n\nAs demonstrated in our ablation study in Appendix Table 7, the impact of the image-text alignment stage on the knowledge from the text-speech alignment stage is minimal. Specifically, unfreezing the LLM for training in the second stage causes only a 2.8% performance drop (12.10 vs. 12.45) on AI-Shell and a 2.1% performance drop (11.55 vs. 11.8) on LibriSpeech, with no catastrophic forgetting observed.  \n\nDuring the speech generation stage, the LLM remains frozen, ensuring that catastrophic forgetting does not occur at this stage either.  \n\nWe hope this clarification highlights the robustness of OpenOmni’s multi-stage learning process and strategies for effectively handling catastrophic forgetting."}, {"review_id": "RgJ8eAPiIi", "reviewer": "Reviewer_GJPB", "summary": "This paper introduces OpenOmni, an open-source omni-modal large language model designed to unify vision, text, and speech modalities via a progressive two-stage training pipeline. First, OpenOmni aligns speech-text and image-text data independently using pretrained encoders and language models. Then, it learns a lightweight streaming speech decoder for real-time and emotionally expressive speech synthesis, using DPO. The authors propose that this decomposed strategy bypasses the need for tri-modal datasets while achieving better generalization.", "questions": "1.\tHow exactly does OpenOmni’s alignment strategy differ from that in Qwen-Omni or AnyGPT, which also used text-pivot alignment and pre-trained encoders? A quantitative or ablation-based comparison would strengthen the contribution.\n\n2.\tThe paper claims DPO is more stable than RLHF for emotional speech but does not present empirical comparisons. An ablation with RLHF or reward-based preference methods would make this claim credible.\n\n3.\tAlthough WER and emotion classification are used, human ratings (e.g., MOS or empathy scores) for speech naturalness and emotional congruence would strengthen the case for \"self-aware\" interaction. Have human evaluations been conducted?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe decomposition into two separate alignment stages (speech-text and image-text) using text as a pivot is a pragmatic workaround for the scarcity of high-quality tri-modal datasets. \n\n2.\tThe integration of a non-autoregressive speech decoder trained with DPO for emotional nuance is an engineering contribution. It improves latency and user experience in real-time multimodal dialogue systems.\n\nWeaknesses:\n1.\tThe proposed progressive alignment (text as pivot) is not novel. The approach is reminiscent of modality bridging used in prior work like AnyGPT and Qwen-Omni, both of which are cited but not sufficiently contrasted with in terms of technical differences. \n\n2.\tOpenOmni largely reuses existing components (Qwen2.5 LLM, CLIP, Whisper, GLM4-Voice tokenizer) and standard training techniques (CTC, MoE, instruction tuning). The speech decoder architecture is mostly standard with little new insight.\n\n3.\tThe descriptions are exhaustive but often lack clarity about what exactly is novel or generalizable. For example, the TGM module’s design is described but not evaluated independently, and its impact remains unclear.\n\n4.\tThe progressive alignment strategy, DPO training for emotional speech, and use of speech decoders are all incremental rather than truly novel. Several prior works cited by the authors (e.g., Qwen2-Audio, EMOVA, AnyGPT) have either done similar decompositions or employed similar optimization strategies.", "was_revised": true, "final_justification": "After reviewing the authors’ rebuttal and the discussion among reviewers, I find that several of my earlier concerns, particularly regarding the clarity of contributions and differences from prior work, have been addressed through additional explanations, comparisons, and ablation results. Given that the reasons to accept now slightly outweigh the reasons to reject, I am updating my rating to borderline accept.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1**&**Q1**: progressive alignment (text as pivot) is not novel. The approach...technical differences. How exactly ...? A quantitative or ablation-based comparison...\n\n**Response:**\n\nWe sincerely appreciate the reviewer’s feedback. While most multimodal learning methods [1,2,3] commonly use **text as a pivot** as a foundational approach, their **novelty and innovation** often lie in the **specific alignment strategies** tailored to applications. To highlight OpenOmni’s contributions, we compare its alignment strategy with previous works such as **AnyGPT** and **Qwen-Omni** and summarize the differences in the table below.\n\n| Method   | Data  | Minimum  Memory | joint training | tri-modal data | freeze LLM (speech) | freeze LLM (image) |\n| -------- | ----- | --------------- | -------------- | -------------- | ------------------- | ------------------ |\n| Any-GPT  | ~10M  | 60G             | $\\checkmark$   | $\\checkmark$   | $\\checkmark$        | $\\checkmark$       |\n| OpenOmni | 1.7M  | 40G             | $\\times$       | $\\times$       | $\\checkmark$        | $\\times$           |\n| QwenOmni | ~500M | 100G            | $\\checkmark$   | $\\checkmark$   | $\\times$            | $\\times$           |\n\nAs shown in the table, **AnyGPT** and **Qwen-Omni** rely on tri-modal data and single-stage joint training for all modalities. In contrast, **OpenOmni** employs a **progressive alignment strategy**, aligning modalities in separate stages. Each stage involves at most **two-modality alignment**, reducing GPU memory requirements and lowering the entry barrier for omnimodal learning.\n\nThis approach allows flexible training strategies. Ablation studies in **Appendix Tables 7 and 8** show that **text-speech alignment** achieves competitive results by freezing the LLM, improving training efficiency by **300%**, while **image-text alignment** requires unfreezing the LLM due to larger modality gaps for deeper alignment.\n\nBy tailoring strategies to specific modality characteristics, OpenOmni’s **dynamic training strategy** enhances flexibility, improves efficiency, and maintains competitive results. With less data, OpenOmni achieves **SOTA performance** compared to other open-source models, showcasing its effectiveness and innovation in omnimodal learning.\n\n> **W2**: OpenOmni largely reuses existing components .... The speech decoder architecture is mostly ....\n\n**Response:**\n\nThank you for the reviewer’s valuable feedback. While the reuse of existing components and standard training techniques is common across omnimodal learning frameworks, **OpenOmni** sets itself apart through its **dynamic and flexible modal progressive alignment strategy**, which minimizes dependencies on tri-modal data and GPU requirements—marking a key innovation compared to previous works.  \n\nFurthermore, OpenOmni utilizes a lightweight **CTC-based speech decoder** with notable improvements. It introduces the **MOE mechanism** to handle multilingual conflicts during CTC training and the **TGM module**, which stabilizes the CTC training process. These enhancements retain the real-time efficiency of CTC decoders while achieving speech generation quality on par with autoregressive methods.  \n\nThese contributions highlight OpenOmni's **innovations and findings**, offering valuable insights to the omnimodal learning community and driving advancements in this field.\n\n> **W3**: For example, the TGM module’s design is described but ...\n\n**Response:**\n\nThank you for the reviewer’s feedback. In Appendix Lines 650-654, we qualitatively analyzed Table 4, showing the **TGM module** significantly stabilizes training for both **CTC** and **AR mode decoders**, enhancing speech generation. Additionally, we supplemented this with quantitative experiments for a more comprehensive evaluation.\n\n| mode               | AI-Shell2-Dev (CER)$\\downarrow$ | LibriSpeech-Test-clean (WER)$\\downarrow$ |\n| ------------------ | ------------------------------- | ---------------------------------------- |\n| AR w/o TGM         | 7.4                             | 3.5                                      |\n| AR w/ TGM          | 6.9                             | 2.3                                      |\n| NAR (CTC)  w/o TGM | 8.7                             | 4.3                                      |\n| NAR (CTC) w/ TGM   | 7.3                             | 2.6                                      |\n\nAs shown in the table, the **TGM module** significantly improves content consistency in both **AR** and **CTC modes**, with strong generalization for Chinese and English speech generation. It has also been adopted by subsequent works [4,5] to enhance speech decoder consistency.\n\n> **W4**: The progressive alignment strategy...rather than truly novel. Several prior works (e.g., Qwen2-Audio, EMOVA, AnyGPT) have either done similar ....\n\n**Response:**\n\nThank you for the reviewer’s valuable comments. We clarify the differences between our method and previous works in terms of **progressive alignment strategy**, **DPO training for emotional speech**, and **speech decoders**:\n\n1. **Progressive Alignment Strategy**: OpenOmni employs a flexible modal alignment approach based on ablation experiments, using different strategies for **text-speech** and **image-text alignments**. This achieves competitive results with minimal computational resources and training data.\n2. **Speech Decoders**: OpenOmni uses a **CTC-based approach** for parallel real-time speech generation, unlike autoregressive methods in **AnyGPT** and **EMOVA**. We enhance CTC to support bilingual speech and yield results comparable to autoregressive techniques while maintaining efficiency.\n3. **DPO Training**: OpenOmni focuses on emotional consistency in multi-turn dialogues, similar to **GPT-4o**, allowing the model to learn user intent through context and output appropriate emotional speech for better interaction. This approach has been adopted by subsequent work [6] to improve multi-turn dialogue experiences. In contrast, methods like **EMOVA** rely on predefined emotion keywords for single-turn tasks, limiting their effectiveness in multi-turn scenarios.\n\nFrom these perspectives, OpenOmni fundamentally differs from previous methods.\n\n> **Q2**:The paper claims DPO is more stable.... An ablation with RLHF or reward-based ...\n\n**Response:**\n\nThank you for the reviewer’s comments. We conducted ablation experiments using the PPO algorithm and Verl framework, with the **EO2S-9K** dataset to train the reward model for RLHF. The experimental results and computational resource comparisons are shown in the table below.\n\n| GPU*Hour | Method | Lang | Angry & Disgusted | Fearful  | Happy    | Neutral  | Other    | Sad      | Surprised |\n| -------- | ------ | ---- | ----------------- | -------- | -------- | -------- | -------- | -------- | --------- |\n| 54       | DPO    | ZH   | **96.6**          | 78.4     | **37.7** | **97.1** | 62.8     | 90.7     | 29.8      |\n| 130      | PPO    | ZH   | 94.7              | **79.2** | 36.1     | 94.5     | **64.3** | **91.6** | **30.9**  |\n| 54       | DPO    | EN   | 91.3              | **70.4** | **60.6** | **94.6** | 49.6     | 77.3     | **13.9**  |\n| 130      | PPO    | EN   | **91.6**          | 69.4     | 59.8     | 92.7     | **50.3** | **79.5** | 10.6      |\n\nAs shown in the table, while PPO shows no significant performance advantage over DPO, it requires **loading and training four model copies**, resulting in much higher GPU resource consumption.  \n\nMoreover, during training, the **EO2S-9K** reward model often failed to provide accurate evaluations, causing instability in PPO. Outlier data, where the policy model generated unusual speech and the reward model delivered incorrect emotional judgments, repeatedly led to training failures, requiring **three attempts** to complete.  \n\nIn contrast, DPO is unaffected by reward model performance on small datasets, offering more stable training, lower resource consumption, and results comparable to PPO, making it a more efficient and practical choice.\n\n> **Q3**: Although WER and emotion classification .... Have human evaluations been conducted?\n\n**Response:**\n\nThank you for the reviewer’s comments. We used the **MOS (Mean Opinion Score)** metric, with **10 experts** evaluating the generated speech for Chinese and English sets and EO2S emotional speech. The experimental results are shown in the table below.\n\n| mode         | AI-Shell2-Dev (MOS)$\\uparrow$ | LibriSpeech-Test-clean (MOS)$\\uparrow$ | EO2S (MOS)$\\uparrow$ |\n| ------------ | ----------------------------- | -------------------------------------- | -------------------- |\n| ground_truth | 4.50                          | 4.50                                   | 4.50                 |\n| AR           | 3.89                          | 4.22                                   | 4.14                 |\n| NAR (CTC)    | 3.83                          | 4.16                                   | 4.08                 |\n\nThe experimental results indicate that the MOS evaluations from human experts demonstrate that the speech generated by OpenOmni is **natural, fluent, and exhibits emotional consistency**.\n\n[1] LLaVA: Large Language and Vision Assistant\n\n[2] AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling \n\n[3] EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions\n\n[4] Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model \n\n[5] LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis\n\n[6] OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction"}, {"review_id": "RHGruOYpLW", "reviewer": "Reviewer_ULJw", "summary": "The paper presents technical details of various stages in developing “OpenOmni” – an omnimodal LLM (vision, speech, text). The development includes two main parts: (i) omnimodal alignment and (ii) speech generation.\n\nFirst, in omnimodal alignment, the process involves: (Section3.1) speech-text alignment where speech encoder / adapter is trained on speech-text data (similar to Qwen-Audio, SALMONN, etc); (Section3.2) visual-text alignment including visual encoder pretraining and visual encoder + LLM instruction tuning similar to previous works on visual LLMs (e.g., LlaVA).\n\nSecond, in speech generation, the process involves: (i) speech decoder converting LLM output representation into discrete units (output representation –[upscale + transformer layers]--> discrete unit –[vocoder]--> audio) similar to the architecture of Llama-Omni (with an exception of the MoE component); (ii) emotion speech generation via DPO, similar to Emo-DPO.\n\nThe experimental section provides implementation recipes to develop such as a model, including data in various stages. Evaluation is conducted on OmniBench, and compared against existing omnimodal models. \n\nKey results are  \n- Omni and Vision performance: Compared to VITA, OpenOmni achieves a higher score on OmniBench without using tri-modal data.\n- Speech performance: It performs better than VITA, while being comparable to open-source speech LLMs. Evaluation of the emotion of generated speech shows slight improvements in both English and Chinese.\n\nOverall, this paper provides a technical report on developing OpenOmni. The paper consists of implementation details and experimental results, which can be useful for the open-source LLM development community.", "questions": "First of all, it should be noted that I like this paper as a technical report of a model development, and it could be useful for the open-source community, focusing on multimodal LLMs. However, could you highlight the key innovations, novelties, or findings that were learned during the development of OpenOmni?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper provides technical details on developing OpenOmni with details on architectures, training stages, data, and evaluation results.\n- Empirical results reveal that it can achieve better performance when using multi-stage omni alignment, without tri-modal data (i.e., when compared to VITA).\n- The implementation details and experimental results can be useful for the open-source LLM development community.\n\n**Weaknesses**\nThere is little new novelty in this paper. In my opinion, this paper reads more like a technical report of a model development. It adopts existing components or ideas, and stitches them together into a model. \n\nThe paper would benefit from a deep analysis into ablations of each component (e.g., some questions to investigate can be: how much data does one need to align omnimodality?, Does it learn cross-lingual skills (e.g., visual-text only in English, but speech-text in many languages)?, What are the weaknesses of OpenOmni compared to VITA or SOTA proprietary models such as GPT-4o, or even newer open-source like Qwen2.5-Omni, Is the current emotion speech (through DPO) good enough compared to proprietary and what might be other options to enhance the emotion speech generation further?", "was_revised": true, "final_justification": "I have read the rebuttal, and with further details and ablations, I believe this work could benefit the open-source community in building omnimodal LLMs. I've raised my judgment accordingly. However, in terms of its technical novelty, several points have been proposed and used by prior models.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1&Q5**: There is little new novelty in this paper. In my opinion, this paper reads.... However, could you highlight the key innovations, novelties, or findings ..\n\n**Response:**\n\nWe sincerely thank the reviewer for their thoughtful feedback. While many multimodal learning works may seem to simply integrate pretrained components and train them jointly, we believe their **novelty lies in addressing specific challenges and developing solutions**, which ultimately shape the uniqueness of the work. OpenOmni, as an open-source solution, aims to enable efficient training and competitive performance under limited data and computational resources, requiring just **40GB peak GPU memory**. By making omnimodal learning accessible to the community, OpenOmni bridges the gap between academic research and proprietary models.\n\n##### **Key Innovations, Novelties, and Findings**\n\n1. **Dynamic Progressive Alignment Strategy**:\n   OpenOmni uses a **progressive alignment strategy** to optimize computational resources and data use, achieving **shallow alignment** for text-speech and **deep alignment** for image-text tasks. Ablation studies in **Tables 7 and 8** validate its ability to improve both efficiency and performance compared to joint training.\n2. **Robust Speech Decoding with TGM Module**:\n   The **TGM module** stabilizes training for the **CTC-based speech decoder**, enabling real-time multilingual speech synthesis while enhancing quality comparable to autoregressive models like **EMOVA** and **VITA**. **Figure 4** confirms its effectiveness.\n3. **Multilingual Support with MOE Module**:\n   Conventional CTC-based methods (e.g., **LLaMA-Omni**) are limited to single-language speech generation. OpenOmni introduces the **MOE module**, supporting effective multilingual synthesis during CTC training.\n4. **CTC-DPO Algorithm for Emotional Speech**:\n   OpenOmni’s **CTC-DPO algorithm** generates contextually consistent emotional speech for multi-turn dialogues by leveraging multi-turn datasets, bridging the user experience gap with proprietary models for adaptive interaction.\n5. **Impact on Subsequent Research**:\n   OpenOmni’s innovations have advanced later works:\n   - **TGM module** improves speech decoder performance [1,2].\n   - Multi-turn emotional consistency enhances multi-character interactions [3].\n\nThese contributions underscore OpenOmni’s progress in omnimodal learning, lowering research barriers and fostering broader community participation to advance the field collaboratively.\n\n> **Q1**: how much data does one need to align omnimodality?\n\n**Response:**\n\nThank you for the reviewer’s insightful feedback. Appendix Tables 7 and 8 present preliminary findings from our ablation experiments on the omnimodal alignment strategy:  \n\n1. **Alignment Order**: No significant performance difference exists between **text-speech first** and **image-text first** alignments, as speech and image align independently to the text space.  \n\n2. **Alignment Depth**: **Shallow alignment** (freezing the LLM) is effective for text-speech due to their natural temporal correlation, while **deep alignment** (unfreezing the LLM) is required for image-text to achieve better results.  \n\nWe hypothesize that text-speech alignment demands far less data than image-text alignment. To verify this, we conducted experiments using varying training data amounts, as detailed below.\n\n| Speech      | AI-Shell2-Dev (CER)$\\downarrow$ | LibriSpeech-Test-clean (WER)$\\downarrow$ |\n| ----------- | ------------------------------- | ---------------------------------------- |\n| 100 Hours   | 16.2                            | 14.7                                     |\n| 500 Hours   | 8.6                             | 6.9                                      |\n| 900 Hours   | 7.2                             | 4.0                                      |\n| 1,300 Hours | 6.9                             | 3.3                                      |\n| 1,600 Hours | 6.8                             | 3.1                                      |\n\n| Image | MMStar | MMB  | MMMU | AI2D |\n| ----- | ------ | ---- | ---- | ---- |\n| 0.2M  | 35.6   | 46.3 | 29.5 | 54.6 |\n| 0.6M  | 43.5   | 60.8 | 37.9 | 64.0 |\n| 1.0M  | 49.8   | 69.9 | 42.3 | 70.4 |\n| 1.4M  | 51.2   | 72.5 | 44.1 | 72.3 |\n| 1.7M  | 52.3   | 76.2 | 46.7 | 74.8 |\n\nFor text-speech alignment, **900 hours of data** achieved **97%** of the performance obtained with **1,600 hours**, showing the efficiency of natural temporal correspondence where **data quality** outweighs **quantity** in later stages.  \n\nFor image-text alignment, performance improves as data increases, with slower gains in later stages but room for growth, suggesting a larger gap between image and text representations where additional data enhances alignment.  \n\nThese findings highlight differing data requirements for text-speech and image-text alignments, offering valuable insights for future research.\n\n> **Q2**: Does it learn cross-lingual skills?\n\n**Response:**\n\nTo evaluate multilingual generalization during speech-text alignment, we used the **Multilingual Speech Commands Dataset** validation set, containing speech data in 14 non-English languages. Zero-shot command recognition experiments were conducted on OpenOmni during text-speech alignment, and the accuracy results are shown in the table below.\n\n| Steps     | Kazakh      | Russian     | Tatar       | Arabic      | Turkish    | French    | German          |\n| --------- | ----------- | ----------- | ----------- | ----------- | ---------- | --------- | --------------- |\n| 100       | 12.8        | 38.5        | 20.3        | 25.5        | 22.7       | 30.1      | 34.2            |\n| 300       | 18.7        | 43.8        | 27.1        | 31.0        | 28.3       | 35.8      | 40.9            |\n| 500       | 25.2        | 48.9        | 34.0        | 37.0        | 34.2       | 41.5      | 47.6            |\n| **Steps** | **Spanish** | **Italian** | **Catalan** | **Persian** | **Polish** | **Dutch** | **Kinyarwanda** |\n| 100       | 30.2        | 28.1        | 22.5        | 18.3        | 20.4       | 24.7      | 10.8            |\n| 300       | 36.0        | 33.9        | 29.2        | 23.7        | 26.0       | 30.6      | 16.4            |\n| 500       | 41.8        | 39.7        | 35.8        | 29.1        | 31.5       | 36.4      | 22.0            |\n\nAs shown in the table, OpenOmni’s text-speech alignment with Chinese and English data demonstrates strong cross-lingual generalization, enabling consistent improvements in non-English speech recognition.\n\n> **Q3**: What are the weaknesses of OpenOmni compared to ..,\n\n**Response:**\n\nWe appreciate the reviewer’s detailed feedback and would like to clarify the distinctions between **OpenOmni**, **VITA**, and proprietary models like **GPT-4o**:\n\n1. **Architectural Differences**:  VITA uses a pipeline approach, generating text responses first and then passing them to a TTS model, which risks **error accumulation**. It also requires duplex communication with two model deployments, significantly increasing GPU memory needs. OpenOmni adopts an **end-to-end design**, feeding LLM features into a **CTC-based decoder** for real-time parallel decoding, enabling **emotionally consistent speech** in multi-turn dialogues—something VITA struggles with.\n\n2. **Training Strategies**:  OpenOmni employs a **progressive alignment strategy** that reduces tri-modal data dependency and increases efficiency, requiring only **40GB peak GPU memory** and **1,400 GPU Hours** for strong performance. By contrast, VITA uses **joint training**, consuming **three times more data** (**5M vs. 1.7M**) and **80GB GPU memory**, with nearly **10,000 GPU Hours**, making it significantly more computationally expensive.\n\n3. **Advantages of OpenOmni**:  OpenOmni is **faster, more efficient**, and excels at producing **emotionally consistent speech** for better interaction experiences. Its design lowers research and deployment barriers, making omnimodal research more accessible.\n\n4. **Comparison with Proprietary Models**:  OpenOmni lacks access to large-scale industry datasets or computational resources like **GPT-4o** and **Qwen2.5-Omni**. While its **CTC-based decoder** offers efficient real-time emotional speech generation, speech quality is slightly inferior to the autoregressive methods used in proprietary models.\n\nIn summary, OpenOmni optimizes efficiency, accessibility, and performance, contributing significantly to omnimodal research while lowering entry barriers for the academic community.\n\n> **Q4**: Is ... emotion speech (through DPO) good enough .. what might be other options to ..\n\n**Response:**\n\nThank you for the reviewer’s feedback. We are pleased to discuss emotion speech generation further.  \n\nAlthough OpenOmni still has gaps compared to proprietary models like **GPT-4o**, its focus on **multi-turn emotional speech generation** and strong context-learning capabilities has significantly narrowed the gaps with previous methods (such as **EMOVA** and **Qwen-Audio**) and proprietary models.\n\nRecent studies [4,5] show that large-scale datasets of millions of hours enable adaptive, contextually coherent speech generation. However, such data and resources remain beyond the academic community’s reach. Meanwhile, **post-training strategies like CTC-DPO in OpenOmni** provide a practical, efficient solution for producing reliably consistent emotional speech.  \n\n\n\n[1] Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model \n\n[2] LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis\n\n[3] OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction\n\n[4] Higgs Audio V2: Redefining Expressiveness in Audio Generation\n\n[5] MOSS: Text to Spoken Dialogue Generation"}]

Author rebuttal:

Meta-review: The paper introduces OpenOmni, an open-source omnimodal large language model (LLM) framework that integrates vision, text, and speech modalities, with a focus on real-time emotional speech synthesis and efficient multimodal alignment.

Key Contributions:

•	Modality-progression training: A two-stage alignment strategy enables effective tri-modal integration without requiring tri-modal data.

•	Real-time emotional speech synthesis: Achieved via a lightweight non-autoregressive decoder optimized with Direct Preference Optimization (DPO).

•	Efficiency and performance: Outperforms state-of-the-art models like VITA with 5× fewer training samples, smaller model size, and sub-second latency.

•	Benchmark results: Strong performance across OmniBench and other multimodal benchmarks, with improvements in emotion classification accuracy.

Reviewers’ overall attitudes:

•	Reviewers appreciated the technical depth, implementation clarity, and practical relevance of the work.

•	The rebuttal addressed key concerns, including architectural choices and training strategies.

•	While some noted that certain components build on prior work, the overall contribution is seen as valuable for the open-source community.

•	Reviewers generally raised their scores, with one moving to a borderline accept and others expressing strong support.
Decision: Accept (poster)