NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
Authors: Run Luo
Xiaobo Xia
Lu Wang
Longze Chen
Renke Shan
Jing Luo
Min Yang
Tat-Seng Chua
Venue: ICLR 2026 Poster (2026)
Keywords: Omnimodal
Multimodal Learning
Discrete Flow Matching
Primary area: foundation or frontier models, including LLMs

Abstract: Next-generation multimodal foundation models capable of any-to-any cross-modal generation and multi-turn interaction will serve as core components of artificial general intelligence systems, playing a pivotal role in human-machine interaction. However, most existing multimodal models remain constrained by autoregressive architectures, whose inherent limitations prevent a balanced integration of understanding and generation capabilities. Although hybrid and decoupling strategies have been explored to address these tasks within unified frameworks separately, their redundant, non-integrated designs limit their applicability to broader scenarios, such as cross-modal retrieval. In this work, we introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms. By leveraging metric-induced probability paths and kinetic optimal velocities, NExT-OMNI natively supports any-to-any understanding and generation with enhanced response efficiency, while enabling broader application scenarios through concise unified representations rather than task-decoupled designs. Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal understanding and generation benchmarks, while outperforming prior unified models in multi-turn multimodal interaction and cross-modal retrieval, highlighting its architectural advantages as a next-generation multimodal foundation model. The code is available at https://github.com/ritzz-ai/Next-OMNI.

Published as a conference paper at ICLR 2026
NEXT-OMNI: TOWARDS ANY-TO-ANY OMNI- MODAL FOUNDATION MODELS WITH DISCRETE FLOW MATCHING
Run Luo Xiaobo Xia Lu Wang Min Yangl.2.6+ Tat-Seng Chua
Longze Chen Renke Shan
Jing Luo
"Shenzhen Institutes of Advanced Technology. Chinese Academy of Sciences
"University Chinese Academy of Sciences "National University of Singapore University Science and Technology of China *Rtizz-AI
Shhenzhnn University of Advanced Technology
) (r. luo, min. yang) f iistt......... ac cn, xiaoboxia ia. uni @emmallll... com
ABSTRACT
Next-generation multimodal foundation models capable of any-to-any cross- modal generation and multi-turn interaction will serve core components of arti- ficial general intelligence systems, playing a pivotal role in human-machine inter- action. However, most existing multimodal models remain constrained by autore- gressive architectures, whose inherent limitations prevent balanced integration of understanding and generation capabilities. Although hybrid and decoupling strategies have been explored to address these tasks within unified frameworks separately, their redundant, non-integrated designs limit their applicability to broader scenarios, such as cross-modal retrieval. In this work, we introduce NExT- OMNI, an open-source omnimodal foundation model that achieves unified mod- eling through discrete flow paradigms. By leveraging metric-induces probability paths and kinetic optimal velocities, NEXT-OMNT natively supports any-to-any understanding and generation with enhanced response efficiency, while enabling broader application scenarios through concise unified representations rather than task-decoupled designs. Trained on large-scale interleaved text, image, video, and audio data, NEXT-OMNI delivers competitive performance on multimodal under- standing and generation benchmarks, while outperforming prior unified models in multi-turn multimodal interaction and cross cross-modal retrieval, highlighting its ar- chitectural advantages as next- gexeratnnnttion multimodal foundation model. The code is available at https::///// thub. com ritzz- /Next-CMM
INTRODUCTION
Unified multimodal understanding and generation has emerged as a critical bottleneck for achieving artificial general intelligence (AGI), attracting growing academic attention (Dong et al., 2024; Ge al., 2024; Fu et al., 2024; Xie & Wu, 2024). Numerous studies (Wu et al., 2024c; Dong et al., 2024; Ge et al., 2024; Wu et al., 2024b) have attempted to leverage successful autoregressive (AR) techniques from large language models (LLMss) (Touvron et al., 2023; Yang et al., 2024; Wu al., 2024d) to achieve unified modeling of multimodal understanding and generation. However, these attempts have failed to achieve desired outcomes due to inherent conflicts within AR paradigms when handling understanding and generation tasks (Wu et al., 2024b;c; Dong et al., 2024).
Subsequent works have adopted hybrid architectures (Xie et al., 2024; Zhou et al., 2024) and mod- ular decoupling techniques (Wu et al., 2024a; Deng al., 2025; Chen et al., 2025a) to separately process understanding and generation tasks within relatively unified frameworks. These methods improve performance both task categories and narrow the gap between open-source models and closed-source systems (OpenAI, 2025; DeepMind, 2025). Nevertheless, such AR-based hybrid ar- chitectures primarily rely decoupling rather than fusion design. Although effective in under-
"Corresoonding authors.

---

Published as a conference paper at ICLR 2026
Gonerii siles led molloy revord Nower baaed piva Image.
Fiberol visee and namrhha eheast the voloo cats!
The noloo shove o doswsp of lorpe ple mest fealy clicken deing cociadd on grik, The neor sovieed in srace paubly a macnode o carbece scues. gri to actioors indlcted growey
the ond hou the dlkak' of the cooking process
it unl
galli calos vidi roulili.
Syskoure selerd, from fellwing whe's boee the whiis of thew Aealty TV shows?
seers kie car sugine renving
Caneration
Geocclee ckp autaa Delorted gulton, shedy bas, shop than boatt
A AAES MLT SOSEE d SUPEMIL SO 0S 2016 R Ro
Aetioval
Generass o photo of Ae Adtenaut ralnn lert....
What dfffrme beboure lunpe Laage
tvo inagas the cienecte aloen hagging Dorcamson
repooit
Dotabase
retioeble Wireeess behween
Ao chocadc''s / Lift buy caloaiaa snterdd and proms siceed' godget is atroshed to ther best whise Us mares round reeeve
the tmags
chert, serend rerpe,
godget
trreooree into leert sepeed park okpect
Figure I: Overview of the NEXT-OMNI framework. NEXT-OMNI a unified framework for om- nimodal tasks, offering strong understanding, generation, and retrieval capabilities. The framework enables any-to-any operations across modalities for both generation and understanding tasks, while achieving accelerated processing. It also supports cross-modal retrieval by leveraging rich multi- modal representations aggregated within its architecture.
standing and generation tasks, they inevitably introduce additional parameterized modules, thereby increasing structural complexity and slowing inference. Moreover, this separated design struggles to support tasks that demand deep multimodal feature fusion and flexible any-to-any cross-modal tasks, such as mrosll retrieval, which limits their applicability to more general multimodal scenarios.
In recent years, models based on discrete diffusion and flow matching have demonstrated competi- tive advantages over traditional AR counterparts across multiple domains, including language mod- eling (Nie et al., 2025; Ye et al., 2025a; Fan al., 2026), image modeling (Rombach et al., 2022a; Li et al., 2024f), and audio modeling (Du et al., 2024; Li et al., 2025c). Unlike sequential AR models, these models begin with completely corrupted sequences and iteratively denoise entire sequences in parallel, enabling richer bidirectional information integration to enhance task performance. In addition, they achieve flexible and controllable generation through inherent iterative refinement pro- cesses, while demonstrating accelerated sampling potential through parallel decoding, providing more promising fusion perspective for any-to-any multimodal understanding and generation tasks. However, this research direction remains underexplored, with its potential in unified understanding. generation, and retrieval yet to be fully realized.
To address this research gap, we propose NEXT-OMNI, a fully open-source omnimodal foundation model based on discrete flow matching (DFM) techniques (Shaul et al., 2024), trained on large- scale, carefully curated interleaved multimodal datasets encompassing images, text, video, and au- dio. As shown in Figure 1, NEXT-OMNI achieves faster any-to-any omnimodal generation through streamlined unified architecture, while enabling more precise cross-modal retrieval through uni- fied representations with intermediate feature fusion. Overall, NEXT-OMNI not only demonstrates competitive performance with reduced latency on standard multimodal understanding and genera- tion benchmarks, but also exhibits superior performance in multi-turn multimodal interaction and cross-modal retrieval. These results highlight that DFM-based unified multimodal understanding and generation modeling architectures provide powerful fusion perspective for advancing multi- modal unification with broader applicability.
The contributions of this paper are summarized as follows: open-source omnimodal model built entirely on DFM, which is capable of achieving any-to-any gen- eration across text, images, video, and audio with faster inference. 2) We design reconstruction-
1) We propose NEXT-OMNI, the first
2

---

Published as a conference paper at ICLR 2026
Cross Entropy Loss
Tokes
Vise faice
Vision Head
Audio
Pour foken
Audo Tokun
Velochy Prediction
Coo Abade Puriicall United Rxpressttrioon
Attention
Unified Discrete Flow
Matcchng Somple
Cookboak Echadding
Ward Embedding
Cobesiin
Excoder
Encoder
Asy-to-Asy Goneration
Visio Decoder
Reconstruction Loss
Audia Decodar
Figure 2: Pipeline of the NEXT-OMNI framework. NEXT-OMNI employs a DFM paradigm for unified omnimodal training, with multimodal self-attention at every layer to deeply fuse information across modalities. Unlike prior methods using multiple encoders or mixture-of-experts it trains single encoder simultaneously for understanding and generation, producing unified representations that enable my multimodal tasks with streamlined architecture and strong generalization.
enhanced unified representation with intermediate feature fusion. This design not only enables pre- cise cross-modal retrieval but also supports multi-turn any-to-any multimodal interactions, demon- strating advantages over separated AR-based frameworks. 3) We conduct extensive experiments across understanding, generation, and retrieval benchmarks. Results show that NExT-OMNI consis- tently achieves competitive or superior performance with reduced latency, validating the potential of DFM-based architectures promising paradigm for unified multimodal modeling.
METHOD
2.1 ARCHITECTURE
Modality Encoders. We design vision and audio encoders grounded in unified representation prin- ciples (Luo et al., 2024a; Wu et al., 2024c; Ma et al., 2025a), enabling unified representational modeling. This design allows single-modal encoders to support both generation and understanding tasks, while also mitigating encoder redundancy.
Backbone. NEXT-OMNI is initialized from the pretrained weights of AR-based LLMs and employs discrete flow matching within a three-stage progressive training framework on carefully curated om- nimodal data. Following previous work (Gong et al., 2024b; Ye al., 2025a; Wang et al., 2025), we retain the shifting operation to output logits by one position during training, enabling our model inherit the next-token prediction capabilities of AR-based LLMs the greatest extent possi- ble. To expand model application scenarios such as cross-modal retrieval while streamlining model structure, we utilize deep bidirectional attention feature fusion rather than relying on additional MoE/MoT decoupling mechanisms (Xic al., 2024; Deng et al., 2025).
Modality Heads. Since NEXT-OMNI employs discrete flow matching, it eliminates the need for additional diffusion or flow heads (Dong et al., 2024; Luo et al., 2024a; Han et al., 2025; Chen al., 2025a) specifically designed for generation optimization. Instead, only requires lightweight heads for discrete token decoding, thereby substantially improving training efficiency and accelerating generation response. Furthermore, we introduce separate modality-speciif heads for decoding each type of modality data, rather than extending the language model's vocabulary head directly. This
3

---

We first perform warmup training for unified representation modeling of the modality encoders. Two objectives are combined: (i) a reconstruction loss L implemented with an auxiliary VQVAE (Van Den Oord al., 2017) quantizer and modality-speciff decoders that capture low-level details; (ii) a semantic alignment loss CM with an auxiliary text encoder or text decoder that emphasize high- level semantic aligmments. Here, the superscript M corresponds to the modality. Given a continuous input vector zM, which is obtained from a modality input XM encoded by the corresponding modality encoder EM, the goal is to map process is formulated as z arg mineggM ||zM c|22, where CM {cM c22 ................................ " J. and K is the number of codebook entries. The aim is to map the input zM to the most representative vector ox CM via a VQ loss - Then we use the corresponding modality decoder DM restore the modality input XM conditioned on the representative vector via a modality restoration loss 4 The reconstruction loss can be formulated as L. C LN + CM where L is discriminator loss. modality encoders can be Published as a conference paper at ICLR 2026
design effectively preserves text generation capabilities. Additional details on the modality encoders and heads are provided in Appendix D.
2.2 UNIFIED REPRESENTATION MODELING
the closest vector in learnable codebook CMM. The quantization
M
The total warmup training objective for our unified representation-base expressed:
(1) In more detail, to address the distinct semantic granularity requirements of vision modality V and audio modality A, we adopt token-level caption generation alignment L = L for the audio en- coder, following Whisper (Radford et al., 2023), and sentence-level contrastive semantic alignment f.m Connstra for the vision encoder, following CLIP-ViT (Dosovitskiy et al., 2021).
C
sem: M (A, V}.
rec
2.3 DISCRETE FLOW MATCHING MODELING
As illustrated Figure 2, given an omnimodal dision-text-aude sequence input sampled from target distributions q.)), NEXT-OMNI first utilizes VQVAE-basees modality encoders and text tokenizers to convert into discrete target token sequences X, x xx)), x(), where each element xf is arranged in the order in which they appear in the original content, At each training step, a time 1 E [0, 1] is uniformly sampled, and noisy sequence x, is sampled according to the probability path pr (-(xx) defined in Appendix A. Then, the model receives a noisy sequence X; an input and predicts X1. outputting per-token logits for each position. Note that, unlike previous methods (Wang extract continuous representa-
et al., 2024b; Han et al., 2025) that directly utilize discrete tokens, tive vector .M with rich semantic and detailed information from the corresponding quantizer code- books CM of modality encoders based on discrete tokens, and achieve dimensional alignment with text embeddings through lightweight projection. This simple yet effective method enables the model to achieve superior performance in subsequent optimization. During training, we only perform cor- rection training on the response portions of instruction data. The discrete flow matching (DFM) modeling is defined as the expected cross-entropy loss between the ground-truth sequence X1 and the model's predicted distribution as follows:
Lee E,,1(00,11, x(((()) x(pp)())
Σ Pije (xf [xxs)
In addition to the cross-entropy loss mentioned above, to prevent the model from overly favoring semantic information during DFM training while discarding fine-grained information embedded in the unified representations of modality encoders, which would degrade model performance on understanding and generation tasks, we constrain the DFM training by reusing the corresponding reconstruction losses from the modality encoders in unified representation modeling. This maintains rich and detailed information, which not only improves understanding and generation performance but also enhances deep multimodal feature fusion, providing more precise gross-modal retrieval capabilities. The overall training objective can then be rewritten below:
Loverall X1 Loe X2 CV + A3 CA.
where \1. A2, and Ag are the coefficient that controls the trade-offs between DFM modeling and the modality reconstruction loss. We adopt the GradNorm (Chen et al., 2018) method to dynamically adjust the coefficient, ensuring equal gradient update contributions to the model during training.
4

---

Published as a conference paper at ICLR 2026
2.4 MORE DETAILS OF TRAINING AND INFERENCE
During.join training, different data modalities require different training modules due to modality dif- ferences, and random mixed training leads load imbalance that wastes substantial computational resources. To improve training efficiency, we conduct training for only one modality within any given training batch, achieving the joint training objective through interleaved training multiple tasks and gradient accumulation, effectively enhancing computational resource utilization efficiency and achieving a 1.4x improvement in training efficiency.
As illustrated in Figure 3, to further improve NEXT-OMNT's performance on understanding tasks, we design dynamic length generation strategy. Specifically, during training, we insert additional <PAD> tokens to ensure that response sequences participating in training are multiples of the block size. During inference, leveraging the properties that simple tokens can be determined in single de- noising step, dynamically adjust to appropriate preset generation lengths in block size increments based on <EOS> confidence, then perform multi-step iterative denoising. This strategy dramati- cally enhances the model's text generation capabilities at minimal cost. Furthermore, phenomena similar to DLLM (Liu et al., 2025c) during the multi-step denoising process in inference, where most features change minimally throughout the multi-step denoising procedure, providing op- portunities for inference acceleration using caching mechanisms. We cache and perform minimal updates on the instruction portion throughout the entire inference process, while adaptively updat- ing during the response process based on the cosine similarity between value features and cached features. This vanilla adaptive cache implementation, combined with the parallel decoding advan- tages of NEXT-OMNT's DFM architecture, achieves a 1.2x inference response speed improvement compared AR architectures. Our cache acceleration and dynamic generation strategies enable superior performance with faster response speed.
observe
Inafrsction
0=
Response
FOS
Training
Block Size
PDD
Randon
Expard
0.203a225
Inatruction
Inference
Resporsw
Expard
0.43-075
Coche
ouep
Adoptive
Coche
Expard Stop
Figure 3: Illustrations dynamic generation strategy (left) and vanilla adaptive cache design (right). During training, responses padded to multiples of the block size, allowing the model to extend preset response lengths block-size increments during inference. This improves perfor- mance understanding tasks that require dynamic-length generation. The vanilla adaptive cache caches instruction features and selectively caches response features based on feature cosine similar- ity, accelerating inference and decoding. Combined with parallel decoding, this simple yet efficient caching design enables NEXT-OMNI to generate responses faster than AR-based models.
EXPERIMENTS
In this section, extensive experiments are conducted to demonstrate the superiority of the proposed NEXT-OMNI and justify our claims. We first introduce implementation details (Section 3.1). Fol- lowing this, series of results on various practical tasks and subsequent discussions are provided, including omnimodal understanding (Section 3.2), vision interaction (Section 3.4), speech interac- tion (Section 3.3), and multimodal retrieval (Section 3.5). Finally. the ablation study (Section 3.6) presented to examine the contributions of different components in NEXT-OMNI, offering deeper insights into the factors that drive its success. For experiments on single-turn interactions such as text-to-image and text-to-audio generation, please refer Appendix H for more details.
5

---

Published as a conference paper at ICLR 2026
3.1 IMPLEMENTATION DETAILS
Model Configurations. We initialize our vision encoder and audio encoder with CLIP-ViT- Large (Dosovitskiy et al., 2021) and Whisper-Turbo (Radford et al., 2023) weights, respectively, then perform joint training for reconstruction and semantic alignment using auxiliary VQVAE (Van Den Oord et al., 2017) and corresponding decoders. Specifically, we conduct warmup training on 70M image-text pairs constructed from LAION (Schuhmann et al., 2022) and DataComp (Li al., 2024d). The image resolution is set to 256x256 with a downsampling rate of 16. We reuse the text encoder from CIIP-VIT to extract caption semantics for the computation of the contrastive loss f-comstr For the audio encoder, we conduct warmup training using a combination of open-source datasets, including LibriSpeech (Panayotov et al., 2015), WenetSpeech (Zhang et al., 2022), and AudioCaps (Kim et al., 2019), totaling 2,000 hours, supplemented by proprictary speech and music datasets amounting to 100,000 hours. During this process, we set the maximum audio clip length to 15 seconds and employ Qwen2.5-0.5B (Yang et al., 2024) for the computation of the audio cap- tion loss Lapp We initialize NEXT-OMNI with Qwen2.5-7B (Yang et al., 2024) weights, equipped with the warmup-trained vision encoder and audio encoder, along with lightweight modality heads containing nearly 128M parameters, and conduct progressive three-stage DFM training on carefully constructed omnimodal data. Throughout this process, we set the classifier-free guidance (Ho & Salimans, 2022) probability to 0.1 for multimodal generation tasks and the response padding block size to 64 for multimodal understanding tasks. We reuse the reconstruction terms Lee and f from the modal encoder warmup. These steps enhance multimodal generation, understanding. and retrieval capabilities comprehensively.
Pre-Training (PT), To efficiently and stably conduct flow matching modeling, we perform om- nimodal joint training during the alignment pre-training stage using short audio clips within 15 seconds, single images at 256 resolution with 16x downsampling, and short text with max- imum context window of 2K, leveraging large amounts of low-quality data to rapidly achieve om- nimodal alignment. We train a mixture of image-text pairs and audio-text pairs. Specifically, ImageNet- 1K (Deng et al., 2009), JourneyDB (Pan et al., 2023), LAION (Schuhmann et al., 2022), and FLUX (Labs, 2024) synthetic data for image generation; re-captioned image-text pairs from COYO (Byeon et al., 2022), CommonCraw/ (Common Crawl, 2007), LAION (Schuhmann et al., 2022), and DataComp (Li et al., 2024d) for image understanding: LibriSpeech (Panayotov et al., 2015), WenetSpeech (Zhang et al., 2022), AudioCaps (Kim et al., 2019), and proprietary data for audio understanding and generation. To prevent degradation of the model's 's textual capabilities, we sample shorter pure text data from Infinity-Instruct (Li et al., 2025a) and Evol-Instruct (Xu et al., 2024), and incorporate them into the mixed training dataset.
Continued Pre- Training (CPT). During the CPT stage, we increase image resolution to 3 184x388 384 and introduce long text, interleaved image-text, video, and long audio data with a maximum context window of 16K. This enables the model to support natural multi-turn visual and audio interactions while possessing the capability to understand and preliminarily generate short videos. For video data, we uniformly extract 8 frames them into multiple chunks with a maximum length of 15 seconds. Our experiments demonstrate that this simple strategy efficiently endows the model with video and long audio understanding and generation capabilities. Beyond partial resampling of pre-training stage data, we also incorporate PixMo (Deitke al., 2024) and LLaVA-One Visiion (Li et al., 2024a) training data for image under- standing; FLUX (Labs, 2024) synthetic data for image generation; MMC4-Core (Zhu et al., 2023), OmniCorpus (Li et al., 2025b), and ShareGPT4 Video (Chen et al., 2024c) supporting multi-image and short video understanding: Open Vid (Nan et al., 2024) and internal video data for video genera- tion; OpenOmni (Luo al., 2025a) synthetic data exceeding 15 seconds and proprietary audio data for audio understanding and generation.
multi- image input. For long audio data, we decompose
Supervised Fine-Tuning (SFT). In the SFT stage, we train the model to learn all multimodal- related instruction data, equipping it with any-to-any generation capabilities to accomplish diverse multimodal generation and understanding tasks. We collect LLaVA- Vision (Li et al., 2024a) and PixMo (Deitke et al., 2024) instruction data for image generation; LLaVA-Video (Zhang et al., 2025c) instruction data for video understanding, OpenOmni (Luo et al., 2025a) for multi-turn audio interaction; InterSyn (Ma et al., 2025b) for multi-turn image interaction: along with BLIP3-o (Chen et al., 2025a), ShareGPT-4o-Image (Chen et al., 2025b), Saao-coosisttett (Ye et al., 2025b) image generation, and TIP-12V (Wang& & Yang, 2024) video generation to construct an omnimodal training

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with existing state-of-the-art omnimodal models on omnimodal under- standing benchmarks, including OmniBench, WorldSense, and AV-Odyssey. Here, "T", "V", and "A" represent text, vision, and audio modality inputs, respectively. We mark the best perfor- mance in bold.
OmniBBnnch
WorldSense
AV-Odyssyy T+A+V
Model
AVG.
T+VV
T+A
T+A+V
A
T+A
T+A+V
AnyGPT (Zhan et al., 2024)
20.1
16.2
18.0
16.5
17.2
16.2
OmniFlow (Li 2025c)
20.4
18.7
20.6
20.4
18.9
19.7
Video SALLONA (Sun al., 2024a)
34.9
35.9
35.6
25.3
Unified[02--agg (Lu ct 2024)
29.1
29.1
27.1
25.2
20.9
23.3
26.0
25.8
Lnifeedi02-xlarge (Lu et 2024)
34.8
31.2
38.0
23.4
20.7
24.7
26.3
28.4
Unffeed002-xxarge (Lu ct 2024)
33.5
32.5
34.0
25,9
23.7
25.9
27.2
29.0
NEXT-GPT (Wu 2023)
22.1
21.4
24.3
25.5
23,9
OncLLM (Han et al., 2024)
32.3
28.7
30.5
23.0
28.6
22.8
27.4
27.6
VideoLLaMM2 (Cheng al., 2024)
31.7
26.2
28.9
23.8
28.5
25.4
26.8
27.3
VITA
2024)
33.5
30.1
33.1
30.5
31.2
26.4
31.0
VIIA 1.5 (Fu et al., 2025)
34.7
31.2
33.4
32.9
37.5
36,9
30.6
33,9
OpenOmin al., 2025a)
37.4
32.8
36.5
NEXT-OMNI
41.4
39.5
40.7
37.2
42.1
40.5
36.4
39.7
dataset. To further enhance the model's understanding, reasoning, and generation reasoning capa- bilities, we obtain 4M high-quality reasoning capability enhancement data through MMEvol (Luo al., 2024b) sampling and filtering, and synthesize 5M reasoning-generannn FLUX (Labs, 2024) for image generation improvement. Based on high-quality instruction fine- tuning data, NEXT-OMNI achieves superior performance across multiple evaluation benchmarks.
image data based on
3.2 OMNIMODAL UNDERSTANDING
To assess the omnimodal capabilities of NEXT-OMNI, we conduct evaluations against current state- of-the-art omnimodal large language models (OLLMs) across three canonical benchmarks, includ- ing OmniBench (Li et al., 2024e), WorldSense (Hong et al., 2025), and AV-Odyssey (Gong et al., 2024a). As shown in Table 1, NEXT-OMNI exhibits superior or comparable performance relative to advanced autoregressive-based OLLMs under various modal combination input conditions. In comparison with OpenOmni, our model achieves 3.2 absolute average performance improvement across the three datasets. These results indicate that discrete flow matching (DFM) demonstrates potential as a viable alternative to the autoregressive (AR) paradigm for omnimodal modeling.
3.3 SPEECH INTERACTION
Table 2: Comparison with existing state- of-the-art unified speech-language model on multi-furn speech interaction bench- marks Spoken QA. Here, "T" and "S" rep- resent text and speech (belonging to the au- dia modality) inputs, respectively. We mark the best performance in bold and the second- best performance with an underline.
To verify NEXT-OMNI's capabilities in multi- turn speech interaction, we conduct experiments on knowledge-based LLaMA Question and Web Question, covering both speech-to-text (S-T)) and speech-to-speech (S+S) tasks. As shown in Ta- 2, under training with multi-turn speech in- struction data of similar scale, compared AR- based
models such as Stream- Omni (Zhang al., 2025b), NEXT-OMNI demon- strates competitive knowledge-based speech interac- tion capabilities on Spoken QA. This indicates that DFM-based omnimodal models can handle complex scenarios of multi-turn speech interaction, providing strong support for unified omnimodal generation and understanding tasks based on DFM.
Model
Llama
Web Q.
AVG.
S-5T
S-55
S-IT
S-15
5-17
S-15
SpecehGPT (Tung et al., 2023g)
2166
6.5
14.1
Mosti (Delowez et 2024)
623
21.0
26.5
9.2
44.5
15.1
GLM 4LM4 Vuice (Zeng al. 2024)
66.7
50.7
32.2
13.9
48.5
33,3
Freeze Omm (Wing et 2024c]
72.0
46.7
58.4
LLMMA Onni (Pang et al,, 2024)
67.7
49.0
33.4
23.7
50.6
36.4
VITA 1.5 (Po al. 2025)
76.7
42.7
49.7
Strcam-Dami (Zhang Openthoni (Lao et 2025g)
30256)
26.3
65.0
44.2
27.5
61.3
46.3
746
67.2
44.5
26.9
39.6
48.1
NEXT-OMNI
78.4
66.4
45.6
28.3
62.0
47.4
3.4 VISION INTERACTION
To explore the high-level capabilities of NEXT-OMNI multi-turn vision interaction, we evaluate it the interleaved image-text generation benchmark OpenING (Zhou et al., 2025). This bench-

---

Published as a conference paper at ICLR 2026
Table Comparison with existing -ftt-oof- unified vision-language model on multi-turn vision interaction benchmarks OpenING. We mark the best performance in bold.
GPT Evaluation
Intjudge Evaluation
Model
AVG.
FDT
w/o Tie
w/ Tie (0)
w/ Tie (.5)
FDT
w/o Tie
w/ Tie (0)
w/ Tie (.S)
MiniGPT-5 (Zheng al., 2023)
28.6
28.4
28.0
28.7
24.5
9.9
27.9
23.9
NEXT-GPT (Wu
2023)
22.6
22.4
22.1
22.7
31.0
21.7
13.4
32.6
23.5
DEEM (Luo et al.. 2024a)
25.6
25.4
25.2
25.9
31.2
21.3
13.6
32.3
25.1
Show-o (Xie
2024)
30.6
31.5
12.5
27.4
Emu2 (Sun et al., 2024b)
41.7
41.6
40.6
41.9
36.3
33.8
21.9
39.5
37.2
SEED-LLaMA (Ge
2023b)
41.0
40.9
40.5
41.0
50.1
47.7
31.6
48.5
42.7
VILA-U al., 2024c)
50.5
50.1
50.3
50.5
$1.4
51.2
32.3
50.9
48.4
Anole (Chem et al. 2024)
53.4
53.1
52.6
53.1
53.4
$2.0
33.9
51.3
50.4
SEED-X et 2024)
55.1
MMaDA (Yang et al., 2025)
51.4
52.6
51.1
52.5
47.6
47.2
31.8
47.4
47.7
FUDOKI (Wang et al., 2025)
47.6
49.2
47.8
48.6
44.4
44.1
30.1
44.2
44.5
NExT-OMNI
58.3
58.6
56.3
37.5
55.4
55.0
mark requires models to perform multi-turn interleaved image generation hased on input content and employs additional judge models for content consistency scoring, challenging the model's abil- ity to naturally determine image generation positions and contextual understanding capabilities. As shown in Table 3, compared to vision-language unified models MMaDA (Yang et al., 2025) and FUDOKI (Wang et al., 2025) with similar architectures, NEXT-OMNI demonstrates superior perfor- mance in multi-turn interactive generation for real- world usage scenarios, reflecting NEXT-OMNI's advantages in general capabilities under multi-turn real-world contexts. Furthermore, compared to AR-based classical works VILA-U (Wu et al., 2024c) and SEED-X (Ge et al., 2024), NEXT-OMNT also exhibits superior effectiveness, indicating that the DFM strategy possesses considerable poten- tial interactive generation consistency and merits further attention.
3.5 MULTIMODAL RETRIVAL
To provide more insights into the impact of paradigms and unified representations in broader multi- modal task scenarios such multimodal retrieval, we adopt the MM-Embed (Lin et al., 2024) ap- proach to sample 100K subset from the dataset M-BEIR (Wei et al., 2024) for multimodal retrieval training. Specifically, for input multimodal queries and retrieval candidates, we extract features from the <EOS> token after model encoding for multimodal retrieval ranking fine-tuning, and test on multiple multimodal retrieval benchmarks, including InfoSeek (Chen et al., 2023), OVEN (Hu et al., 2023), FashionIQ (Wu et al., 2021), and CIRR (Liu al., 2021). We select classical works (Wu et al., 2024a; Deng al., 2025; Wang al., 2025; Wu al., 2024c; Xie et al., 2024; Yang et al., 2025) with different paradigms and representations, and report Top retrieval accuracy in Table 4.
We observe two phenomena. First, models based on discrete flow or diffusion (such as FU- DOKI (Wang et al., 2025) and MMaDA (Yang et al., 2025)) outperform AR hybrid architecture models (Wu et al., 2024a; Deng et al., 2025; Xie et al., 2024; Wu et al., 2024c; Ye et al., 2025c). We attribute this to the fact that corrective bidirectional information encoding training methods better aggregate contextual multimodal information compared to AR architectures based on causal masking mechanisms. During single feature extraction, they degrade BERT-like feature extrac- tion approaches (Lee et al., 2024), providing superior multimodal representations and demonstrat- ing broader application potential nisms (multiple encoders and MOT (Liang et al., 2024) mechanisms like Bagel) performs better on multimodal understanding and generation tasks, compared to unified representation methods (Wu et al., 2024c; Xie et al., 2024; Yang et al., 2025; Li et al., 2024c; Zhang et al., 2026), they es- sentially involve routing between different models. The encoded features remain overly separated, making it difficult to produce unified representations, resulting in suboptimal performance on fea- ture similarity- of these methods. Based these two findings, NEXT-OMNI, which employs unified representa- tion and DFM paradigm modeling, also demonstrates considerable potential application scenarios beyond multimodal generation and understanding.
DFM. Another finding that while using decoupling mecha-
multimodal retrieval tasks, which conversely limits the application scenarios
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison with existing classic unified models on various multimodal retrieval benchmarks, including InfoSeek, OVEN, FashionIQ, and CIRR. Here, "T", "N", and "V" repre- sent text, vision, and audio modality inputs, respectively. We mark the best performance in bold.
Parudigm
Rep.
InfaSeck
OVEN
FachionIQ
CIRR
V+T-+T
V-T-V+I
V+T-+T
V+T-V+T
V+I-V
Janas (Wu et al., 2024a)
AR
Decoupied
21.3
35.4
22.1
37.8
12.4
30.1
26.6
Bagel (Deng et al,, 2025)
AR Diff.
Decoupied
23.1
38.2
24.5
39.6
13.1
32.4
28.5
FUDOKI (Wang et al., 2025)
DFM
Decoupled
25.4
40.0
25.3
41.6
15.5
34,9
30.3
VILA-U (Wa al., 2024c)
AR
Unified
23.3
37.4
24.0
38.5
13.6
33.8
28.4
Show-o (Xie
Diserete
Unilied
MMaDA (Yang al. 2025)
Cnitedd
NEXT-OMNI
DFM
Cnitted
27.6
41.5
27.1
44.6
18.9
37.6
32.9
Table Ablation study on several key components of NEXT-OMNI. Here, "S" represents speech (belonging to the audio modality) inputs. We mark the best performance in bold.
Und.
Gen.
Retrival
Paradigm
Rep.
DGS.
Recon.
AVG.
VQA
AudioCaps
GenEval
Spoken QA (S
InfoSeck
OVEN
AR
Decoupled
x
X
55.2
62.8
53.4
16.4
28.3
32.1
41.4
DFM
Decoupled
60.1
DFM
Unified
51.7
59.4
59.2
19.5
32.8
35,6
43.0
DFM
Unifted
x
54.3
61.9
59.4
19.8
33.1
35.4
43,9
DFM
Unified
V
56.2
63.4
62.6
21.7
33.7
36.1
45.6
3.6 ABLATION STUDY
We conduct ablation studies on key components of NEXT-OMNI, including the modeling paradigm, representation methods, dynamic generation strategy (DGS), and reconstruction item. Bench- marks include image understanding (VQA (Goyal et al., 2017)), audio understanding (Audio- Caps (Kim al., 2019)), image generation (GenEval (Ghosh al., 2023)), speech generation (Spo- ken QA (Fang et al., 2024)), and multimodal retrieval (InfoSeek and OVEN).
Compared to the AR-based baseline with decoupled representations (i.e., inderstanding-oriemn encoder and a generation-orient -oriented encoder), replacing the training paradigm with DFM (see Table 5) leads to a decline understanding performance, but yields notable improvements in generation and retrieval tasks. When shifting further to unified representations, conflicts emerge due to the different granularity requirements of generation and understanding, resulting in an overall performance drop. Nevertheless, fcature-base multimodal retrieval still benefits under this setting. These observations are consistent with our earlier findings, suggesting that unified representations may hold greater potential for broader applications.
When we introduce the DGS during training to better serve text generation tasks requiring dynamic length generation capabilities, we can significantly improve the performance on multimodal under- standing tasks, achieving competitive performance with AR models. When we incorporate modality reconstruction loss terms during training, the model's performance significantly enhanced, while also providing some gains for understanding tasks. This indicates that reconstruction add more lowelevel fineggrained information constraints to features encoded by visual encoders, alleviating the model's bias toward excessive focus on high-level semantic infor- mation, thereby enhancing fine- ein-igraeedd information in unified representations and providing good support for subsequent generation and retrieval tasks.
generation and retrieval tasks
4 CONCLUSION
In this paper, we introduce NEXT-OMNI, the first omnimodal foundation model fully built on dis- crete flow matching, which supports understanding, generation, and retrieval across text, images, video, and audio within unified architecture. By incorporating eccontrnctioonffedbak--elttt unified representations and dynamic-lenge generation strategies, NEXT-OMNI achieves deep fu- sion of multimodal features while substantially reducing model complexity. This design not only strengthens generation, understanding, and retrieval capabilities, but also establishes a new paradigm for unified multimodal modeling. Extensive experiments demonstrate the effectiveness of NExT-

---

Published as a conference paper at ICLR 2026
OMNI and provide insights into how architectural design interacts with unified representations in multimodal tasks. Looking ahead, we plan to extend NEXT-OMNI to broader domains, such as action trajectory generation in ssioo-language-actoon models and video generation for physical Al understanding in world models, where we expect it to play an even greater role.
ACKNOWLEDGEMENTS
Min Yang was supported by National Key Research and Development Program of China 20024YFO9082000, National Natural Science Foundation of China (Grant No. 62376262), and Nat- ural Science Foundation of Guangdong Province of China 1024A1515030160 2025B151
This research/project is supported by the National Research Foundation, Singapore under its Na- tional Large Language Models Funding Initiative (AISG Award No: AISG-NML.22024-000 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do reflect the views of National Research Foundation, Singapore.
ETHICS STATEMENT
This work advances unified omnimodal large language models by introducing discrete flow match- ing modeling paradigms and reconstruction-ennanne unified representations, enhancing multi- modal understanding, generation, and retrieval capabilities, enabling efficient text-vision-audio in- tegration for applications such as assistive tools, creative content, and education. However, high- quality image and audio generation also poses risks, including potential misuse for misinformation or manipulation. While our model is not Alentity-specffic downstream applications should include safeguards such watermarking and prompt filtering. We advocate for ethical use, emphasizing fairness, robustness, and transparency.
REPRODUCIBILITY STATEMENT
Experimental settings are carefully described and listed in Section 3. We detail the model design, data curation, and supplement implementation details in Appendix D, Appendix E, and Appendix F, respectively. To further ensure reproducibility, release training details, data protocols, and open- source both the code and model checkpoints.
REFERENCES
Joshua Ainslie, James Lee-Thee hee-Th Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.. 132.. 2023.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2 10.002050 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. OpenAl blog, 2023.
10

---

Published as a conference paper at ICLR 2026
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Saehoon
Kim. C0yo-700m: Image-text pair dataset. patps://githubee com. http://ggtt.bbccccccccaaa/////// kawp/kkaaabaiin////
coyo-dataset,299 2022.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR. pp. 961-970, 2015.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In ACI, pp. 190-200, 2011.
Jiuhaí Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architeccure,, training and dataset. arXiv preprint arXiv:2505.0955 095688 2025a.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion trans- former for 4k text-to-image generation. In ECCV, 2024a.
Junying Chen, Zhenyang Cai, Pengcheng Chen, Shunian Chen, Ke Ji, Xidong Wang. Yunjin Yang, and Benyou Wang. Sharegpt-4o-imagee Aligning multimodal models with gpt-4o-level image generation. arXiv preprint (11i:::550610990 2025b.
Kai Chen, Yunhao Gou, Runhui Huang. Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409, 18042, 2024b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, pp. 370- 387, 2024c.
Xiaokang Chen, Chengyue Wu, Zhiyu Wu, Yiyang Ma, Xingehao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus-pro: Unified multimodal understand- ing and generation with data and model scaling. arXiv preprint arXiv:2501 178111 2025c.
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Chang. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302. 11713, 2023.
Ming-Wei Wei
Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pp. 794-803,
2018.
Zesen Cheng. Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint wwX.::2240000777777 2024.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arxiv:2407.06/133 2024.
Yunfei Chu, Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023.
Common Crawl. Common crawl open repository of web crawl data., 2007. URL https:// // commoncrawl. org/.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose with instruction tuning. In NewrIPS, pp. 49250-49267, 2023.
purpose vision- Lasouulanguuag models
Google DeepMind. Gemini 2.0: Enhanced multimodal world models. https: httee://blloo /bllg.
google/ 2025. Accessed: 2025-04-06.
11

---

Published as a conference paper at ICLR 2026
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dia- logue. arXiv preprint arXiv:2410.0005/ 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv e-prints, pp. arXiv-2409, 2024.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Wei- hao Yu, Xiaonan Nie, Ziang Song. Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In ICLR, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.06 054.0, 2024.
Chenghao Fan, Wen Heng. Bo Li, Sichen Liu, Yuxuan Song. Jing Su, Xiaoye Qu, Kai Shen, and Wei Wei. Stable-diffcodee Pushing the frontier of code diffusion large language model. arXiv preprint arXiv:2601. 15892, 2026.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-ommi: Seamless speech interaction with large language models. arXiv preprint urXiv:2409.06666. 2024.
Rongyao Fang, Aldrich Yu, Chengqi Duan, Linjiang Huang. Shuai Bai, Yuxuan Cai, Kun Wang, Si Liu, Xihui Liu, and Hongsheng Li. Flux-reason-6m prism-bench: A million-scale text-to- image reasoning dataset and comprehensive benchmark. arXiv preprint (r0Xi::5000096500 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao Yifan Zhang, Xiong Wang. Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal 11m. arXiv preprint arXiv:2408.052: 2024.
Chaoyou Fu, Haojia Lin, Xiong Wang. Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.0199 2025.
Yuying Ge, Yixiao Ge. Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprini urXiv:2307.0804 2023a.
Yuying Ge Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilaaa see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023b.
Yuying Ge. Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- multi-granularity comprehension and generation. arXiv preprint arxiv:2404. / 14396, 2024.
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating text-to-image alignment. In NewrlPS, 2023.
12

---

Published as a conference paper at ICLR 2026
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction. arXiv preprint 1rXf1:2404.19737, 2024.
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng. Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, al. Av-odyssey bench: Can your multimodal 1lms really understand audio-visual information? arXiv preprint arXiv:24/2.02611 2024a.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410. 1789/ 2024h.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904-6913, 2017.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework align all modalities with language. In CVPR, pp. 21584-26595, 2024.
Jiaming Han, Hao Chen, Yang Zhao, Hanyu Wang, Zhao, Ziyan Yang, Hao He, Xiangyu Yue, and Lu Jiang. Vision as a dialect: Unifying visual understanding and generation via text-aligned representations. arXiv preprint wxXiv2256./8889 2025.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598. 2022.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating
real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint 1xXi::2005.15666 2022.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal. Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. ICCV, pp. 12065-12075, 2023.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment, 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, pp. 21807-21818, 2024.
IDEFICS. Introducing idefics: An open reproduction of state-of-the-art visual language model. https: htthhhgggggggacccec co/blog/ idefics, 2023.
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint ar arXiv:2408.1653 16532, 2024.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin CHEN, Chengru Song, dai meng. Di ZHANG, Wenwu Ou, Kun Gai, and Yadong MU. Unified language- vision pretraining in LLM with dynamic discrete visual tokenization. In ICLR, 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4410 2019.
13

---

Published as a conference paper at ICLR 2026
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In ACL, pp. 119-132, 2019.
Black Forest Labs. Flux, 2024. URL http:///tt////atttaa..////aeeeea
flux.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms generalist embedding models. arXiv preprint S.di::2405.14428 2024.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Allava-onevision Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Daiging Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402. 1724442,... 2024b.
Jijie Li, Li Du, Hanyu Zhao, Bo-wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis to enhance language models. arXiv preprint arXiv:2506. 11116, 2025a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023a.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, et al. Omni- corpus: A unified multimodal corpus of 10 billion-level images interleaved with text. In ICLR, 2025b.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Omniflow: Any-to-any generation with multi-modal rectified flows. CVPR, 2025c.
pp.
Wenbing Li, Hang Zhou, Junqing Yu, Zikai Song, and Wei Yang. Coupled mamba: Enhanced multi-modal fusion with coupled state space model. arXiv preprint arXiv:2405. 18014, 2024c.
Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang. Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with Ilama-3? arXiv preprint arXiv:2406. 084788 2024d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 2023b.
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint trXiv:2209.152777 2024e.
Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In CVPR, pp. 4641-4650, 2016.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffuusin for unified image generation and understanding, 2024f. URL https: //axxii.//axvv.. org/abs/ aaa.bb/a/ 2000 00289.
Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, al. Mixture-of -fftuurr---trannsfommesss A sparse and scalable architecture for multi-modal foundation models. arXiv preprint arXiv:2411.04996 2024.
Bin Lin. Yang Ye, Bin Zhu, Jiaxi Cui. Muman Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
14

---

Published as a conference paper at ICLR 2026
Sheng- Chich Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal 1lms. arXiv preprint arXiv:2411.02571, 2024.
Sheng-Chieh
Yaron Lipman, Ricky TQ Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprini arXiv:2210.02747, 2022.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arxiv:2402.08268 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint urXiv:2310033744, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2024b.
Xiaohao Liu, Xiaobo Xia, Jiaheng Wei, Shuo Yang, Xiu Su, See-Kiong Ng. and Tat-Seng Chua. Calibrated multimodal representation learning with missing modalities. arXiv preprint arXiv:2511.12034 2025a.
Xiaohao Liu, Xiaobo Xia, Weixiang Zhao, Manyi Zhang, Xianzhi Yu, Xiu Su, Shuo Yang, See- Kiong Ng, and Tat-Seng Chua. L-mtp: Leap multi-token prediction beyond adjacent context for large language models. In NeurlPS, 2025b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmhench: Is your multi-modal model an all-around player? In ECCV, 2024c.
Zheyuan Liu, Cristian RodriguezzOpazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained Aisionanddlanguuee models. In ICCV, pp. 2125-2134, 2021.
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint arXiv:2506.0622 2025c.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, pp. 26439-26452, 2024.
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232, 2024a.
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang. Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. Mmevol: Empowering multimodal large language models with ivol-instruct. arXiv preprint arXiv:2409.0555 05840. 2024b.
Run Luo, Ting-Ei En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. Openomni: Advancing apen-source omnimodal large lan- guage models with progressive multimodal alignment and real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.0456/ 2025a.
Run Luo, Renke Shan, Longze Chen, Ziqiang Liu, Lu Wang, Min Yang, and Xiaobo Xia. Vem: Vision concept modeling based on implicit contrastive learning with vision-language instruction fine-tuning. arXiv preprint arXiv:2504.1960 2025b.
Weijiang Lv, Yaoxuan Feng, Xiaobo Xia, Jiayu Wang, Yan Jing, Wenchao Chen, and Bo Chen. Spd-faith bench: Diagnosing and improving faithfulness in chain-of-thought for multimodal large language models. arXiv preprint arXiv:2602.0 07836, 2026.
Chuofan Ma. Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint 203321, 2025a.
15

---

Published as a conference paper at ICLR 2026
Yiyi Ma, Yuanzhi Liang. Xiu Li, Chi Zhang, and Xuelong Li. Intersyn: Interleaved learning for dynamic motion synthesis in the wild. arXiv preprint arXiv:2508. 10297, 2025b.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint WrXiv:2306.05424, 2023.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang. Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.0233 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint S0V:2502.00992, 2025.
OpenAL. Sora: A data-nriven physical engine for world modeling. https://openai.cmmm 2025. Accessed: 2025-04-06
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb: A benchmark for generative image understanding, 2023.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, pp. 5206-5210, 2015.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:2307.01952. 2023.
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang. Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Ze- huan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03065 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, pp. 28492-28518, 2023.
Antony Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In ICASSP, pp. 749-752, 2001.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022b.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint 021552 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b; An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Kar- rer, Yaron Lipman, and Ricky TQ Chen. Flow matching with general discrete paths: A kinetic- optimal perspective. arXiv preprint arXiv:2412.03487. 2024.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406. /5704, 2024a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In ICLR, 2023.
16

---

Published as a conference paper at ICLR 2026
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR, 2024b.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09811 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arxiv:2312. 2023.
Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arxiv:2401. 10208, 2024.
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arxiv:2302.13977 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. In NeurlPS, 2017.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.220 20314, 2025.
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your lims see, draw, and self-enhance. arXiv preprint arXiv:2412.06673, 2024a.
Jin Wang, Yao Lai, Aoxue Li, Shifeng Zhang, Jiacheng Sun, Ning Kang, Chengyue Wu, Zhenguo Li, and Ping Luo. Fudoki: Discrete flow-hased unified understanding and generation via kinetic- optimal velocities. arXiv preprint arXiv:2505. 2025.
Wenhao Wang and Yi Yang. Tip-i2v: A million-scale real text and image prompt dataset for image- to-vidco generation. arXiv preprint arXiv:2411.04709, 2024.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprini a4xiv:2409. 18869, 2024b.
Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen 1lm. arXiv preprint arXiv:2411 1.0417100744 2024c.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In ECCV, pp. 387-404, 2024.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848 2024a.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: new dataset towards retrieving images by natural language feedback. In CVPR. pp. 1 11307-1131, 2021.
Junfeng Wu, Yi Jiang, Chuofan Ma. Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable modll generators. arXiv preprint arXiv:2412.0433 04333, 2024b.
17

---

Published as a conference paper at ICLR 2026
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal Ilm. arXiv preprint (rxiv:2309.055199 2023.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation. 2025. URL https::///. : arxiv http://wwwvvl.......// wttpoorgg//a//oo/eooeee 21979.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang. Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint ar Xiv: 0044999 2024c.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseck-v12: Mixture Mixture-of-cxpeee ox-exts usioo-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10300 2024d.
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu. Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of thining-time and inference- time compute in linear diffusion transformer. arXiv preprint arXiv:2501. 11.225001.84227 /8427, 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou, Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arxiv:2408. /2528. 2024.
Zhifei Xie and Changqiao Wu. Mini-ommi2: Towards open-source gpt-4o model with vision, speech and duplex. arXiv preprint arXivv2240.111 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In ICLR, 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, pp. 1645-1653, 2017.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.2022 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi. Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17665 2025b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprini 242.:2412.1551555 2024.
Ling Yang. Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. 15487, 2025a.
Junyan Ye, Dongzhi Jiang, Zihao Wang, Leqi Zhu, Zhenghao Hu, Zilong Huang, Jun He, Zhiyuan Yan, Jinghua Yu, Hongsheng Li, Conghui He, and Weijia Li. Echo-4o: Harnessing the power of gpt-4o synthetic images for improved image generation. https://arzivorr//ab/// 09988, 2025b.
Liliang Ye, Yunyao Zhang, Yafeng Wu, Yi-Ping Phoebe Chen, Junqing Yu, Wei Yang, and Zikai Song. Mvp: Winning solution to smp challenge 2025 video track. arXiv preprint a0Xiv:2507.00950. 2025c.
18

---

Published as a conference paper at ICLR 2026
Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, et al. Llasa: Scaling train-time and inference- time compute for llama-base based speech synthesis. arXiv preprint arXiv:2502.0411 041288 2025d.
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICMI, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, pp.
2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024.
URL httph::p//// http://axivvv./ org/ abs //o//2411 2412. 02612.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arxiv:2402. 12226, 2024.
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP, pp. 6182-6186, 2022.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Findings of EMNLP, pp. 15757-15773, 2023a.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction- tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02855, 2023b.
Haonan Zhang, Run Luo, Xiong Liu, Yuchuan Wu, Ting-En Lin, Pengpeng Zeng, Qiang Qu, Feiteng Fang, Min Yang, Lianli Gao, et al. Omnicharacter: Towards immersive role-playing agents with seamless speech-language personality interaction. ACL 2025, 2025a.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, pp. 586-595, 2018.
Shaolei Zhang, Shoutao Guo, Qingkai Fang, Yan Zhou, and Yang Feng. Stream-omni: Si- multaneous multimodal interactions with large language-ission-spechh model. arXiv preprint arXiv:2506.13644 2025b.
Xinglang Zhang, Yunyao Zhang, ZeLiang Chen, Junqing Yu, Wei Yang, and Zikai Song. Log- ical phase transitions: Understanding collapse in I1m logical reasoning. arXiv preprint arXiv:2601.02902 2026.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research, 2025c.
Hao Zheng, Zirui Pang, Zhijie Deng, Yuhan Pu, Zhaowei Zhu, Xiaobo Xia, Jiaheng Wei, et al. Offside: Benchmarking unlearning misinformation in multimodal large language models. arXiv preprint arXiv:2510.2255: 2025.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language genera- tion via generative vokens. arXiv preprint arXiv:2310.02239, 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma. Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arxiv:2408. 11099, 2024.
Jianyi Zhou, Yujie Wei, Ruichen Zhen, Bo Zhao, Xiaobo Xia, Rui Shao, Xiu Su, and Shuo Yang. Inject once survive later: Backdooring vision- laaioon-annuug--acttt models to persist through down- stream fine-tuning. arXiv preprint 2602.000000000000 2026.
19

---

Published as a conference paper at ICLR 2026
Pengfei Zhou, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang, Ziyao Guo, Hao Zhang, Yuqi Lin, Yefei He, et al. Opening: A comprehensive benchmark for judging open-ended interleaved image-text generation. In CVPR, pp. 56, 2025.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. In NeurlPS, pp. 9958-8974 2023.
20

---

Appendix
Preliminaries of Discrete Flow Matching
22
Related Work
22
C Limitation and Discussion
23
D Model Design
24
D.I Modality Encoders
24
D.2 Modality Heads
25
Data Curation
27
E.1 Dataset Curation Details
27
E,2 Dataset Synthetic Details
27
Additional Implementation Details
27
Additional Ablation Study
28
H Additional Experiments Results
28
I Additional Visualization
31
J The Use Large Language Models
32
21

---

Published as a conference paper at ICLR 2026
A PRELIMINARIES OF DISCRETE FLOW MATCHING
We provide a brief introduction to the mathematical processes of training and inference in discrete flow matching (DFM) (Shaul al., 2024). Analogous to continuous flow matching (Lipman al., 2022), DFM defines family of time-dependent probability distributions {pr (x) he(o. that define smooth transition, or probability paths, from source distribution p(x) to target distribution (xx). Here, {x' x² ...... x)) lies the discrete space S = T where D denotes the number of discrete variables and T = [ 21, . represents the finite set of possible discrete values. Each (x(x) is constructed as pe (x( ExIes (x (x(x), where the conditional distribution is
factorized across dimensions. Namely, (x(x)x]) = I Px [x]] Note that each pa (x*1xxx) (x]) defines
univariate interpolation between base distribution p(x') and one-hot distribution 6x; (x'). A
common design for the interpolation is the mixture path, defined via time-dependen. scheduler (x(x) E [0, i.e.
(x = (1 (1-xx(x])) (x1=(((())p(('') + (x) )0f (x'),
(4)
where ko(+) = O and Kn (+) = 1. To obtain a more semantically meaningful transition path, we can adopt metric-induced probability path follows:
(x [x]) Softmax(-3:d(x} xx))),
(5)
where d(.,-) is the cosine distance function between token embeddings, and Br E [0, oo] is mono- tonic schedule.
After minimizing the kinetic energy (Shaul et al., 2024), the probability velocities can be formulated as follows:
(x(x',, = = (xf) 03, Ōt
max {d(z', x)) - d(x', xi).
Intuitively, for the i-th coordinate z' E T, this velocity ensures that probability mass flows from the state z' to the state x² only when x² lies closer to x than z' does, i.c., d(x', xx) < d(z', ((zz'x]). As result, the flow monotonically progresses toward xf.
During training, we sample x from (x( and feed it into the model to fit the target X1. During inference, we employ an Euler solver for enhanced sampling robustness as recommended in (Shaul et al., 2024). This solver simulates the continuous-time Markov chain (CTMC) process {x, he (,1)- Given that x ~ Pt, the solver updates the i-th coordinate from time t to t++h following the procedure below:
Sample xf ~ (11xg) from our model;
Compute the total conditional transition rate X Exieee (xt,,( (see Eq. (6));
Draw uniform random variable 7 Tchange 410, 1]:
Sample x/44, as follows: if Lihange S expl-hX1 sample x from MA (xs) (-);: otherwise set x x/.
(1
X
The procedure begins with completely corrupted sequences and iteratively denoises entire sequences in parallel, enabling richer bidirectional information integration to enhance final performance. We
set 3r = E
with = 3 and a = 0.9, as suggested in (Shaul et al., 2024).
B RELATED WORK
Unified LaaguaLangguage Models. Unified vision-language models (Team, 2024; Dong et al., 2024; Tian et al., 2024; Luo et al., 2024a; Wu et al., 2024b) have attracted significant research attention due to their powerful multimodal understanding and generation capabilities. SEED and the Emu se- ries (Ge et al., 2023b; 2024; Sun al., 2023; 2024b; Wang al., 2024b) adopt discrete autoregres- sive modeling approaches, unifying multimodal understanding and generation through next-token
Iff x' xi. 6. (x')=1; clse, 82g (x')=0.
22

---

Published as a conference paper at ICLR 2026
prediction paradigms. However, this coarse unification approach is susceptible to granularity mis- matches between understanding and generation tasks, hindering further performance improvements. The Janus series models (Wu et al., 2024a; Chen et al., 2025c) decouple visual encoding for under- standing and generation to address granularity mismatch issues, but introducing additional encoders limits flexibility. Subsequently, VILA-U (Wu et al., 2024c) and UniTok (Ma et al., 2025a) focus on constructing unified representations, using unified encoders to alleviate granularity conflicts be- tween generation and understanding tasks, but still adhere to autoregressive modeling paradigms and lag behind specialized models (Rombach et al., 2022a; Xie et al., 2025) in generation tasks. Some works, such as BLIP-3o (Chen et al., 2025a), introduce additional diffusion- generation models for generation optimization to achieve impressive results, but further increase methodological complexity. To balance the effectiveness of generation and understanding tasks, some works attempt to use hybrid architectures. For instance, Show-o (Xie et al., 2024) and Trans- fusion (Zhou et al., 2024) integrate diffusion objectives into large language models for image gen- eration, but this design breaks the autoregressive paradigm and complicates the unification of both tasks. Bagel (Deng et al., 2025) introduces MOT (Liang et al., 2024) architecture to successfully achieve excellent performance in both tasks within relatively unified hybrid architecture, but the introduction of these decoupling mechanisms makes generation and understanding separate compo- nents, departing significantly from the ideal of concise unified architecture. Some works (Yang et al., 2025; Wang et al., 2025) attempt to completely abandon autoregressive architectures, pursu- ing unified rision-languag modeling from the perspective of discrete diffusion or flow matching, achieving considerable results. However, due to the lack of robust language foundation model sup- port and engineering optimization, speed and effectiveness remain suboptimal.
specialized
Omnimodal Language Models. With the advancement of multimodal research, models are increas- ingly shifting toward unified frameworks that seamlessly integrate diverse input and output modali- ties. By tokenizing different data types into shared representations, models such as AnyGPT (Zhan et al., 2024) and Unified-1O2 (Lu et al., 2024) achieve seamless cross-modal task adaptability, en- abling them to process audio, text, and images without requiring significant architectural modifi- cations. OneLLM (Han et al., 2024) and NEXT-GPT (Wu et al., 2023) enhance generation and understanding performance by unifying input spaces and introducing additional diffusion heads. Meanwhile, MiLeNN (Sun et al., 2024a) enhances video understanding by incorporating fine-grained temporal modeling, improving the model's ability to interpret speech and actions within videos. To enhance human-computer interaction, the VITA series (Fu et al., 2024; 2025) introduces duplex communication schemes, enabling fluid and intuitive exchanges between users and AI mod- els. EMOVA (Chen et al., 2024b) and OpenOmni series (Luo et al., 2024b; 2025a; Zhang et al., 2025a) further extend the expressive capabilities of multimodal systems by integrating controllable emotional speech synthesis, providing more natural and engaging user interactions. The Qwen- Omni series (Xu et al., 2025a;b) further expanded the scale of training models and data, greatly enhancing the performance of the omni models. However, these works still rely autoregressive architectures and additional large-scale continuous flow matching modeling heads for omnimodal modeling, while more unified and lightweight discrete flow matching and diffusion architectures remain unexplored. To address this gap, we propose NEXT-OMNI in this work.
C LIMITATION AND DISCUSSION
Limitation. Due to resource constraints, we conduct training and validation only at the 7B param- eter scale with 2T tokens. While NEXT-OMNI provides insights into how discrete flow matching can better unify generation, understanding, and retrieval tasks, its full potential has not been demon- strated due to the lack of corresponding large language model foundation support. In the future, we hope to explore broader application scenarios, such as trajectory generation in vision-language- action models and world model exploration, where visual generation assists physical perception, to demonstrate the potential of NEXT-OMNI.
Discussion. Here, we provide further discussion on the future of omnimodal unified models. Some argue that building unified models consumes substantial resources yet struggles to achieve perfor- mance comparable to generation-only and understanding-onf models, questioning the necessity of constructing unified models. We address this concern follows: unified models are built to achieve greater generalizability. In the future, unified omnimodal models will serve as a "world brain" to interact with the real world, with their general capabilities expected to be continuously
23

---

Published as a conference paper at ICLR 2026
Recomtraction
Recondraction.ss
VILT
Vilen Decoder
Audo Esooder
Audo
Escodar
VQ-
Se
Agprop
Sout
Corfeoe
NECTONI Jorund Mull Ovail focdlaie
Aubi Deorny Hanking
Shook
Figure 4: The Pipeline of vision encoder (left) and audio encoder (right) warmup training. Dur- ing the modal encoder warmup training phase, self-supervised reconstruction is performed through additional quantizers and modal decoders, while semantic alignment at different granularities is achieved through text encoders or text decoders.
enhanced through multimodal data collected via interactions, thereby evolving iteratively to ex- pand the model's capabiliiyy boundaries and ultimately achieve Artificial General Intelligence (AGI). Our goal in building unified models is to enable mutual reinforcement among diverse tasks such as understanding and generation, ultimately expanding the boundaries of intelligence. For instance, through extensive vision generation learning, models can deepen their understanding of physical laws by precisely generating imagined moments, subsequently producing reasonable actions. Con- versely, a deeper understanding of physical laws can enable models to generate more realistic image sequences. Based on this rationale, while building unified models may incur certain performance trade-offs, their general capabilities offer more promising prospects. On this spiraling path of devel- opment with its inevitable challenges, we must persist in this direction.
D MODEL DESIGN
Here, we provide additional details regarding the design principles and training process of modality encoders and modality heads.
D.1 MODALITY ENCODERS
Similar to previous work (Luo al., 2024a; Wu al., 2024c; Ma al., 2025a; Ye al., 2025d; Wan et al., 2025; Luo et al., 2025b; Lv et al., 2026; Zhou et al., 2026; Zheng et al., 2025; Liu al., 2025a), we design modality encoders based on unified representation methods, conduct- ing simultancous self-supervised reconstruction and semantic alignment optimization during the warmup training stage. As shown in Figure 4, we employ additional quantizers and modality de- coders to assist reconstruction training, while utilizing text encoders and decoders for sentence- level and token-level semantic alignment training, respectively. To pursue superior performance, adopt multi-codebook quantization (MCQ) (Ma et al., 2025a) for quantization, where separating the codebook into multiple independent sub-codebooks significantly enhances both reconstruction and semantic alignment effectiveness. However, also observe that while increasing the number of sub-codebooks improves performance on reconstruction and downstream multimodal understanding tasks, it degrades performance downstream multimodal generation tasks, predicting multiple sub-codebook indices at single position becomes more challenging. Therefore, to achieve an op- timal trade-off, we set vocabulary sizes of 4x 4096 and 2x2048 for the vision encoder and audio encoder in the warmup training, respectively.
Vision Encoder. We initialize our vision encoder with CLIP- (Dosovitskiy et al., 2021) weights and conduct unified representation training nearly 70M image-text pairs composed of LAION (Schuhmann et al., 2022) and DataComp (Li et al., 2024d), incorporating both sentence- level image-text contrastive loss C and VQVAE-based reconstruction loss Z optimization. In more detail, the VQVAF--baed reconstruction loss consists of pixel-level reconstruction loss CY perceptual loss CV based on the LPIPS metric (Zhang et al., 2018), a discriminator loss C to enhance reconstruction fidelity (Karras et al., 2019). and vector quantization loss L to
minimize distance between the encoder output and its nearest code entry. It is denoted as L
24

---

Published as a conference paper at ICLR 2026
Table 6: Comparison with existing state-of-the-art vision tokenizer and audio tokenizer. We mark the best performance in bold.
Model
Codebooks UTMOS?
PESQ1
STOT:
Model
Codebooks
rFID Acct
WarTokenizer
4096
4.048
2.373
0.914
UniTok
8x409%
0.38
78.6
Audio Encoder (Ours)
2x2048
4.126
2.467
0.923
Vision Encoder (Ours)
4
0.33
79.4
Cg. where À denotes the weight factor for the corresponding loss term. The image-text contrastive loss term Comsttr basically the same as in CLIP-VIT (Dosovitskiy al., 2021). Therefore, the final loss term be written L CV S During this process, we maintain training hyperparameters consistent with UniTTok (Ma et al., 2025a).
C Avo +Ap
Audio Encoder. We initialize audio encoder with Whisper-Turbo (Radford al., 2023) weights and conduct unified representation training on nearly 102K hours of audio-text pairs composed of LibriSpeech (Panayotov et al., 2015), WeNetSpeech (Zhang al., 2022), AudioCaps (Kim al., 2019), and proprietary data, incorporating both token-level audio caption loss Lap and VQVAE- based reconstruction loss L optimization. The VQVAE- reconstruction loss consists of mel-spectrum reconstruction loss L a feature matching loss L based on a I.2 norm loss, a discrim- inator loss & (Ji al., 2024), and vector quantization loss Lig to minimize distance between the encoder output and its nearest code entry. It is denoted as LA = LR + Avo L + Xp L$ Àg GG- where À is the weight factor for the corresponding loss term. The audio-text caption loss term L is basically the same as in Owen--Audio (Chu et al., 2023). Therefore, the final loss term can be written as overall = C + L During this process, we maintain training hyperparameters consistent with WayTokeenize (Ji et al., 2024).
Quantitative and Qualitative Analysis. To validate the effectiveness of our warmup-trained vision encoder, we conduct image reconstruction and classification evaluation on ImageNet (Deng et al., 2009), reporting rFID (IDEFICS, 2023) and zero-shot classification accuracy, and compare with the state-of-the-arr unified representation vision encoder UniTok (Ma al., 2025a). Similar to vision encoder configuration, we perform speech reconstruction comparison tests on the LibriSpeech test-clean split (Panayotov et al., 2015), employing UTMOS (Saeki et al., 2022). PESQ (Rix et al., 2001), and STOI (Ji et al., 2024) metrics, and compare with the state-of-the of-the-arttttt WavTokenizer (Ji et al., 2024). As shown Table 6, our modality encoders demonstrate superior reconstruction and semantic classification performance, capable of producing reliable unified representations for multimodal generation, understanding, and retrieval. Furthermore, to more intuitively demonstrate the superiority of our modal encoders, we provide visualization results of reconstructions. As shown in Figure 6, we can observe that our vision encoder achieves better reconstruction performance compared UniTokin details such as font edges and bear eye colors. As shown in Figure 5, we can see that our audio encoder produces clearer reconstructed mee-speecss for different audio types compared to WavTokenizer. These results validate the effectiveness of our modality encoder warmup training.
Table Ablation study on scalability NEXT-OMNL Here, "S" represents speech (belong- ing to the audio modality) inputs. We mark the best performance in bold.
Model Training Steps
Und.
Gen.
AVG.
VQA
AndioCaps
ActivityNet.
GenFval
Spoken QA (S-$S
VBench
D.5B
0.5K
42.1
48.6
28.3
18.2
12.7
26.4
L,5B
0.5K
45.3
$2.3
31.7
22.8
11.2
16.3
30.0
7B
0.5K
48.7
$5.9
34.2
28.5
13.7
20.1
33.5
52.4
45,2
27.4
39.6
7B
1.5K
54.9
61.7
37.4
58.9
20.1
32.8
44.3
63.4
34.6
D.2
MODALITY HEADS
Since we employ multi-codebook quantization (MCQ) for quantization, prediction requires fore- casting multiple sub-codebook indices for single position, which poses challenges for vision and
25

---

Published as a conference paper at ICLR 2026
tranmtiier: (NADRIN Dinntuu,, feriRruoaami coReenn Aitetl///d
Original
Ours
WovTok
Figure 5: Qualitative results on audio reconstruction in max duration of 15s.
Original
Ours
Unitok
Original
Ours
Unitok
Figure 6: Qualitative results on image reconstruction in resolution of 256x
audio head architectures. To overcome this problem, we design two different modality head struc- tures, as illustrated in Figure 7. The left structure represents an autoregressive index prediction modal head, which uses the hidden features output by NEXT-OMNI each position as conditions to complete the prediction of multiple sub-codebook indices through the next-token prediction paradigm. The right structure employs multiple separate heads to complete the prediction of multiple sub-codebook indices in parallel through multi-token prediction (Gloeckle et al., 2024; Liu et al., 2025b). Under equal parameter counts, the former provides more stable prediction re- sults compared to the latter at the cost of slightly increased computational overhead, and is therefore adopted our framework.
F-3
2.
Head
Head
Troud
Multi Head Atfention
1A
1-
1-a
P-1
2-2
NEXT-OMNI
NEXT-OMNI
Figure 7: Illustration of modality heads. We design two different lightweight modal head struc- tures for multiple sub-codebook indices prediction: next-token prediction (left) and multi-token prediction (right).
26

---

Published as a conference paper at ICLR 2026
Table 8: Overview of stages, training data, tasks, and sizes used for building NEXT-OMNI. Here, "T", "T", "A", and "V" are the abbreviations of the image, text, audio, and vision, respectively.
Model
Stuge
Data
Tesk
Size
Vision Frooder
Wirmup
LAION (Schhhmann ot sl. 2022) DaraComp (Iictall, 20046),0000 (Rycnn al, 2023)
1-41, 1-41, I-+T
70M
Audio Encoder
Warmup
Tihrippochh (Prnsgntov at 2015), WeneeSperch (Phing et 2027) AudiCCug (Kim et al., 2019), Propeistary Dai
A-A.A-T
102K Houns
1 WA..Roaap--C-- et 2016; Common Croul, 2007), PaeaComp eli sh. 20240)
1-+T
32M
PT
Blipt-o (Chen 2015o), ImageNot-1E
etal. 2009), JoumsyDR (Pin 2023)
T-+1
25M
Enstnity-Esstuc (Li 2025g), Evol fastruct (Ku 2004)
Text
4M
LibriSpesdh (Pusgyotov
Proprietry Dara AudioCaps al. 2019)
WenetSpeech GZhang 2012)
A-+T
16M
Propictary AupioCaps
2019)
dla
(Deille M00mmo.A
HI
Pllipt-a(Ches 2015s), Flax-Rosso (Pang 2015)
12M
ShuneGPT4 2023), OmnnCCoppus 20256)
12M
2024c)
OpenVid (Nan al, 2024),
Infinity-astruc (13 2025s), Reol-fastrue ol., 2004)
23M
Proprietary Data, ApdioCaps (Kim et 20/9)
A-+7
12M
OpenOmmi (Lao et al., 20250) AudioCaps (Kim et al., 2019)
T->A
1.3M
LLaVA ( et al,, 2024a), PixMo (Deiike et al,, 2024), MMDvol (Lue et al., 2024b), Und 4M
I-T
7.6M
OpenOmii (Lae et sl, 2025a)
T+A-T=+
0.5M
SFT
Sye (Ma al, 20256)
T+1-T+41
1.7M
fastry- (Li et al, 2025a), Eval-fastrue (X 2004)
Test
0&M
LLaVA Video (Zhing et all, 205cc)
V-+T
0.9
Open (Nan et 2024), TIP (Wang Yang, 2024)
T-V.T+I-V
IM
BLIPS 60K (Chen et al, 2025a), ShareGPT to Image (Chen et 20256)
Gen-SM, Numo-omiiseett (Ye et sl, 2025))
5.M
E DATA CURATION
E.I DATASET CURATION DETAILS
In addition to proprietary data, we collect publicly available data encompassing diverse tasks for three-stage progressive training, including generation-related tasks such as text-to-image, text-to- video, and text-to-audio, as well as understanding-relate tasks, including image understanding, video understanding, and audio understanding. We summarize the important information of all training data in Table 8.
E.2 DATASET SYNTHETIC DETAILS
Visual Understanding. We randomly sample 1.2M data from and PixMo (Deitke et al., 2024) datasets as seed data, and apply the MMEvol (Luo et al., 2024b) algorithm using the Qwen2.5-VI. (Bai al., 2025) model the generator. Through three rounds of evolution, we construct approximately 4M instruction data (Und-4M) that are rich in diversity and complexity, enhance the model's capability for solving complex problems.
(Li et al., 2024a)
Image Generation. Based on user prompts from JourneyDB (Pan et al., 2023) and Midjourney- Prompts datasets, we use the Qwen2.5 (Yang et al., 2024) model to construct numerous similar syn- thetic prompts containing more finegrriied descriptions. Subsequently. we employ FLUX (Labs, 2024) as the image generator to rapidly generate large quantities images based on user and syn- thetic prompts under the configuration of 4 sampling steps and 512x512 resolution. Finally, we collect approximately 5M high-quality synthetic data (Gen-5M).
F ADDITIONAL IMPLEMENTATION DETAILS
We report additional implementation details about training paper-pprameters and training data in
Table 9.
27

---

Published as a conference paper at ICLR 2026
Table 9: Training recipes for NEXT-OMNI. The three training stages are introduced in Section 3.1. 1. stage I: Pre-Train Trainingn (PT), Stage II: Continue Pre-Training (CPT), Stage III: Super- vised Fine-tuning (SFT).
SLage
SLage II
SLage III
Phase
PT
CPT
SFT
Training Hyper- Parametery
Image Resolution
256x256 115
384
384
Audio Duration
155 <8
2:55s
Video Frrmmes
LLM
Qwen2.55 7B
Qwen2.5 7B
Qwen2.5 7B
20-5 (modality encoder&dcoode)) (others)
10-6 (modality encoderádooddr) (others)
1o-6 (modality encodordddodde)
Learning Rate
(others)
Optimizer
A
Adam
Adam
Optimizer hyper sper-parameters
A M, c=0.9. 0.995, 0.05
Br,8, c00.99 0.999, 1e-8
Ar.8,.c033 0.999, 1e-8
Weight Decay
0.05
0.05
Training
Warmup steps
IK
500
500
Learning Rate Scheduler
Cosine
Cosine
Cosine
Batch Size Per GPU
Maximum Token Length
2K
16K
16K
Training Data
Size
-3M Pair
Data Type
Paio/ltterkawe
Instruction
G ADDITIONAL ABLATION STUDY
Although NEXT-OMNI demonstrates competitive performance against the SOTA methods, its seal- ability has yet to be validated. As is well known, scalability is crucial for model performance. We conduct ablation experiments assess the scalability concerning data count and model size. As shown in Table 7, gradually increasing the training data enables the model to successfully scale while achieving improved results. Additionally, increasing the sizes of backbone leads sustained performance enhancements, indicating that NEXT-OMNI possesses good scalability.
H ADDITIONAL EXPERIMENTS RESULTS
Image Understanding. We evaluate the image understanding capabilities of NEXT--XMN on sev- eral benchmarks, including POPE (Li et al., 2023b), MMF-P (Fu et al., 2023). SEED (Ge et al., 2023a), MMB (Liu et al., 2024c), GQA (Ainslie et al., 2023), MMMU (Yue et al., 2024), and MM- Vet (Yu et al., 2024). As shown in Table 3, NEXT-OMNI not only outperforms models MMaDA and FUDOKI with similar architectures, but also achieves highly competitive results compared to autoregressive (AR)-based multimodal large language models. Notably, NEXT-OMNI unleashes the potential of discrete flow matching (DFM) in understanding tasks through dynamic length gener- ation and caching methods, while achieving faster generation speeds compared to AR models as shown in Figure 12. These results demonstrate that DFM represents a highly promising alternative to AR architectures and merits significant attention.
Image Generation. We evaluate the image generation capabilities of NEXT-OMNI on the widely used GenEval (Ghosh et al., 2023) and DPG-Bench (Hu al., 2024) benchmarks. NExT- OMNI achieves competitive overall performance in both Table 11 and Table 12, scoring 0.85 on GenEval and 84.46 on DPG-Bench, demonstrating strong performance in both generation-only and categories with smaller model parameters and faster response speeds. These results highlight the advantages of the discrete flow matching framework modeling, which allows bidirectional integration of visual information, first generating image layouts and then pro- gressively filling in details, better aligning with the natural characteristies of image generation.
cerrsstandiingggeerttii
Audio Generation and Understanding. We validate our audio understanding and generation capa- bilities on benchmarks LibirSpeech (Panayotov et al., 2015) and AudioCaps (Kim et al., 2019). As shown in Table 13, compared to other omnimodal models, NEXT-OMNI, as the first model to em- ploy discrete flow matching modeling, achieves significantly superior performance in natural audio generation, understanding, speech translation, and speech synthesis. These results further validate the generalizability and scalability of discrete flow matching modeling, providing potential for future applications in protein molecular structure prediction, 3D model generation, and other domains.
28

---

Published as a conference paper at ICLR 2026
Table 10: Multimodal understanding performance on various benchmarks. "Und." and "Gen." denote the abbreviations of "Unnerstanding" and "Generation". denotes models that integrate an external pre-trained diffusion model.
Paradigm
MME-P
GQA
MM-
Und. Only
LLaVA (Lau et al., 2024b)
AR
7B
76.3
809.6
38.7
33.5
25.5
L1aVA.v155
2023)
1510.7
InstractBLIP (Dai al.,
AR
7B
53.4
49.2
26.2
Owen-VL-Chat (Bai 2023)
AR
7B
1487.5
60.6
58.2
57.5
IDEFICS-9B (IDEFICS, 2023)
AR
8B
48.2
38.4
Emu3-Chat (Wang ot al,, 2024b)
AR
8B
85.2
1244.0
58.5
68.2
60.3
31.6
37.2
InstracBLLPP (Dai et al., 2023)
AR
13B
78.9
1212.8
49.5
25.6
Und. and Gen.
LavIT' (Jin et 2024)
AR
7B
46.8
MetaMorph! (Tong et ol., 2024)
AR
8B
75.2
71.8
Gemini-Nano- (Team al., 2023)
1.8
26.3
ILLUME (Wang el 2024a)
AR
7B
88.5
1445.3
65.1
72,9
38.2
37.0
TokenFlow (Qu 2024)
LWM (liu 2024n)
AR
7B
75.2
44.8
9.6
VILA-U (Wu 2024c)
AR
7B
85.8
14018
59.0
60.8
33.5
Chameleon (Team, 2024)
AR
7B
22.4
8.3
Janus (Wu et al. 2024a)
AR
1,58
87.0
1338.0
69.4
63.7
59.1
30.5
34.3
Janus-Pro (Chen et 2025c)
AR
1.5B
16.2
1444.0
75.5
68.3
59.3
36.3
39.8
Show
2024)
948.4
D-Dit 2024f)
Discrete Dift.+Diff.
2,08
1124.7
FUDOKI (Wang et 2025)
DFM
1.5B
86.1
1485,4
73.9
68.2
57.6
34.3
38,0
MMaDA (Yang el all, 2025)
Discrete Dift.
8B
86.
1410.7
68.5
64.2
61.3
30.2
NEXT-OMNI
DFM
7B
87.4
1337.8
78.9
76.3
62.7
43.7
40.1
Table 11: Visual Generation Results GenEval (Ghosh et al., 2023). "Und." and "Gen." denote the abbreviations of "Understanding" and "Generation".
Paradigu
Single Ohj
Obj
Counting
Overall
Gen. Only
SDv1.5 (Rombach al., 2022a)
Diff
0.97
0.38
0.35
0.76
0.04
0.06
0.43
PaAAn- (Chen al., 20248)
Ddtt.
0.98
11.00
01.44
0.80
0.08
12077
0.48
SDv2.1 (Rombach 2022a)
Diff.
0.98
0.51
0.44
0.85
0.07
0.17
0.50
Emm3-
2024b)
(Pudell 2023)
al.. 2023)
SD3 M0d-Medium (Rombach et al., 20226)
Diff
0.99
0.94
0.72
0.89
0.33
0.60
0.74
SANA-15 (Xie et al. 2025)
Ddft.
0.99
0.93
0.86
0.84
0.59
0.65
0.81
Und, and Gen
Chameloon (Tem, 2024)
0.30
LWM
SEED-X 2024a)
0.09
0.67
2024)
0.14
Show-o
2024)
AR-Disccctt
Tranffssion (Zhou
2024)
AR+DifT
0.63
D.D (Li at. 20240)
Discrete Diff.+
0.97
0,80
0.54
0.76
0.32
0.50
0.65
ILLUME (Wang et al., 2024a)
AR
0.99
0.85
0.45
0.71
0.39
0.28
0.61
Janus (Wa et 20240)
20244)
AR
0.97
1368
0130
0.84
0.46
1042
0.61
Harmon (Wa al, 2025)
0.99
0.85
0.66
0.85
0.74
0-48
0.76
Janus Pro (Chen
2025c)
0,80
2025)
(Yang
FUDOKT (Wing al,, 2025)
0.68
0.67
NEST-OMNT
DFM
0.99
0.92
0.79
0.85
0.78
0.74
0.85
Video Understanding. We evaluate the video understanding capabilities of NEXT-OMNI on several benchmarks, including MSVD-QA (Chen & Dolan, 2011), MSRVTT-QA (Xu et al., 2017),TGIF- QA (Li et al., 2016), and ActivityNet-QA (Caba Heilbron et al., 2015). As shown in Table 14, com- pared to models in understanding only or unified understanding gennnaaiinnnnnnnn NEXT-OMNI achieves superior performance across all metrics, demonstrating that the DFM-based framework possesses considerable capability in understanding spatiotemporal relationships.
Video Generation. For video generation, we evaluate NEXT-OMNI VBench (Huang al., 2024) and compare it against classical approaches, including Open-Son OpennSoor (OpenAI, 2025), VILA-U (Wu et al., 2024c), and Video (Hong et al., 2022). The results presented in Table 15 demonstrate that our method achieves superior performance compared to these autoregressive (AR)-based classical methods, highlighting the potential of discrete flow matching (DFM) in short video generation.
29

---

Published as a conference paper at ICLR 2026
Table 12: Visual Generation Results DPG Bench (Hu al., 2024). "Und." and "Gen." denote the abbreviations of "Understanding" and "Generation".
Model
Paradigm
Global
Entity
Attribute
Relation
Other
Overall
Gen. Only
(Rombach 2022a)
PixArt-o (Chen et al,, 2024n)
Diff.
74.97
79.32
78.60
82.57
76.96
71.11
Emu3-Gen (Wang ct al., 2024b)
AR
85.21
86.68
83.15
SDXL (Podell
2023)
Dil..
83.27
82.43
80.91
86.76
80.41
74.65
Playground v2.5 ct al., 2024b)
83.06
82.59
81.20
83,50
75.47
PixArt-E (Chen al., 2024a)
AR
86.89
82.89
88.94
86.59
87.68
80.54
DALLE3 (Belker al., 2023)
DiTT.
90.97
89.61
88,
90.58
89.83
83,50
SD3-Medium (Romhach al., 20226)
Diff.
87.90
91.01
88.83
80.70
88.68
84.08
Und. and Gen.
Show-o (Xie et al., 2024)
AR+Discrete Diff.
67.48
TokenFlow-XL (Qu 2024)
AR
78.72
79.22
81.29
85.22
71.20
73.38
Janus (Wu al., 2024a)
AR
82.33
87.38
87.70
85.46
86.41
79.68
Janus Pro (Chen
2025c)
AR
87.58
88.63
88.17
88.98
88.30
82.63
BLIP-30 (Chen et al., 2025a)
AR+Diff.
81 .60
Tar (Han 2025)
AR
83.59
89.35
82.96
MMaDA (Han et al., 2025)
Discrete Diff
77.81
78.48
81.74
84.79
63.20
69 .7
FUDOKI (Wang 2025)
DFM
80.55
89.73
88.05
83.63
NEXT-OMNI
DFM
81.09
89.76
88.36
94.37
81.63
84.46
Table 13: Comparison with state-of-the-art methods on speech-language and audio-language benchmarks. Here, "T", "S", and "A" represent text, speech (belonging to the audio modality) audio inputs, respectively.
Librispeech (EN-WER)
AudioCaps
Model
Test cletn
oest.
Test
S-+T
T-S
S-T
T-5
A
T-
Speech Only
SpeechTS (Ao et al., 2021)
2.4
5,8
SALMONN
Mini Omni (Xie & Wu, 2024)
4.7
9.4
Freeze-Omii (Wang
2024c)
Qwen2 Audio (Chu et al., 2023)
2.0
4.5
Omnimodal Und.
VITA (Fu al., 2024)
8.1
18.4
EMOVA (Chen al., 2024b)
3,4
VITA 1.5 (Fu al., 2024)
3,4
7.5
OpenOmni (Luo et 2025a)
3,4
7.0
Stream-Ommi (Zhang et al., 2025b)
3.0
7.2
Omnimodal Und. and Gen.
AnyGPT (Zhan
2024)
Unified022 xxllrge (I.a al., 2024)
48.9
2.64
OmniFlow (Li et 2025c)
78.4
NEXT GPT (Wu al., 2023)
81.3
1.74
NEXT-OMNI
3.1
7.0
84.8
1.65
Table 14: Comparison with state-of-the-art methods on the video understanding benchmarks.
Model
MSVD-QA MSRVTT-QA
TGIF-QA
ActiviteNetOAA
Und.
Video-Chit 20236)
LaMo (Zhong 20236)
31.6
29.6
Vilo-ChaGGP (Miiz al, 2023)
64.9
49.3
51.4
35.2
Vidoo LLava (Lin et 2023)
70.7
59.2
70.0
45.3
Und. und Gen.
Unifedl022 (1.n
2024)
52.1
42.5
2025)
(Sin
18.8
Fmu22 (Sun et 2024b)
31.4
28.7
SEED-LLaMA (Ge 2023b)
40.9
30.8
VILA-U (Wo al, 3024g)
73.4
58.9
$1.3
51.6
NEXT-OMNI
76.2
56.4
30

---

Published as a conference paper at ICLR 2026
Table 15:
Comparison with state-of-the-art methods on video generation benchmarks
VBench (Huang et al., 2024).
Model
Total Score
Quality Score
Semantic Score
Open-Sora (OpenAI, 2025)
75,9
78.8
62.3
CogVideo (Hong et al., 2022)
67.0
72.1
46.8
VIL.A-U (Wu et al., 2024c)
74.0
76.3
65.0
NEXT-OMNI
80.1
80.8
77.5
Coul you dhee ow a si a byok inrword gooth not orea and pobt sokos qeocl??
Change glowing leat depeet dhe plare TMETTOWMM
barf, is mutalii engroved 159s.
NEXT-OMNI IS best
Tomord gesterees ihe Schoolooe fosalives Telay Choss the Sorkge Alalla, colord
for fui
thew abrest boo from excrobe! careusti aduuted ts say lequwokrre aille teroses fure dreagh reserd dipastoon pricenyy of cerberers
Piee rélenaa offect color ferros geotborsal erez?
Placre help zw odd the logo
NENT-OMII TH BEST
vibust cales ailer
saike
(worge cell and illo (eild. Offieee valeook oiflel
speefic vevelngtti of light when why the donvel
cober. for batsers cata riflec yelloe 1i4t wish bas aaide refeel longer vovelegth, appearep crarge re!.
Figure 8: Visualization case of multi-turn vision- vision-language interaction. NEXT-OMNI is capable of performing multi-turn image generation based on predefined image generation locations (left) as well as spontaneously determining image generation locations to achieve natural multimodal interactions multi-tura dialogue setting (right).
I ADDITIONAL VISUALIZATION
In addition to the above experimental results, provide additional visualization cases supple- ment the demonstration of NEXT-OMNI's extensive application scenarios and interesting properties.
Interesting Properties. We demonstrate NEXT-OMNT's iterative refinement inference process across different modal data in Figure 11. Figure 16 presents image generation quality comparisons with other similar models. Figure 21 showcases zero-shot cross-modal generalizability, capable of accepting arbitrary data inputs and generating relevant outputs in other modalities. Figure 13 illus- trates discrete flow matching's single forward pass extraction of unified representations for cross- modal retrieval. Moreover, as shown in Figure 22, NEXT-OMNT can spontancously perform "think with images" without relying on external tools. By unlocking image generation during the reasoning process, it can better solve complex tasks and improve performance, demonstrating the considerable potential of NEXT-OMNI.
High-Levvel Multi- Turn Interaction. We demonstrate multi-turn visual interaction capabilities in Figure 8, where the model can autonomously perceive and select appropriate positions for image generation, manually control image generation positions. Figure showcases multi-turn speech interaction capabilities, where the model can accept speech inputs and produce speech outputs.
Basic Single-Turn Interaction. We also supplement visualization cases of NEXT-OMNI's ss single- turn interactions, including image generation in Figure 14, audio generation in Figure 15, video gen- eration in Figure 17, and omnimodal understanding in Figure 10, image understanding in Figure 18, audio understanding in Figure 19, and video understanding in Figure 20. Overall, NEXT-OMNI can accomplish fundamental single-turn multimodal interactions with strong capabilities.
31

---

Published as a conference paper at ICLR 2026
Text Transeription:
[So, what's your take on, you know, the ethics of those Reality TV shows??
Text Transcription:
|Reality TV shows often blur the line between entertainment and exploitation, raising concerns about the well-being of participants and the patential consequences of their actions.]
Figure 9: Visualization case of multi-turn speech-languee interaction. NEXT-OMNI is capable of flexibly using text or speech as input and output facilitate multi-turn interactions.
Tea! Tranuaniptian
people talking
A: Welcome to Twneva Oaks, Mr. O Oarra.
you kindly, India. doughter's getting every day, John.
girls. greef
<image>>/ in Kwave> (n What is hoppaning to Mr. o'Hara?
He to Westem wedding.
Figure 10: Visualization case omnimodal understanding. NEXT-OMNI demonstrates excep- tional general capabilities in omnimodal understanding by effectively comprehending and respond- ing to complex questions that encompass visual, textual, and audio information.
J THE USE OF LARGE LANGUAGE MODELS
We declare that large language models (LLMs) employed to assist with the refinement of this manuscript, specifically, for grammar checking, language polishing, and improving the clarity and fluency of the text. Additionally, LLMs were used in limited capacity for minor debugging and syntactic correction of code snippets included the work.
32

---

Published as a conference paper at ICLR 2026
ANcoMe Tornh Dew Mode
Davee New Morinn
owerng
Mocordale Albegoo doa ty Neocein
Figure 11: Visualization case of iterative refinement process. NEXT-OMNI enables the genera- tion of data in any modality by iteratively performing denoising through bidirectional encoding and refinement random inputs.
514
Figure 12: Visualization case of speed comparison on VBench (Huang et al., 2024). We select several classic methods and tested the generation speeds of these models under cached settings on VBench for 8-frame video generation. It can be observed that NExT-OMNI achieves significantly better acceleration compared to the AR-based Janus, verifying the advantage of DFM's parallel decoding mechanism in inference speed.
Instuctiore Retrieve image
Intruction: Retrivee Wiki
Instructio:: Wili anticle my question
Qvery Text Image
Query Text Image
Query Text + Image
Emercid fumminghird
Which prodiver?
What was fià pointing created?
Unified Rep.
Image Text+inage
Text Condidate
The Coto-Colo Compony. cslabisted 1896, Georgic, USA, and is She world's largez beverage
The 'Mora biso was created batween 1500 and 1517, dipicting elegant and scrone femole figure, conudered typicol representitive
Aietll foruay
beurgeon women during the rise of copitalion.
Figure 13: Visualization case of multimodal retrieval based on unified representation. NExT- OMNI extracts deeply fused unified representations of multimodal data through single forward, enabling cross-modal retrieval. This more generalized capability highlights the superiority of the DFM structure compared to AR in handling multimodal tasks.
33

---

Published as a conference paper at ICLR 2026
Figure 14: Visualization case text to image generation sampled from GenEval (Ghosh et al., 2023). NEXT-OMNI leverages a discrete flow matching mechanism to accomplish text-to- eot-to--mage generation tasks that adhere to both aesthetic and semantic alignment.
Bafe Caplen Drsenial woom fern, and light Motten
Original
Ours
WovTok
Figure 15: Visualization case of audio generation. NEXT-OMNI utilizes discrete flow matching mechanism to accomplish various types of audio generation tasks, including speech synthesis and music generation.
34

---

Published as a conference paper at ICLR 2026
MMaDA
FUDOKI
Ours
MMaDA
FUDOKI
Ours
flamingo stanaling moon, panda holding balloons, surrounded by colorful in the background
lian with butterfly wings, flynn above meadow
A polor bear wearing red seart, anjoying ice cream
Figure 16: Visualization case of image quality comparison. Compared with FUDOKI (Wang et al., 2025) and MMaDA Yang et al. (2025) on various text prompts, NEXT-OMNI achieves superior text-image alignment and aesthetics.
<image>>y ly Generate a video of bird walking towards flower bosed the given image.
Figure 17: Visualization case of video generation. NEXT-OMNI employs discrete flow matching mechanism to generate short videos, demonstrating outstanding scalability.
Image li <image> 'nn Image 2: <imoge>> Y/n What is the difference botween Image 2?
images characier hupging Daraemon holding hearlshopped object, againal bookdrop of pinl-orange suesef and greanery. However fhere noticeable outword,
image, and green square-shaped godgel is altuched to thair bwl, whis Dorcmmon's round bell remains visible his chest. the second replaced with a pink cylindrical device on the belt, and Dorcemon's bell hos transformed into hearl shapl-s pink abject.
scones:
log
the character's leg tucked inward,
gedget
Visualization case image understanding. As we can see from the additional qual- itative examples, NEXT-OMNI consistently delivers more accurate and well-aligned reasoning sponse on visual question answering tasks.
Figure 18:
35

---

Published as a conference paper at ICLR 2026
<wave> \n Tell me something about the wave
Distorted guitars, steady bass, sharp drum beats, and fight rock
<wave> \n Turn the speech input into text.
#2 34 345%. #
Figure 19: Visualization case of audio understanding. NEXT-OMNI is capable of performing au- dio text tasks, including speech transcription and audio to caption generation, showcasing strong generalization abilities.
<ingge>>>>>>>>>>>-> <imogge<<mmgg->>> <imoge <image- Sinnoge>>immme>>>>>> <imagee><imoges <image>+ n Eleborate on the visuo and norrative alaments of the video in detail.
The video shows close-up of large pile of meat, likely chicken, being cooked grill. The sauce, possibly marinade inolicated the blurred greenery in the background. The souce. The grill outdoors. tray, which plocad on the grili. The cooking proceas is shown in thren stages: the meat first ploced on the grill, then covered with the red savee, and finally, is cooked until reaches golden-brown color. The style the video is realistic captures the detaill cooking process.
being cooked
Figure 20: Visualization case of video understanding. NEXT-OMNI is capable of processing video inputs, demonstrating spatiotemporal perception abilities.
36

---

Published as a conference paper at ICLR 2026
SEATAT
Audio Caption: [ seeme like engire rerving
.....
Audio Caption: Disforted guiters, shorp drom beals, and a light rack atrucfure.
Figure 21: Visualization case of cross-modal generation. NEXT-OMNI exhibits strong zero-shot cross-modal generation capabilities, enabling the generation of outputs in various modalities based on imputs from different modalities. This any-to-any generation ability highlights its generalized advantage achieving deep alignment across all modalities.
(Think Ensapd,
The fruit plate lo 2=2igtd, with each call containing trait. The intial state is followse Apple flarona Orange
pustiions rightt conner lisil tl.
keel truit bettuca
Reeraage accurdingg fallowing ralee
truits plawel the apper part the
grid.
Step 3: Rotate the fruit prsitions "edcchniu by prolioos.
Outpet plate
operation
First present the initial state sinagedd>. Nost we avia the top- lelt and bettoes right fraits disgge>.
werearaage according
bdaw))
Walt appearcthas that the roand fintts anwupple and seangs, while mon-mased fraits banano pear. made an ever. New rearrangiag acoarding the ruke sinaged
3t somns deess sotsty the minierue operatine cost roqultement and needs correctine
dadoo
Orange
Apple
Figure 22: Visualization case of thinking with images. NExT-OMNI can enhance its reasoning ability through the "thinking with images' mode, demonstrating strong potential in integrating visual cues into intermediate reasoning steps, which improves interpretability and enables more accurate problem-solving across complex multimodal tasks.
37


Reviews:
[{"review_id": "2TdNplI5PJ", "reviewer": "Reviewer_nB7e", "summary": "This paper introduces NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling of understanding, generation, and retrieval across text, images, video, and audio using discrete flow matching (DFM) techniques. Unlike existing autoregressive (AR) models that struggle with conflicts between understanding and generation tasks, or hybrid architectures that rely on task-specific decoupling, NExT-OMNI employs a streamlined unified architecture. The model leverages metric-induced probability paths and kinetic optimal velocities to enable bidirectional information integration, achieving faster inference through parallel decoding. Useful strategies include reconstruction-enhanced unified representations, dynamic length generation strategies, and vanilla adaptive caching. Experimental results demonstrate competitive performance on standard benchmarks while excelling at multi-turn multimodal interaction and cross-modal retrieval tasks.", "questions": "- It would be better if the authors could provide detailed training costs (e.g., GPU hours) for the proposed model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Unified Architecture: Successfully demonstrates that a single DFM-based architecture can handle understanding, generation, and retrieval, challenging the dominance of the AR-based paradigm.\n\n- Comprehensive Evaluation: Extensive experiments across 7+ benchmarks covering all major modalities and tasks, with careful ablation studies validating design choices.\n\n- Useful strategies: Provides reconstruction-enhanced unified representations training, dynamic length generation strategies, and vanilla adaptive caching, which are quite helpful to the development of the community.\n\n- Inspiring results: Effectively demonstrates how unified representations enable superior retrieval performance compared to decoupled architectures.\n\n- Open Source Commitment: Authors promise to release code, models, and training protocols, which would significantly benefit the community.", "weaknesses": "- I see no other major weakness. Thanks for the authors' hard work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AXOcyOlRNH", "reviewer": "Reviewer_T8p4", "summary": "The paper presents NExT-OMNI, which is an omnimodal foundation model built on discrete flow matching (DFM), targeting unified multimodal understanding and generation. It addresses the limitations of autoregressive (AR) approaches, which suffer from inherent conflicts between understanding and generation tasks as well as slowed inference due to decoupled designs. Unlike these, DFM introduces parallel information processing with bidirectional integration, enabling both efficient any-to-any cross-modal generation and enhanced multimodal understanding. Besides, NExT-OMNI achieves precise cross-modal retrieval and robust multi-turn multimodal interactions, surpassing prior AR-based and hybrid approaches. The model consistently delivers competitive or superior performance with reduced latency across standard multimodal benchmarks.", "questions": "1. Lumina-Dimoo [2] also adopts a similar discrete diffusion modeling approach. Why was there no comparison with it? The authors should further clarify this issue.\n\n2. While unified modeling for both generation and understanding has become a trend, it seems that separating these tasks and modeling them independently yields better performance. Why is it still necessary to persist with a unified model design?\n\n3. The appendix demonstrates that increasing the number of sub-token tables in the warmup encoder improves reconstruction performance. Why is there still a need to aggregate these into high-dimensional features for the backbone, and is there experimental evidence to support this design choice?\n\n   \n\n[2] An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. NExT-OMNI introduces a novel unified modeling approach using DFM to integrate multimodal understanding and generation tasks. Unlike previous work, which requires additional diffusion decoders and increases parameter size, NExT-OMNI achieves compactness by reducing the need for extra modules. Its bidirectional feature fusion design effectively enhances cross-modal interactions, enabling better feature integration across modalities. \n\n2. NExT-OMNI introduces dynamic length generation optimization. By using the EOS token's confidence scores, the model dynamically adjusts text generation lengths. This can effectively improve multimodal understanding and generate more natural outputs in text-based tasks.\n\n3. NExT-OMNI integrates an adaptive caching mechanism to leverage the parallel decoding strengths of DFM, leading to a 1.2× increase in inference speed compared to AR architectures. \n\n4. The model is tested not only on single-turn tasks such as text-to-image and text-to-audio generation but also on real-world scenarios requiring multi-turn interactions. It demonstrates clear improvements in unified understanding and generation capabilities, particularly in dynamic multi-turn exchanges.", "weaknesses": "1. While the discrete DFM effectively unifies understanding and generation tasks, its reliance on discrete representations, as opposed to continuous flow-based approaches, may lead to some performance degradation in generation tasks due to information loss during the discretization process.\n2. After completing the encoder warmup phase, the model requires joint optimization with reconstruction losses during subsequent multi-stage training. This additional reconstruction loss, compared to exclusively optimizing with cross-entropy loss, can inevitably lower training efficiency and lengthen the overall training process.\n3. Recent studies [1] show that discrete DFM and diffusion-based models demand more computational resources during training compared to autoregressive architectures. This is attributed to the complexity of learning difficult tasks within the discrete modeling framework.\n\n[1] Training Optimal Large Diffusion Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PibigRFkzL", "reviewer": "Reviewer_DQzr", "summary": "This paper proposes a DLLM-like model that aims to build a unified, end-to-end architecture for multimodal understanding and generation. The authors conduct detailed experiments and evaluate the model across three modalities—text, audio, and image—demonstrating its overall effectiveness and strong performance.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed architecture is highly innovative. To the best of my knowledge, this is the first unified model designed for discrete-unit generation and understanding across all modalities.\n\n- In terms of performance, the model outperforms several existing approaches (although many of the baselines are not state-of-the-art).", "weaknesses": "- The comparison with baselines is not sufficiently comprehensive. Important recent models such as Qwen2.5-Omni and Qwen3-Omni, which outperform Next-Omni on OmniBench, should be included for a fair evaluation. Similarly, XCodec2 represents the most recent state-of-the-art in audio tokenization and should be considered as a baseline.\n\n- For audio-related tasks, perceptual audio quality is critical. It would be very helpful if the authors could provide qualitative examples or audio case studies to allow readers to better assess the perceptual quality of the generated audio. I look forward to seeing such demonstrations in future revisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5iSgduPam8", "reviewer": "Reviewer_MbBb", "summary": "In all, NExT-OMNI is a well-executed application model that demonstrates strong engineering by effectively integrating Discrete Flow Matching into a 7B-scale, any-to-any multimodal system., But its innovation is combinatorial rather than fundamental and lacks analysis of its slight underperformance in speech tasks and doesn't prove DFM's efficacy on strictly sequential tasks. Moreover, the application of NExT-OMNI on large scale model remains to verify.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.NExT-OMNI first trains a model with \"any to any\" on the 7B scale and achieves SOTA performance on multiple tasks, expanding the application of Discrete Flow Matching(DFM).\n\n2.It does realizes the full integration of DFM, unified discrete representation, lightweight multi-head, parallel decoding, and multi-turn multimodal instruction tuning methods, and provides design details and a series of data synthesis. Relevant ablation experiments prove the effectiveness of each part of the model.\n\n3.For the first time, the paper combines block size padding with dynamically adjusting the preset generation length in steps of block size based on <EOS> confidence, which not only saves computing power but also prevents truncation and loss of generated content.", "weaknesses": "1.The paper is a combinatorial innovation at the methodological level, its contributions lie in engineering implementation and scaling up to a 7B full-modal model, without proposing new ideas for addressing the problem of balancing model understanding and generation tasks.\n\n2.In Table 2, NExT-OMNI seems to be slightly inferior in speech-to-speech tasks comparing with auto-regressive models, but no further explanation or analysis of it.\n\n3.The paper does not experimentally demonstrate how DFM, which inherently lacks such an inductive bias, can match or surpass auto regressive models on tasks that strictly require sequential adherence such as code generation and step-by-step reasoning.\n\n4.In Figure 22, the paper only qualitatively illustrates the model’s ability to “think with images”, but it does not systematically evaluate how effectively the model leverages its generation capability for complex reasoning tasks. For example, there is no assessment of whether the model can assist mathematical or logical reasoning by generating images, nor are quantitative metrics such as MMU[1], MMBench[2], or ScienceQA[3].\n\n**References:**\n\n[1] Yue X., Ni Y., Zhang K. et al. “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI.” arXiv:2311.16502, 2024.\n\n[2] Liu Y., Zhang H., Chen J. et al. “MMBench: Is Your Multi-modal Model an All-around Player?” arXiv:2307.06281, 2023.\n\n[3] Lu P., Mishra S., Xia T. et al. “Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.” NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)