XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models
Authors: Xingrui Wang
Jiang Liu
Chao Huang
Xiaodong Yu
Ze Wang
Ximeng Sun
Jialian Wu
Alan Yuille
Emad Barsoum
Zicheng Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Omni-modal Benchmark
Cross-modal consistency
Primary area: datasets and benchmarks

Abstract: Omni-modal large language models (OLLMs) aim to unify audio, vision, and text understanding within a single framework. While existing benchmarks have advanced multimodal evaluation, it remains unclear whether OLLMs achieve modality-invariant reasoning or inherit modality-specific biases. We introduce \textbf{XModBench}, a large-scale tri-modal benchmark explicitly designed to measure cross-modal consistency. XModBench contains 60K multiple-choice questions across five task families and systematically covers all six cross-modality directions, enabling diagnosis of task competence, modality disparity, and directional imbalance. Experiments show that even the strongest model, Gemini 2.5 Pro, (i) struggles with spatial and temporal reasoning, achieving less than 60% accuracy, (ii) suffers from modality disparities, with performance dropping by over {20 points} on average when audio inputs replace text, and (iii) exhibits directional imbalance, with a {9-point gap} when using vision as context versus using text as context.
The findings suggest that OLLMs fall short of modality-invariant reasoning, and XModBench provides a fundamental diagnostic tool for evaluating and improving their overall cross-modal competence.

Published as a conference paper at ICLR 2026
XMODBENCH: BENCHMARKING CROSS-MODAL CA- PABILITIES AND CONSISTENCY IN OMNI-LANGUAGE MODELS
Xingrui W Liu Chao Huang! Xiaodong Yu', Ze Wang!
Ximeng Sun', Jialian Wu', Alan Yuille", Emad Barsoum' Zicheng Liu' Advanced Micro Devices
JJohns Hopkins University University of Rochester
Project Page
Dataset Card
Code
Cross-Modal Triplet
Modality Configuration
Task Families
Posceptiin
JoH Sponial
Tempomil
The
cltp hep
onder
Balanced
A Linguistic
Q
Exterral
Oee ta righ
Design
iauge milu
Witah audia
lmnowdedge
josjec ppott
Tayler Sunkh
cus aeto loal
5 task fomilies, 17 subtasks 61,320 question-answerkkn pairs
(b)
(6)
Audioe Taes
Whids
planse deseibe
Liuzs zedia city Waich
what ose the stage??
decorpticn hest maicher bour!
A Wticweee (alanl)-aaaaaal Cli
berdin
B heopinaccis
Dog hhppkraaciig Crow
Taie lcatig
D Nu pice
Diee placo
Audi
Or Which image mos Early prodaco wherr Via hear atyy
For tis pleve May boking wlick image twd mushecake
the harking which sadio soend
whar is druribed
Figure 1: Overview of XModBench. (a) Instances are built from aligned lext-image-audio triplets; (b) instantiated into six modality configurations by permuting context and candidate modalities; (c) spanning five domains with 17 subtasks and 61,320 question-answe answer pairs; and (d) illustrated with example multiple-choice questions under balanced modality settings.
ABSTRACT
Omni-modal large language models (OLLMs) aim to unify audio, vision, and text understanding within single framework. While existing benchmarks pri- marily evaluate general mooss- question-answent clear whether OLLMs achieve modality- invariint reasoning exhibit modality- specific biases. We introduce XModBench, large-scale tri-modal benchmark explicitly designed to measure cross-modal consistency. XModBench comprises 61,320 multiple-choice questions spanning five task families and systematically covers all six modality compositions in question-answer pairs, enabling fine- grained diagnosis of an OLLM's modality-invariant reasoning, modality dispar- ity, and directional imbalance. Experiments show that even the strongest model, Gemini 2.5 Pro, (i) struggles with spatial and temporal reasoning, achieving less than 60% accuracy, (ii) reveals persistent modality disparities, with performance dropping substantially when the same semantic content is conveyed through au- dio rather than text, and (iii) shows systematic directional imbalance, exhibiting lower consistency when vision serves context compared to text. These findings indicate that current OLLMs remain far from truly modality-invarian reasoning, and position XModBench as fundamental diagnostic tool for evaluating and im- proving cross-modal competence. All data and evaluation tools will be available https: /gi hub. com/ Xingrut Wang/ XMocBench.
ability, it remains un-
Corresponding author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Omni-modal large language models (OLLMs) integrate text, vision, and audio into a unified rea- soning framework (Comanici et al., 2025; Xu et al., 2025; Xing al., 2025; Su et al., 2023; Fu et al., 2025b; Cheng et al., 2024; Zhong et al., 2025). However, despite impressive advance- ments and expanded modality coverage, a key question remains: do these models reason in truly modality-invariant manner, or do they still exhibit systematic biases tied to specific input modal- ities? For humans, cross-modal integration is typically seamless, yet it remains unclear whether OLLMs demonstrate comparable consistency. When the same semantic content is presented in dif- ferent soons-spoen audio, written text, or visual images- do models still converge on the same correct answer? We refer to this property cross-modal consistency: the ability to maintain sta- ble predictions regardless of input modality, thereby demonstrating reasoning over shared semantic representations rather than relying modality-specific cues. Although directly diagnosing whether current OLLMs achieve this goal is non-trivial, we can evaluate them through carefully designed benchmarks that expose inconsistencies. For instance, by posing semantically identical questions under different modality settings, we can test whether predictions diverge across modalities - an indicator of reliance on surface-level patterns rather than genuine modality-invarian reasoning.
Recent benchmarks have taken promising steps toward evaluating OLLMs, particularly through audio-visual tasks that reveal baseline cross-modality performance. Datasets such as Music AVQA (Li et al., 2022), AV-Reasoner (Lu et al., 2025), and Pano-AVQA (Yun et al., 2021) pri- marily probe fine-grained audio-visuse reasoning, while broader efforts like AVQA (Yang et al., 2022), WorldSense (Hong et al., 2025), AV-Odyssey Bench (Gong et al., 2024), and OmniBench (Li et al., 2024b) expand to general multimodal understanding across diverse contexts. However, these benchmarks largely overlook whether models remain consistent across modalities. While other works (Park et al., 2025; Zhang et al., 2024) attempt to assess modality consistency, they are re- stricted to the vision-text setting within vision-language models.
To address this gap, we introduce XModBench, a benchmark specifically designed to evaluate cross- modal consistency in omni-modal large language models. We formulate all questions in a multiple- choice format, where each question naturally contains two components: (i) a context describing an object event, and (ii) set of candidates from which the model must select the correct one. Unlike prior benchmarks that typically fix either the context or the choices to a single modality (Yang et al., 2022; Li et al., 2024b), XModBench systematically covers all six cross-modal directions among dio, vision, and text (see Tab. 1). To ensure broad coverage and rigorous evaluation, XMODBENCH spans five pemnains--rrcepttoon spatial reasoning, temporal reasoning, linguistic understanding, and external knowledge. We curate data across these domains through re-annotation, synthetic con- struction, and targeted web collection, ensuring both diversity and balance across modalities. The resulting benchmark comprises 61,320 multiple-choice question answer pairs (10,220 unique in- stances), each instantiated in six modality configurations that preserve identical semantics across audio, visual, and textual forms. This enables both large-scale evaluation and fine-grained diagnosis of cross-modes consistency. An overview of the benchmark design is illustrated in Fig. 1.
Table I: Comparison of multimodal question-answering (QA) benchmarks by modality coverage, task domains, and modality consistency.
Benchmurk
eQ
Contest Modality
Candidate Modality
Task Demain
Mod. Consist.
Vision
Andio
Vision
Aniio
Parup
Spatial
Tempueal
Ling.
Know
MMR Beach at
MMfRench 20144
20045y
SEED-Bend-2
20144)
AndieBench Wang (2024)
24,371
Audiopalia (1) al, 2022g
45,867
MMAU (Sakala et 20024)
10,000
AVgA 2003
$7,335
Pano AVQA (tt al, 2021)
$1,700
X
X
Misic-AVQA et. 202)) 2022)
45,867
SAVE Bench (Sun et 28144)
4,350
Vide>-MMi (Pout al, 2025a)
7,200
WorkdSrme (Hingettal 2025)
3,172
AV Ressoner (La atal,, 2025)
1,027
AV Odysey Bench (Gong et all, 2004)
1,142
Ommillench (Li (024))))
4.555
XMadBench (Ours)
61,320
2

---

Published as a conference paper at ICLR 2026
We systematically evaluate models on XMODBENCH, going beyond overall accuracy to provide fine-grained diagnosis of cross-modal reasoning. Specifically, we analyze three complementary di- mensions: (1) Task competence by averaging over all six modality directions, assess model performance across perception, spatial, temporal, linguistic, and knowledge tasks, yielding task- centric comparisons of multimodal competence: (2) Modality disparity we measure consistency when the same question is posed in different modalities, where high variability signals reliance on modality-speciiff cues rather than shared semantic representations; and (3) Directional imbal- ance we compare accuracy when context and candidate modalities are swapped, revealing asym- metries in cross-modal grounding.
Our experiments show that current OLLMs fall short along all three axes. They perform strongly on perception and linguistic tasks (hest models reach around 70%), but degrade by 15-25 points on spatial and temporal reasoning. Performance also drops sharply whenever audio is involved, un- derscoring that auditory representations remain the weakest link. Finally, accuracy is consistently higher when text serves the candidate modality, highlighting incomplete bidirectional alignment across modalities. Together, these findings demonstrate that today's s OLLMs remain far from achiev- ing mondality-ivvariany reasoning, underscoring the diagnostic value of XMODBENCH.
In summary, XMODBENCH makes the following key contributions:
1. Cross-modal consistency benchmark. We present X.MODBENCH, the first tri-modal multiple-choice guestion-answering benchmark explicitly designed to evaluate cross-modal consistency, covering all six modality mappings among audio, vision, and text.
2. Comprehensive coverage. The benchmark spans five task families with 17 subtasks and 61,320 muestion-answer pairs, ensuring broad domain coverage and fine-graines diagnos- tics, while its balanced design enables fair assessment of modallty-invariian reasoning.
3. Diagnostic metries. We introduce modality disparity and directional imbalance directly measure robustness and bidirectional alignment across modalities. Our experiments reveal systematic weaknesses in current OLLMs, providing actionable insights for developing more modality-invariint architectures and training strategies.
2 RELATED WORK
Multimodal Question Answering (QA) Benchmarks. number of benchmarks have been devel- oped to evaluate multimodal large language models (MLLMs). Grouped by modality composition, Yin et al. (2024), Liu al. (2024), and Li al. (2024a) focus on the vision-text setting (covering both images and videos). For audio-texx evaluation, representative efforts include Wang et al. (2024) and Sakshi al. (2024). When combining audio and vision with text, variety of benchmarks have emerged, including Yang et al. (2022), Li et al. (2022), Yun et al. (2021), Sun et al. (2024), Hong et al. (2025), Lu et al. (2025), Gong et al. (2024), et al. (2024b), and Zhou et al. (2025). Other re- cent works, such Yang et al. (2025), further extend evaluation to diverse multimodal combinations. Despite their breadth, these benchmarks primarily emphasize coverage across tasks and modalities, while less attention has been paid to evaluating cross-modal consistency-whettee stable answers when the same semantic content is expressed in different modality forms. Our work fills this gap by explicitly designing a benchmark centered on modality-invarian reasoning.
models produce
Cross- Mrdas-Modallty Consistency. Recent work has begun to investigate whether multimodal mod- els behave consistently across modalities. Park et al. (2025) introduced the Modality Importance Score to quantify modality bias, which measures how much each modality contributes to answering questions in VideoQA. Zhang et al. (2024) further proposed the notion of cross modal consistency between text and image, defining consistent model as one that applies the same internal reasoning to semantically identical inputs across modalities, thereby yielding consistent outcomes. In con- trast, other studies, such as Sung-Bin et al. (2024) and Choong et al. (2024), report instances of inconsistent audio-video reasoning, where models hallucinate non-existent sounds or visual signals, thereby exposing modality bias and cross-modal inconsistency. While these efforts provide pioneer- ing insights into gross-modas consistency, they are typically confined to specific modality pairs. Our work not only expands the scope to cover broader range of modality combinations for of the-art OLLMs, but also conducts a deeper analysis of their cross modality reasoning behavior on a comprehensive task suite.
3

---

Published as a conference paper at ICLR 2026
3 XMODBENCH: COMPREHENSIVE CROSS-MODAL BALANCED BENCHMARK
We introduce XModBench, a comprehensive multiple-choice question-answering (QA) benchmark designed to evaluate the coos- capabilitiies and consistency of OLLMs across audio, vision, and text. A key feature of XModBench is its modality- balanced design. which creates six cross-modal variants of semantically identical questions to enable a controlled and fair evaluation of cross-modal capabilities and consistency (Sec. 3.1). The benchmark offers extensive domain coverage through five task families and seventeen subtasks (Sec. 3.2), all built upon meticulously curated, high-quality, and diverse tri-modal data (Sec. 3.3).
3.1 BENCHMARK DESIGN
The central objective of XMODBENCH is to evaluate whether models preserve crosy-modal consis- lency when the same semantic content appears in different modalities. Each item is a four-choice multiple-choice question consisting of a <context> (question stem) and four <cand i dates> (answer options). By systematically permuting text (T), vision (V), and audio (A) across the <context> and icandidatess we generate six modality configurations of the same ques- tion (see Fig. 1 (b) and (d)). This balanced design ensures that no single modality is privileged and enables consistent evaluation across all directions, which supports three diagnostic properties aligned with the goals of our benchmark:
(1) Task competence. Since each task is instantiated uniformly across all modality pairs, we mea- sure competence by averaging accuracy across all context-candidate configurations. This yields fair estimate of model's overall capability for each task, independent of modality-specific biases.
(2) Modality disparity. By presenting semantically identical questions under different modality configurations, we keep the content fixed while varying only the modality. For example, to compare audio and vision, we examine cases where text provides the context with audio candidates (T-+A) versus text with visual candidates (T-V),, and similarly compare A-+T against V-+T settings. Dif- ferences in accuracy under these controlled comparisons reveal modality disparities, indicating the relative competence across different modalities.
(3) Directional imbalance. We examine inverse settings by swapping the modalities of context and candidates. For example, a model may perform well when vision serves the context and text provides the options (V-T), but perform worse when the same semantic content is presented as a text context with visual candidates (T~VV. Such differences indicate asymmetric grounding between the two modalities, and comparable asymmetries are also observed in the audio-text and audio-vision pair.
Singer 900 (1.47%)
3.2 TASK TAXONOMY
Modia 1,200 (2
XModBench covers five task families with seven- teen subtasks, spanning perception, spatial reason- ing, temporal reasoning, linguistic understanding, and external knowledge (sce Fig. 2). Each task is formulated in the multiple-choice format and fol- lows the modality-balanced configuration described in Section 3.1: <context> is drawn from one modality and four <candidates: from another. In this section, we detail the design of these subtasks and specify how each instance is instantiated across modalities within every task.
Muuicc Genre
General
Activitirs
6:000 1a sne
0, mui
greined (P.%
Rtreegnition
4,002 (8.5A))
Taak 5
Extemal
Knowledae
Ranurai Enwionment 8,000 (4 prg
Translation
Task Lingustic
Task 1 Peroeption
4,212 (8)
Emooion 2,200 sha
Task
Instrumenta
8o09 is thi
Temportl
Taak 2
Task Perception. This task evaluates whether models can recognize the same object, activity, or scene across modalities. For example, barking dog may appear as an image, as its sound, or as the text description "dog barking." Here, visual in- puts are images, audio inputs are recordings of cor- responding sounds, and text inputs are short labels or phrases. The data drawn from diverse domains, including human activities, animal behaviors, musi- cal instruments, and natural environments.
Order mia
reasoning
Reasoning
Spatal
Canrureent
3,010 yeo)
Count aonce a9o
Anangamen
Casculation
We Daee ds poneeeee
Figure 2: Distribution of XModBench's questions across five task families with spe- cific subtasks.
4

---

Published as a conference paper at ICLR 2026
Vision costest
Unedlaas Winch mane de indkin
Vision context
Qhackkan: whick lest deceptice (m acounityy you ebanel inlout Clons
Vision context
Quniisa: Whick sere onr eade! invisenci lhis imggit Anwer caustin with A, B, orD
tis image?
Plaae prrvid A D2
C, crD
Hee V-ritail witurge
(fe toik)
Oie Ve tal)
A Suuppan Vala
contect)
from bel
D for rights
(heleccore)) Intooeenennt oottos)
C-. fenaish (0', hedvtpecterr (Se
A- Bos Gow soaaeeeero
UN0%
A Vend
(Taad oxtiod)
aeraenc bues ian
co aave e
Answer:
A
Amnerz
(u) Perception: Instruments
(b) Spatial Reasoning: 3D Movements
(c) Spatial Rensoning: Loculization
Vision context
Vision context Qhandion:
Vision context
Lock, digge is deree is the image? Anever
whic
the sacher th audio afa repored acties the dipeeccentt....
Del same contuer danm la di image? Answer the quoskon
(Be' V--Tree (uld
(Bx V VoeTTio DSC) Ve
A Kiiees
Arian Crank
Hure Stlke
Cuuda peacill t He
2
C Dille Eliis
Laby Gags
Uer V msk)
(ie Voo
040-44
11000
of
Clini boddy
too crnccl) Cags
rpptiiie
(d) Temporal Reasoning: Count
(e) Linguistie: Recognition
(f) External Knowledge: Singer
XModBench task examples. We show sample questions from six subtasks in the bench- mark. Each question includes possible contexts from different modalities, and for the vision-context example, the candidates are given in either text or audio.
Figure 3:
We divide perception into several subtasks. General activity recognition mixes candidates from diverse domains to test broad semantic alignment, while fine-grained activity recognition restricts candidates to single domain (e.g.. animal species or instrument types), thereby increasing diffi- culty and requiring precise discrimination. We further design domain-specite subtasks to capture unique challenges: recognizing natural environments (e.g., rainfall, wind, fire), distinguishing instruments (c.g.. violin, bass, cello). and identifying instrument compositions where multiple instruments played together (e.g., violin and bass, cello and flute). Illustrative examples shown in Fig. (d) and Fig. 3(a).
Task 2. Spatial reasoning. This task evaluates whether models can interpret object positions and motion in 2D and 3D space, which is an important factor in vision-language models (Chen et al., 2024). We extend this ability to the omni-modal setting and design three subtasks. The first is 2D rangement, where the model determines the left- right order of objects such as musical instruments. Visual inputs are images of ordered layouts, audio inputs are stereo recordings with left-right cues, and text inputs describe the relative arrangement; distractors are generated by swapping or permut- ing positions. The second subtask, 3D localization, using panoramic videos from Shimada et al. (2023), requires identifying the orientation of events in video frames, spatialized audio, and short textual descriptions (e.g.. "a man speaking from the front-left"); distractors are produced by shifting the same scene to nearby but incorrect directions through camera audio rotation. The third sub- task, 3D movement understanding. focuses on motion directions such as left-right or front-back, instantiated with street-view or action videos, spatialized audio of approaching receding sounds, and textual trajectory descriptions (Fuentes et al., 2022); distractors are clips with incorrect motion patterns or mismatched vehicle types. Examples for the 3D movement and localization tasks shown in Fig. 3(b) and (c). respectively.
Task 3. Temporal reasoning. This task evaluates whether models can understand event order and frequency across time in video and audio. We design three subtasks. The first is temporal order, where models infer the correct sequence of events from muted video segments, audio clips, or textual descriptions and align them across modalities. The second, temporal counting, requires recognizing the number of repeated actions such as tennis hits, jumps, or drum beats, with distractors differing in count. For example, a video may show tennis player hitting the ball three times, and the model must select the audio clip with exactly three hits or the text "3 times." " The third, temporal calculation, extends counting by applying simple arithmetic to the repetition number. For instance, a video shows a person jumping three times and the query applies 2 x count, the correct answer should correspond six repetitions, given either times." An example of the temporal counting task is in Fig. 3(d).
audio clip with six jumps the text "6
Task 4. Linguistic understanding. This task covers recognition of linguistic content and inter- pretation of affective meaning. While prior work separates OCR for vision and ASR for audio (Fu et al., 2024b; Wang et al., 2024), XModBench unifies them in cross-modal setting. We design
5

---

Published as a conference paper at ICLR 2026
three subtasks. The first, linguistic recognition, focuses on transcribing text from images, audio, or phrases; correct candidates require word-level precision, while distractors differ by only one or two words (see Fig. 3())). The second, translation, evaluates ities, with distractors introducing subtle shifts such as antonyms, degree modifiers (e.g., "very" + "a little"), or small changes in numbers and entities. The third, emotion classification, targets af- fective understanding in dialogue: audio inputs are spoken conversations, visual inputs are muted video clips with transcripts, and candidates represent emotions such as joy, sadness, or anger, with distractors drawn from closely related categories.
translation across modal-
Task 5. External knowledge. Beyond perceptual and reasoning skills, some tasks require linking multimodal content with world knowledge. We design three subtasks. The first, movie recognition, presents audio clips from trailers, visual posters, or short text deseriptions the plot, with candidates drawn from films of similar genres or storylines. The second, music genre classification, uses album covers, short audio clips, or textual genre labels, with distractors from closely related genres (e.g., "jazz" vs. "blues"). The third, singer identification, provides names, portrait images, or audio clips of songs, with distractors sampled from artists of similar musical styles (see Fig. 3(f)..
3.3 DATA CURATION
The construction of XModBench follows a three-stage pipeline. We begin by collecting large-scale sox--vsion-audde triplets across all task domains, then generate task-specific multiple-choice questions, and finally apply both automated filtering and human verification to ensure quality and consistency.
Cross-modal data collection. We curate large corpus aligned across vision, audio, and text by combining three sources: (i) re-annotated and extended data from existing multimodal datasets, such as adapting VGG-Sound for perception tasks or STARSS23 (Lee et al., 2022; Shimada et al., 2023) for spatial reasoning; (ii) synthetic or model-generated content to cover missing modalities, for example generating speech audio with FireRedTTS (Guo et al., 2025) producing rendered text images for translation tasks; and (iii) web-collected samples for domains not well represented in public resources, such singer portraits and songs for the Singer Identification task or trailers and posters for Movie Recognition from public You'Tube videos. This design ensures both coverage and balance across all five task families. Detailed sources and processing procedures are described in Appendix G.
Question candidate generation. To ensure the correctness of both the generated questions and answers, we first construct task-specific multiple-choiee templates using our annotated tri moddllyyy data. The question descriptions are then refined by GPT-5 (OpenAI, 2025) solely improve lan- guage fluency and stylistic diversity. Importantly, this refinement does not introduce any new infor- mation or alter the underlying semantics. Fach question is instantiated with context and four candi- dates under the directions. Distractors are created to be semantically challenging but unambiguous, while templates are diversified with both human-written prompts and LLM-assisted variations.
configuration, ensuring consistent evaluation across all modality
LLM filtering and human-in-the-lone verification. To guarantee data quality at scale, we first adopt foundation models (OpenAI, 2025; Comanici et al., 2025) to filter out low-quality ambigu- ous samples. Human annotators then double-check the filtered results to ensure accuracy. After questions are constructed, internal round of testing is conducted by annotators, who resolve ambiguities and validate correctness. Feedback from this process is used to regenerate and retest questions until high-quality items obtained.
Overall, this pipeline yields a high-quality benchmark with diverse and well-aligned multimodal content. More detailed descriptions of dataset sources, generation strategies, and signal-processing techniques are provided in Appendix G.
4 EXPERIMENTS
4.1 BASELINES
We evaluate XMODBENCH on a diverse set of recent omni-modal large language models. The Gem- ini series (Comanici et al., 2025; Team et al., 2024) represents state-of-ther thr-thh-rrtt closed-source omni- modal models, and we include multiple variants ranging from Gemini 1.5 Pro to Gemini 2.5 Pro. Note that OpenAT APIs do not currently support processing audio and visual modalities jointly

---

Published as a conference paper at ICLR 2026
Table 2: Results on Xenchh We report (a) the performance under different input modalities across the full benchmark, and (b) the summary of average accuracy for each of the task families. The highest scores are bolded, and the second highest are underlined.
Model
Perc.
Accuracy
Spat.
Temp.
Tuus Families Ling.
Knwl.
A A T
A+ V
T
Modulity Configuration
A
T+ V
V-
V-+
Sid.
Avg.
Context
25.3
Qwen2.5.VL
91.3
51.4
40.9
84,1
77.2
60.1
74.7
67.4
75.0
68.7
73.7
PandaGPT
24.6
25.7
24.4
25.5
23.1
24.5
25.0
23.8
25.2
24.5
25.1
0.5
24.7
Unified-1O
36.1
23.6
23.8
30.4
26.8
28.9
24.0
25.4
32.0
25.7
32.7
3.7
28.1
Unified-1O XL
42.2
25.0
26.
30.8
29.5
33.3
27.0
27.1
32.9
26.5
37.4
30.7
Unified-1O) XXL.
43.7
28.3
27.7
31.2
34.0
37.4
25.0
31.2
37.8
26.7
39.9
6.3
33.0
VidcoLLaMA
VITA
34.8
34.0
29.4
46.1
32.6
40.2
26.0
29.5
26.8
29.9
59.3
12.8
35.4
Baichuan Orni 1.5
58.9
34,9
30.0
62.8
56.7
35,8
40.5
56.2
38.6
73.0
14.0
48.7
Echolak-R
56.4
49.9
77.6
11.3
Qwen2.5-Dmol
75.5
38.4
32.3
74.1
72.8
6.2.0
48.0
55.4
59.6
50.5
76.3
10.1
58.6
Pro
Gemini 2.0 Flash
66.2
48.4
44.8
70.2
78.
63.7
49.0
52.2
71.5
47.6
85.2
15.2
61.2
Gemini 2,5 Plash
66.1
48.0
48.6
73.1
82.8
62.6
51.2
58.1
75.7
51.9
86.0
14.2
63.77
Human
91.0
89.7
88.9
93.9
93.9
92.4
91.5
91.1
91.8
86.4
95.6
3.0
91.5
within single query; therefore, omit the GPT series from evaluation. For open-source mod- els, we include the latest Qwen2.5-0mni (Xu et al., 2025), Baichuan Omni 1,5 (Li et al., 2025), and Echolnk-R1 (Xing al., 2025). Additional open-source omni-modal baselines include Vide- oLLaMA 2 (Cheng et al., 2024), VITA (Fu et al., 2025b), the Unified-IO 2 series (Large, XL, and XXL variants) (Lu al., 2024), and PandaGPT (Su al., 2023). Together, these models represent a broad spectrum of both closed- and open-source OLLMs.
4.2 MODEL PERFORMANCES
Table 5 reports results across five task families and six cross moosl directions (Audio + Text, Au- dio + Vision, Text - Audio, Text ++ Vision, Vision Audio, Vision + Text). The first subtable summarizes average accuracy across all tasks for each modality configuration, while the remain- ing subtables present detailed performance within each task family. The highest scores bolded, and the second highest are underlined. For each model, we also report the overall average accuracy (Arg.) and standard deviation (Std.) across the six configurations quantify robustness to modality shifts. Details the human evaluation are provided in Appendix F.
Performance by task families. Overall, the Gemini 2.0 and 2.5 series outperform all open-source models. Among open models, Qwen2.5-Omni and Echolnk-RI R1 are the strongest baselines, surpass- ing Gemini 1.5 Pro by 3.6 and 4.2 points, respectively. Across the five task families, spatial and temporal reasoning remain the most challenging (Gemini 2.5 Pro achieves 50.1 and 60.8), whereas perception and linguistic tasks reach higher accuracy (75.9 and 76.8). The performance gap be- tween open- and closed-source models extends beyond spatial and temporal reasoning external knowledge: while Qwen2.5-Omni and EchoInk-RI perform comparably to Gemini 2.5 Pro on per- ception, the latter attains 89.3 on external knowledge. These results highlight persistent bottlenecks in open-source models, as closed-source models likely benefit from broader web-scale pretraining and stronger
reasoning capabilities.
Performance by modality configurations. We also analyze performance consistency across modal- ity configurations on the same tasks and observe clear divergences. Vision text settings consistently outperform audio text ones, confirming that visual representations are more strongly grounded than audio. In perception tasks, accuracy exceeds 90% with vision-text inputs but drops by over 20 points with audio-text, Audio-vision combinations without textual anchors yield the lowest scores, highlighting the difficulty of aligning heterogencous signals. Among SOTA models, Gemini 2.5 Pro (Avg. 70.6, Std. 11.7) shows the best balance of accuracy and stability, while Qwen2.5- 10.1) and Echolnk-R1 (Std. 11.3) are the most consistent open models. By contrast, Gemini 1.5 Pro and Baichuan Omni 1.5 have standard deviations exceeding 14, reflecting weaker robustness to modality variation.
(Std.
7

---

Published as a conference paper at ICLR 2026
4.3 MODALITY DISPARITY ANALYSIS
A key challenge for OLLMs is whether they handle audio, vision, and text equally rather than favor- ing one modality. XMODBENCH enables this by instantiating identical semantics across modality
settings. The disparity defined as AT vs. V
(ACCA-VV AC + (Acov-A - A).
which quantify the paired subtraction, c.g.. compares configurations that differ only by substituting text with vision, thereby isolating the effect of modality substitu- tion on accuracy. Results in Fig. 4 show that AT exhibits the strongest dispar- ity (-49 for Gemini 2.5 Pro), Av vs.A is moderate (-33), and AT V remains smallest (-15). These findings highlight audio as the most challenging modality, with vision showing moderate gaps and text remaining the most robust.
www. Pondden
10
-20
Pendacory
Adde
Echnke ofl
quena cheed
deniss
dened
Figure 4: Modality disparity across different configura- tions. Negative scores indicate performance gaps, with the largest disparities observed between audio and text.
4.4 DIRECTIONAL IMBALANCE
We test whether models behave consistently when swapping the roles of context and candidates. We define directional imbalance as Ax AAcoX Y) Ace(Y + X), the accuracy gap be- tween inverse configurations for (X, Y) and audio-text pairs exhibit notable asymmetries: Gemini 2.5 Pro drops by 8.8 points from T-VV to V-TT, and Qwen2.5- shows 16.6-point gap, while audio-text differences remain around 6-8 points. By contrast, audio-vision pairs are nearly symmetric but achieve much lower overall aceu- racy. These findings suggest that directional imbalance mainly arises in text-vision and audio-text pairs, likely reflecting training data biases toward text as the dominant output modality. Imbalance
AV..)): As shown in Fig. 5, rision-texi
Imbalance
Imbalance
Oved
1e 200
Performance Score
Performance Score
Performance Score
Figure 5: Directional imbalance: accuracy gaps between paired inverse settings among audio, vision and text. Models show clear asymmetries, especially in vision-text and audio-text pairs.
4.5 FAILURE CASE ANALYSIS
To better understand model errors, we prompt models like Gemini 2.5 Pro and Qwen2.5-Omm to generate reasoning alongside their answers. As shown in Fig. 6, we observe common failure cases that reflect modality performance gaps and alignment issues. Example (a) shows a mis- match between audio-to-text and audio-to-image reasoning: while the model correctly identifies a didgeridoo by text, it fails to select the matching image, revealing inconsistent grounding. In ex- ample (b), Qwen-2.5 Omni misinterprets spatial audio motion when switching from audio-to-text to text-to-audio, reversing the vehicle's direction. These errors highlight persistent asymmetries in cross-modal reasoning that only emerge when the same task is posed across different modality combinations.
4.6 TRIPLE-DOMAT QUESTION ANSWERING
Real-world omni-modal scenarios often present information jointly across modalities rather than in isolation. To approximate this setting, we extend XMODBENCH tasks to an audio-visual context, where both sound and vision are provided in the question stem, while the candidates remain in text.
We evaluate this dual-context configuration using the Gemini series. Compared with single-mon modality baselines, the results (see Appendix E) show modest but consistent gains, indicating that models can
8

---

Published as a conference paper at ICLR 2026
Which imager) belorges this sound you hear? Answer with A, C,
object,
(Answer D)
talephone finging
plaving harpsichond
playing tymppnn
playing didgesidoo
cudie clip pinched, continunus drone with not vicrating quslity. This the chanacteristic sound of digenddc -. telephone bell has high pitched.
very telephone rnn. Image shows device designed procuce evectly the kind
Response:
audio clip.. The other images didgerides D,
which produce entiraly differrnt of sounds
Q: Resed on the quatial andic for descriptiin) urban street soene and nhalyze the vehicle's directional movement. Which text description best matches your audio analysis?
A,
(Answer:
Toe Cueenal
ketionstale arere some
botioomally
cu mavcig
A has
vehicle prsseni
A car moxting Boon nigtt
[ntpolrtttnny [ ....
| [
right
risht
by Resporse:
description shotes
moving
Syp Resparas The audio hox vehicle sound that is The sound seens maving harizantally from ieft right, the volume and frequency changes suggest getting closer and then moving away..
from left right. In real 3D car moving from ieft to right would produce sound storts from the side mover towards right sicke Audio clip storts from leftt moves towards right...
this description has sound that
(b)
Figure 6: Failure cases. (a) Gemini 2.5 pro correctly identifies a didgeridoo in text but fails to match it with the corresponding image candidates. when switching candidates from text to audio. This cases illustrate asymmetries in cross-modal reasoning.
(b) shows Qwen2.5-0mni misinterprets spatial motion
benefit from simultaneous multimodal cues. However, the improvements are not always additive, suggesting that current models do not yet fully exploit complementary signals across modalities.
Table 3: Performance of Gemini models using both audio and visual context (A + V). Comparing A V with the best single-mon oagle-modality result (max(A, V)) reveals whether adding more context leads to better reasoning or causes harmful interference.
Setting
Gemini 1.5 Pro
Gemini Flash
Gemini 2.5
++T T
52.76
63.71
70.99
79.92
85.20
88.60
A+V
82.53 (+2.61)
79.84 (-5.36)
89.76 (+1.16)
5
KEY TAKEAWAYS FOR OLLM DEVELOPMENT
Our benchmark results serve as a diagnostic tool, revealing how underlying data composition and training methodologies shape OLLMS behaviors. By correlating performance patterns in 4.3 and 4.4 with model architectures and training reports, we derive three critical insights regarding inter- leaved data, task coverage, and how these effect OLLMS after post-training.
5.1 INTERLEAVED DATA REDUCES DIRECTIONAL IMBALANCE
key observation from our imbalance analysis the connection between interleaved training data and the reduction of directional imbalance when modalities are swapped, as seen in the results from Section 4.4.
Public official reports indicate that models such as Qwen2.5- 2mii (Xu et al., 2025) and Gemini-2.5 series (Comanici et al., 2025) incorporate massive scale interleaved multimodal corpora (e-g.. mixed audio-vision conversations). Our benchmark corroborates this: these models exhibit relatively small performance gaps between Audio->Vision and Audio tasks. This suggests that encounter- ing modalities interchangeably in context allows the model to build symmetric modal bridges.
Conversely, models trained primarily on non-interleaved datasets exhibit significant directional asymmetry. For instance, in the case of VideoLLaMA (Cheng et al., 2024), despite having strong backbones, models relying on open-source data with limited interleaved audio-vision instruction pairs show distinct bias. Models fail to generalize when the modalities are swapped (e.g., needing

---

Published as a conference paper at ICLR 2026
to distinguish between four video audio clips), indicating that insufficient interleaved supervision hinders directional robustness.
5.2 TASK COVERAGE GAPS
With its broad domain coverage, XModBench serves as diagnostic tool reveal blind spots in training for both open- and closed-source models. It identifies these gaps through performance inconsistencies, particularly regarding the diversity of audio types.
Spoken vs. Non-Spoken Bias: Many OLLMs prioritize speech as the primary modality for audio data. Models relying pre-trained, speech encoders (e.g., Whisper (Radford et al., 2023) in the Baichuan (Li et al., 2025) model) and trained predominantly on spoken language tasks exhibit sharp performance decline in non-spoken audio domains, such as environmental sound recognition and spatial reasoning. This indicates a significant seman- tic gap between linguistic speech and general acoustic signals, resulting in weakness for comprehensive audio understanding.
Deficiencies in Music Understanding and Spatial Reasoning: Distinct gaps in specific categories shows evidence of missing training data. For example, despite its high overall capacity, Gemini 1.5 demonstrates limited capability in musical reasoning, suggesting an absence of related data in its training corpus. Similarly, Echolnk-RI struggles with spatial- vision tasks relative to related families, pointing to lack of spatial-centric visual content.
5.3 THE IMPACT OF POST- TRAINING ON ALIGNMENT
comparative analysis Echolnk-RI and the Qwen2.5-One series illustrates how post-training strategies can reshape and even degrade multimodal alignment when data is weakness, despite the advance reinforcement learning used to enhances cross-modal reasoning (Shao et al., 2024). Qwen2.5 10122.50mmit integrates interleaved multimodal conversations during post-training, Echolnk-RR RI is further fine-tuned on the data derived from Omnilnstruct corpus (Li et al., 2024b) focus on spoken instruction following. This divergence in training objectives leads to notable behavioral differences:
Although Echolnk-RI R/ is based on the well-aligned Doven2.5-0mit it shows larger modal- ity disparities and directional imbalances. Fig.4 shows it performs worse in Audio->Texx and Vision-<Text modality disparity, and exhibits higher imbalance in Audio Visoo- and Audio-Text pairs in Fig. 5. This suggests that the cross-modal alignment from interleaved pre-training can be eroded if the post-training stage shifts to non-interleaved data.
Despite heavy tuning, Echolnk fails to improve in spatial tasks (Fig. 2), showing that post- training requires comprehensive task coverage. Incomplete fine-tuning cannot fill these gaps and may even degrade the model's multimodal capabilities.
Beyond simple evaluation, XModBench serves diagnostic tool for model development. While training data for SOTA models remains a black box, XModBench reveals the impact of training strategies and task coverage through controlled comparisons from them. Our open-source generation tools also offer the potential to extend to new domains. By using unique modality-swap design, XModBench exposes hidden flaws that current benchmarks miss, making it essential for building robust next generation onni-modal language models.
CONCLUSION
We introduced XModBench, a benchmark for diagnosing cross-modal consistency in omni- language models. By systematically interleaving audio, vision, and text across diverse tasks, XMod- Bench enables fine-grained evaluation of modality disparities, directional imbalances, and modality invariant capability. Our results show that audio remains the most challenging modality, that models often behave asymmetrically in inverse settings such as text-vision and audio-text, and that com- bining audio and vision yields only modest gains. Overall, while current systems are strong in perception and language, they still lack stable and consistent reasoning across modalities, leaving ample room for progress toward truly modality-agnostie intelligence.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our study does not involve private or sensitive personal data. All audiovisual samples are obtained from publicly available official sources, including previously published research datasets, content hosted on established open source platforms such as Hugging Face and Kaggle.
For newly generated labels and annotations, we perform manual verification to ensure correct- ness and to remove any potentially inappropriate content. All web-curated data are from publicly accessible and previously published sources without requiring special authentication. All materials used soiely for non-commercial academic research. We do not redistribute copyrighted video or audio; only derived features, annotations, and evaluation results are released.
REFERENCES
V.R. Algazi, R.O. Duda, D.M. Thompson, and C. Avendano. The cipic hrtf database. In Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01 No.01TH8575), pp. 99-102, 2001. doi: 10.1109/ASPAA.2201.. 969522....
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh. Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing vision- languala models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024.
Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zis- serman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16867-16876, 2021.
Ssu-Yen Chen, Chao-Chun Hsu, Chuan-Chun Kuo, and Lun-Wei Ku, Emotionlines: An emotion 2018.
corpus of multi-pan -arty conversations. arXiv preprint
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476, 2024.
Wey Yeh Choong. Yangyang Guo, and Mohan Kankanhalli. Vidhal: Benchmarking temporal hallu- cinations in vision llms. arXiv preprint
2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626.. 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2024a. URL httLtt////axiie http:::s///xxxv..ooooooo abs/ 2306.13394,
Chaoyou Fu, Yuhan Dai. Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang. Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal lims video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108-24118, 2025a.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint 25010955...1197, 202bb.
Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.0032. 2024b.
11

---

Published as a conference paper at ICLR 2026
Magdalena Fuentes, Bea Steers, Pablo Zinemanas, Martin Rocamora, Luca Bondi, Julia Wilkins, Qianyi Shi, Yao Hou, Samarjit Das, Xavier Serra, et al. Urban sound & sight: Dataset and benchmark for audio-visual urban scene understanding. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 141-145. IEEF, 2022.
GbotHQ. Ocr dataset rendering.
https: / com/ Cbot 2024. MIT License, Accessed: YYYY-MM-DD.
GitHub:
ocr- sor-ddataset-renderiin
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng. Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, al. Av-odyssey bench: Can your multimodal llms really understand audio-visual information? arXiv preprini arXiv:24/2.02611 2024.
Hao-Han Guo, Yao Hu, Fei- Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, and Kun Xie. Fireredtts- Is: An upgraded streamable foundation text-to-speech system. arXiv preprint arXiv:2503.20499, 2025.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025.
Dongjin Kim, Sung Jin Um, Sangmin Lee, and Jung Uk Kim. Learning to visually localize sound sources from mixtures without prior source knowledge. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 26467-26477, 2024.
Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic video generation. In European Conference on Computer Vision, pp. 34-50. Springer, 2022.
Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. Creating multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2):522-535, 2018.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang. Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024a.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19108-19118, 2022.
Yadong Li. Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Bain--5--ommi-1.5 technical report. arXiv preprint arXiv:2501.15368 2025.
Yizhi Li, Ge Zhang. Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang. Jian Yang, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint 2rxhv:2409.15272 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui Hc, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233 Springer, 2024.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem. and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2024.
Lidong Lu, Guo Chen, Zhiqi Li, Yicheng Liu, and Tong Lu. Av-reasoner: Improving and bench- marking clue- stue-grounded audio-visual counting for milms. arXiv preprint urXiv:2506.05:3 2025.
Piotr Majdak, Péter Balazs, and Bernhard Laback. Spatially oriented format for acousties: A data exchange format representing head-related transfer functions. In Audio Engineering Society Con- vention 134, Audio Engineering Society, 2013.
12

---

Published as a conference paper at ICLR 2026
Juan F. Montesinos, Olga Slizovskaia, and Gloria Haro. Solos: A dataset for audioovisusi music
analysis. 2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP),
pp. 1-6, 2020. URL http://p.p..settttcsccccooooo.a.
Andrada Dataset: gt wuw.-rattssstrua-.--eaoaaaaaaaaa
Olteanu.
Gtzan
dataset:
Music
genre classification.
Kaggle
https::/wwee http:///wwwwwwggel..e/// com/ dwt.aattttttaaaeeeeeeeee peeett//nnddaaltteeeen. /tt.c////attttttaaaaaaaateeeaaa
2024.
Accessed: YYYY-
MM-DD.
OpenAl. Introducing gpr-5. https: htttps://openai.. http:///eaaa..cocooeeee///tttrtr -pt- Au- gust 2025.
Jean Park, Kuk Jang, Basam Alasaly, Sriharsha Mopidevi, Andrew Zolensky, Eric Eaton, In- sup Lee, and Kevin Johnson. Assessing modality bias in video question answering benchmarks with multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 19821-19829, 2025.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Rada Mihalcea, and Erik Cambria. MELD; A multimodal multi-party dataset for emotion recognition in conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 527-536, 2019.
Ville Pulkki. Virtual sound source positioning using vector base amplitude panning. Journal of the audio engineering society, 45(6):456-466, 1997.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-285188 PMLR, 2023.
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.191/ 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. URL httth:::///aaaae org abs/ 22022/40220 03300.
Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal anno- tations of sound events. Advances in neural information processing systems, 36: 36:72931-7295 2023.
Yixuan Su, Tian Lan, Huayang Li. Jialu Xu, Yan Wang. and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arX5v:2305.16355, 2023.
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint wwXiv:2406.15700 2024.
Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. arXiv preprint arXiv:2410. 18325, 2024.
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.0553. 2024.
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu. AiTi Aw, and Nancy Chen. Audiobench: universal benchmark for audio large language models. arXiv preprint 2406. 240. 2024.
13

---

Published as a conference paper at ICLR 2026
Christoph Wendler. Renderedtext. Hugging Face Dataset: httpos//hucrrnggacce. datasets http///wwnweeeeeeeeeeeeeeeeeeeee Rtw.eeerr/Redeeeedeeeeee 2024. Accessed: YYYY-MM-DD.
Zhenghao Xing, Xiaowei Hu, Chi-Wing Fu, Wenhai Wang, Jifeng Dai, and Pheng-Ann Heng. Echoink-rl: Exploring audio-visual reasoning in multimodal Ilms via reinforcement learning. arXiv preprint arXiv:2505.0400 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215 2025.
Pinci Yang. Xin Wang. Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering videos. In Proceedings of the 30th ACM inter- national conference on multimedia, pp. 3480-3491, 2022.
Yudong Yang, Jimin Zhuang. Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li Zejun Ma, and Chao Zhang. Acvubench: Audio-centric video understanding benchmark. arXiv preprint arXiv:2503. 19955,, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12):wwa4433 2024.
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visusi question answering 360deg videos. In Proceedings the IEEE/CVF Intern- tional Conference on Computer Vision, pp. 2031-2041, 2021.
Xiang Zhang, Senyu Li, Ning Shi, Bradley Hauer, Zijun Wu, Grzegorz Kondrak, Muhammad Abdul- Mageed, and Laks VS Lakshmanan. Cross-modal consistency multimodal large language mod- els. arXiv preprint 2rXiv:241.00922 2024.
Yunhua Zhang, Ling Shao, and Cees GM Snoek. Repetitive activity counting by sight and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10070-14079, 2021.
Hao Zhong, Muzhi Zhu, Zongze Du, Zheng Huang, Canyu Zhao, Mingyu Liu, Wen Wang, Hao Chen, and Chunhua Shen. Omni-rl: Reinforcement learning for omnimodal reasoning via two- system collaboration. arXiv preprint urXiv:2505.2025: 2025.
Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-ommi: Towards -isuul reasoning with temporal alignment across modalities. arXiv preprint 20X.v 17862, 2025.
14


Reviews:
[{"review_id": "xWfbLQYkuG", "reviewer": "Reviewer_2qMR", "summary": "This paper introduces XModBench, a large-scale tri-modal benchmark specifically designed to measure cross-modal consistency in Omni-modal Large Language Models (OLLMs) by systematically covering six cross-modality directions for audio, vision, and text. XModBench comprises over 60,000 multiple-choice questions across five task families and 17 subtasks, enabling diagnostic assessment of task competence, modality disparity, and directional imbalance.", "questions": "same as weakness", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive Diagnostic Scope: XModBench provides a large-scale, systematically balanced tri-modal QA benchmark, covering all six modality permutations (audio, vision, text) for both the context and candidate answers. The benchmark covers five diverse task families (perception, spatial, temporal, linguistic, external knowledge), each with multiple subtasks.\n\n- Detailed Empirical Analysis: The authors conduct a detailed empirical analysis of cutting-edge OLLMs, including a performance breakdown by task and modality configuration (Table 2). This analysis effectively identifies the significant lack of capability or competence in current OLLMs within the audio domain.", "weaknesses": "- XModBench primarily focuses on isolated cross-modal alignment (e.g., T→V, V→T) and fails to cover true mixed tri-modal capabilities (e.g., Image+Vision+Audio→Text/Image). This combined modality reasoning is arguably the critical differentiator separating OLLMs from Multimodal Large Language Models (MLLMs) and specialized speech models.\n\n-  The data curation shows an over-reliance on GPT-5 as the primary question generation tool. However, the quality assurance (QA) or filtering process for this synthetic data is not clearly elaborated. This risks labeling XModBench as a 'silver' dataset rather than a 'gold' standard, where prioritizing data quality over sheer quantity is paramount.\n\n-  Contextualization against MLLMs: The analysis lacks comparison against the performance of MLLMs focused on traditional ASR or image-to-text (I2T) tasks, such as Qwen-VL or Intern-VL. It remains unclear whether OLLMs maintain a performance advantage in these specific subdomains when compared to these more focused MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rOIpzy6qso", "reviewer": "Reviewer_3qp2", "summary": "This paper introduces XModBench, a large-scale tri-modal benchmark (audio, vision, text) designed to evaluate cross-modal consistency in omni-modal large language models (OLLMs). The dataset contains 60K multiple-choice QA pairs rendered in six modality directions (e.g., audio→text, image→audio), covering five task categories: perception, spatial reasoning, temporal reasoning, linguistic understanding, and external knowledge. The authors benchmark several frontier models (e.g., Gemini 2.5 Pro) and report three key findings: (i) OLLMs still struggle with spatial/temporal reasoning, (ii) performance drops significantly when audio replaces text, and (iii) models show strong directional imbalance (e.g., text→vision vs. vision→text). The benchmark aims to serve as a diagnostic tool for measuring modality-invariant reasoning.", "questions": "Dataset quality\nHave you conducted human verification on a subset of the 60K samples? If so, what is the estimated annotation error rate?\n\nDistractor bias\nCan models guess correct answers without context? Please provide “no-input” or “noise-input” baselines to quantify answer-option bias.\n\nModality swap analysis\nDo any evaluated models train on interleaved multimodal corpora (e.g., narrated video, audiocaps)? If yes, do they exhibit smaller directional gaps?\n\nTable 2 clarity\nWould you consider splitting Table 2 into multiple focused tables (e.g., task competence, disparity, imbalance) to improve readability?\n\nBelow-random performance\nFor conditions where models score <25%, what is the failure mode? Instruction refusal? Systematic misalignment? Poor distractor construction?\n\nBenchmark extensibility\nDo you plan to release tools for adding new modality pairs (e.g., text↔3D, audio↔video)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper explicitly targets cross-modal consistency, a dimension often ignored in existing multimodal benchmarks.\n\nEach question instance is rendered across all six modality mappings, enabling controlled comparison and directional analysis.\n\nLarge scale and broad coverage.\nThe dataset includes >60K QA samples spanning 17 subtasks, with balanced modality construction.\n\nRelevance to current model trends.\nAs many new models claim “omni-modality,” this benchmark fills a timely evaluation gap.", "weaknesses": "Table 2 is overloaded and hard to interpret.\nThe key conclusions (e.g., text→image > audio→text; Gemini has lowest variance) are meaningful, but the table is dense, lacks focused analysis, and could be split into smaller tables aligned with each main claim.\n\nInteresting modality-swap results but no deeper investigation.\nThe paper observes asymmetric performance (e.g., vision→text vs. text→vision) but does not analyze why. For example:\n– Do any models use interleaved multimodal training data?\n– Do models with such data show smaller swap gaps?\n\nDataset quality control is unclear.\nThe benchmark claims 60K samples but does not report human validation, error rate, or annotation quality checks.\n\nNo discussion of answer-option bias.\nSome modalities may allow shortcut guessing (e.g., lexical cues in text choices). There is no “noise-input” baseline to rule this out (e.g., Gemini with shuffled / blank modality input).\n\nLack of analysis for <25% performance cases.\nSeveral settings score worse than random guessing (25% for 4-choice MCQ), but the paper does not explain whether this is due to instruction following, noisy inputs, or poor distractor design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6gvIioNlW", "reviewer": "Reviewer_a37s", "summary": "The paper introduces XModBench, a large-scale, tri-modal (audio, vision, text) benchmark designed to evaluate omni-modal large language models (OLLMs). The core objective is to move beyond task-specific accuracy and measure cross-modal consistency—the ability of a model to produce consistent answers when the same semantic content is presented in different modalities. XModBench comprises 60,828 multiple-choice questions systematically covering five task families (perception, spatial reasoning, temporal reasoning, linguistic understanding, and external knowledge) and all six possible cross-modal directions between the three modalities. The authors use this benchmark to evaluate a range of OLLMs, including the Gemini series and several open-source models. The key findings demonstrate that even state-of-the-art models like Gemini 2.5 Pro lack true modality-invariant reasoning, exhibiting significant performance drops on spatial/temporal tasks, major disparities when inputs are switched (e.g., text vs. audio), and directional imbalances (e.g., V->T vs. T->V).", "questions": "Given the impressive scale of the benchmark, have the authors considered releasing a standardized \"lite\" subset? A smaller, balanced subset would significantly lower the barrier to entry, allowing for more rapid experimentation and broader adoption by the research community.\n\nCould the authors provide an estimation of the computational cost to run the full XModBench evaluation? Specifically, what is the approximate token usage (input and output) for evaluating an API-based model like Gemini, and what are the estimated GPU-hours for an open-source model?\n\nThe performance of Gemini 2.5 Pro is a key data point. While its failures on spatial/temporal tasks are clear , its overall superiority is also evident. Do the authors have any insights or hypotheses as to why this model demonstrates relatively better cross-modal consistency and overall competence compared to the other models tested?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a crucial question: are OLLMs truly modality-invariant? It moves evaluation beyond simple accuracy on multimodal tasks and proposes a novel, principled method for measuring cross-modal consistency. The design, which permutes modalities for the same semantic question, is the paper's core strength.\n\n\n\n2. The benchmark is comprehensive. It contains over 60K questions , covers 5 diverse task families (from perception to external knowledge) , and 17 subtasks. This breadth ensures that the findings are not artifacts of a single domain.\n\n\n\n3. High-Quality Curation. The authors detail a rigorous data curation and verification process. The explicit use of \"human in-the-loop verification\" and multiple rounds of testing by annotators addresses major concerns about the quality and ambiguity of web-sourced or generated data .\n\n4. Actionable Diagnostics. The paper doesn't just rank models. It provides specific, interpretable diagnostic metrics—modality disparity and directional imbalance —that allow researchers to pinpoint where and how their models are failing. The failure case analysis in Section 4.5 and Figure 6 reinforces this with qualitative examples", "weaknesses": "1. Accessibility and Cost: The benchmark's primary strength—its scale—is also a potential weakness for adoption. Evaluating a model on 60,828 question-answer pairs, many of which involve multiple modalities, appears to be a computationally expensive process. The paper does not mention the availability of a smaller, standardized \"lite\" subset for researchers with limited compute. Furthermore, no information is provided on the practical costs of evaluation, such as total token usage for API-based models (like the Gemini series) or GPU hours for open-source models. This omission could be a significant barrier to widespread adoption and reproducibility.\n\n2. Limited Analysis of SOTA Performance: The paper's results clearly establish Gemini 2.5 Pro as the top-performing model, yet one that still has significant flaws. However, the analysis is largely limited to reporting these scores and failures. The paper would be strengthened by a deeper discussion hypothesizing why this model performs so much better on average than its open-source counterparts. Is it its training data, a specific architectural choice, or better-aligned encoders? A more in-depth analysis of the causes of SOTA performance (and its limitations) would be more impactful than just documenting the performance itself. Could be helpful if we could analyze how it was trained even with a guess.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFkDVH67n0", "reviewer": "Reviewer_DEkV", "summary": "This paper presents XModBench, a large-scale tri-modal benchmark for evaluating cross-modal consistency in omni-language models (OLLMs) that handle text, vision, and audio.\nUnlike previous multimodal benchmarks, XModBench focuses on whether models maintain consistent reasoning across modalities.\nIt includes 60K multiple-choice questions spanning five task families and six modality directions, and introduces diagnostic metrics for task competence, modality disparity, and directional imbalance.\nExperiments on leading OLLMs (e.g., Gemini 2.5 Pro, Qwen2.5-Omni) show that current models still lack modality-invariant reasoning, especially in spatial, temporal, and audio-related tasks.", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths\n\nNovel Focus: Targets an important but underexplored problem — evaluating cross-modal consistency rather than just multimodal performance.\n\nComprehensive Benchmark Design: Covers six modality directions and five task families, providing a balanced and systematic tri-modal evaluation.\n\nInsightful Diagnostics: Introduces clear metrics (modality disparity and directional imbalance) that reveal hidden biases and asymmetries in current OLLMs.", "weaknesses": "This paper compares model performance differences across modalities for the same question. However, it does not discuss whether such differences are caused by information loss during modality conversion — for example, when a video or audio question is converted into text, the textual description cannot fully capture the visual or auditory content.\n\nThe paper does not explore whether specific training strategies or data construction methods could help mitigate these shortcomings in cross-modal consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)