OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities
Authors: Lichang Chen
Hexiang Hu
Mingda Zhang
Yiwen Chen
Zifeng Wang
YANDONG LI
Pranav Shyam
Tianyi Zhou
Heng Huang
Ming-Hsuan Yang
Boqing Gong
Venue: ICLR 2025 Poster (2025)
Keywords: Omni-modality Language Model; Omni-Eval; Omni-Reasoning
Primary area: datasets and benchmarks
TL;DR: A benchmark for evaluating the reasoning capabilities of Omni-Modality Language Models

Abstract: We introduce \textbf{OmnixR}, an evaluation suite designed to benchmark state-of-the-art Omni-modality Language Models (OLMs), such as GPT-4o and Gemini. 
Evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. 
Particularly, the user message might often consist of multiple modalities, such that OLMs have to establish holistic understanding and reasoning across modalities to accomplish the task.
Existing benchmarks are limited to single-modality or dual-modality tasks (e.g., image+text or video+text), overlooking comprehensive multi-modal assessments of model reasoning.
To address this, OmnixR offers two evaluation variants: (1) OmnixR-synth: a synthetic dataset generated automatically by translating text into multiple modalities—audio, images, video, and hybrids Omnify!. (2) OmnixR-real: a real-world dataset, manually curated and annotated by experts, for evaluating cross-modal reasoning in natural settings. 
OmnixR  presents a unique evaluation towards assessing OLMs over a diverse mix of modalities, such as a question that involves video, audio, and text, providing a rigorous cross-modal reasoning testbed than any existing benchmarks.
Our experiments find that all state-of-the-art OLMs struggles with OmnixR questions that require integrating information from multiple modalities to answer. 
Further analysis highlight differences in reasoning behavior and underscoring the challenges of omni-modal AI alignment.

Published as a conference paper at ICLR 2025
OMNIXR: EVALUATING OMNI-MODALITY LANGUAGE MODELS ON REASONING ACROSS MODALITIES
Lichang Chen 12 Hexiang Hu', Mingda Zhang', Yiwen Chen' Zifeng Wang', Yandong Li', Pranav Shyam', Tianyi Zhou", Heng Huang", Ming-Hsuan Yang', Boqing Gong' Google DeepMind"; University of Maryland, College Park2
ABSTRACT
We introduce OmnixR, evaluation suite designed to benchmark state-of-the- art Omni-modality Language Models (OLMs), such as GPT-4o and Gemini. Eval- uating OLMs, which integrate multiple modalities such text, vision, and audio, presents unique challenges. Particularly, the user message might often consist multiple modalities, such that OLMs have establish holistic understanding and reasoning across modalities to accomplish the task. Existing benchmarks are lim- ited single-modality dual-modality tasks (e.g., image+text or video+text), overlooking comprehensive multi-modal assessments of model reasoning. To ad- dress this, OmnixR offers two evaluation variants: (1) Omni X syn- thetic dataset generated automatically by translating text into multiple modali- ties-audio, images, video, and hybrids (Omn !y (2) O a real- world dataset, manually curated and annotated by experts, for evaluating cross- modal reasoning in natural settings. Omni R presents a unique evaluation to- wards assessing OLMs over a diverse mix of modalities, such as a question that in- volves video, audio, and text, providing a rigorous modal reasoning testbed than any existing benchmarks. Our experiments find that all state-of-the-art OLMs struggles with Omni questions that require integrating information from mul- tiple modalities to answer. Further analysis highlight differences in reasoning behavior and underscoring the challenges of
AI alignment.
INTRODUCTION
Recent advances in Omni-modality Language Models (OLMs) (OpenAI, 2024b; Gemini-Team. 2024b) has pushed the boundaries of Al by enabling more comprehensive understanding of real- world inputs across diverse modalities, e.g., text, vision, audio, (Lu et al., 2019; Gan et al., 2020; Akbari al., 2021; Zellers et al., 2021) and generating outputs that are more aligned with human communications (Lu et al., 2024; Zhang et al., 2024; Gao et al., 2024).
However, the evaluation of these sophisticated OLMs presents unique challenges. While traditional benchmarks (Ims, 2024) have predominantly focused on models that handle single or dual modali- ties, such vision-language or video-text pairs, they fail to capture the complexities that arise when multiple modalities are involved. In real-world scenarios, user inputs are rarely confined to one or two modalities. Instead, they often consist of diverse combinations of text, images, videos, and audio, necessitating a holistic understanding and reasoning across information presented in these modalities for OLMs to effectively perform tasks. This mismatch between existing evaluation meth- ods and the multimodal capabilities of state-of-the-art OLMs has left significant gap in the assess- ment of these models.
One common flaw in existing OLMs is their inconsistent behavior when presented with the same question in different modalities or mixtures of modalities. Figure presents an example on the Gem- ini 1.5 Flash (Gemini-Teame 2024a) (similar behaviour also observed in other OLMs, see Section 3.2 for analysis). Particularly, when the same math question is presented different modalities, such as rendered as image input, spoke out audio input, the model produces varying responses that ex- hibit significant performance discrepancies, i.e., different reasoning bevhiours or different answers.
"Correspondence to Lichang Chen, bobchen @und. edu, Hexiang Hu,hexi integooggggggggl1ooo e com, and Boqing Gong, boong@googge..coo com.

---

Published as a conference paper at ICLR 2025
Text Input:
The cost producingg pondsonng-ccmmeeet cameras =X
[Prompt] step by step then output seowor in format "Tha arcwen 187" the and.
* Whet evernge comente mode? What la the marginal oost producing cameras?
[Responsetto Texti) find the average cose pratuary camaras gloan cosl function BCXX 2 ees Calcudata
cost of producing 10 cameraa:: $CX10) 2 =1001 2 1000 10028 average coor tonsl cost diiied the Streot/aweage Creet traef(21oog (Cggggg ttooo/0007 28. The closest option lo: answerts
Average 10000unt,, Maigtno
Auerage Manginal
OmniLM Gemini or
Marginal
Auerge Mang
[Responsecto Videol) answerts C.
[Responsatto Audiall
producting
SC/10)) per marginal
Video Input:
sweeage
o125. manginal
produsing
$C111)
Audio Input:
GPT-4o
[Responsetto Imagel) sotal producing camerus total dhiddd 9CM 2 The SC/10/10 12 10" 10"3 1002/10 18 200
producing
Image Input:
mangnal cost
marginal cost producinng camerss #C11 30100012 18000/camerss. Thersfore, the correct answer le: Thhe annwer E1
Figure I: Reasoning Behavior of OLM Varies across Modalities. Taking Gemini-1.5-Flas as an example, on text question, the reasoning behaviour is expected and the answer is correct. When the same question is rendered an image, the model generate reasonable reasoning but incorrect answer. On the video or audio representation of the same question, the model generates no reasoning and produces incorrect answers.
This observation indicates lack of robust cross-modal information integration and reasoning capa- bilities in existing OL.ss. Such inconsistency not only undermines the reliability of these models but also highlights the limitations of current evaluation benchmarks that do not adequately assess performance across diverse modality combinations.
To bridge this critical evaluation gap, we introduce OmnixR, an evaluation suite specifically de- signed to benchmark the reasoning performance of OLMs across a wide range of modalities. Unlike existing benchmarks that are limited to maximum of two modalities, Omnix R provides a com- prehensive testbed that includes complex modality combinations such as vi deo + audio o + text and i image + audio + text, offering more rigorous and holistic evaluation of these models' capabilities. Specifically, OmnixR contains two subsets of the data:
Omni x RSYNTH: a synthetic reasoning dataset constructed with a scalable and low-cost automatic method (i.e., Omnify!) !) to translate information embedded in text to various modalities audio, images, video, and hybrids of them.
Omni REEAL: woal-world reasoning dataset manually collected and annotated with expert an- notators, for evaluating cross- moosl reasoning in the realistic distribution.
In construction of Omni R Om translates text-based inputs into various other modali- ties, such as images, audio, and video, as well as their hybrid combinations, using programmatic text rendering services, programmatic video construction pipeline, and state-of-the-ar text-to-speech service. This scalable synthetic dataset ensures diverse and robust dataset that challenges OLMs to demonstrate their cross-modal reasoning abilities. Meanwhile, Omni x RRFAI. develops a realis- tic test environment for evaluating omni-moda reasoning. Particularly, we crawled 100 YouTube videos whose topics are related to math, physics, chemistry and coding, and manually curate, convert and annotate the quiz questions from those videos, ensuring that each question is associated with multiple variants, each in one modality or a hybrid of many modalities. With both complementary subsets, Omni x R allows us better assess how well OLMs can reason across different modalities and integrate information in a way that mirrors human-like understanding.
Our evaluation of state-of-the-art OLMs on Omnix R has yielded several important findings. No- tably, OmnixR is the first benchmark that quantitatively measured the aforementioned omni-modal behaviour discrepancy, especially in scenarios requiring deep reasoning across multiple modali- ties. Moreover, we also observe that some simple prompting strategy that exploits the underly- ing data creation logic in OmnixXRsyTHH ie., Extract the informat i and Then Answer (ETA prompting), could significantly improve every omini-modality language model's behaviour consistency and final answer accuracy on Omni x RRiiTH These results suggest that the main strug- gle of current model is to establish holistic understanding across modality, where the need to inte- grate and reason across different forms of data becomes crucial. When evaluated on Omni x RREAL where the information across modality is naturally distributed and blended with noises, OLMs can no longer rely on a simple prompting strategy to alleviate the omnimodal behaviour inconsistency, indicating the demand of further training for future omnimodal language models.
2

---

Published as a conference paper at ICLR 2025
Interlesved
Audio
Video Audio and Image Audio)
Quaetion
Image
Bugaun
Text Question
Cunnce Eugare cormran green lagelata the for altooradie muaiito soceel
Omnity!
Figure 3: We propose Canify! to create the synthetic omni-modalit, evaluation data from the original text benchmarks.
OMNIXR R BENCHMARK
In this section, we introduce Omnify! a scalable and low-cost automatic method designed to translate text into various modalities, including audio, image, video, and combinations thereof. The overarching goal of by! to build up a scalable method to generate omni-modality data while keeping information the same across them for evaluating OLMs' reasoning capabilities across modalities. We construct the Omnix benchmark in two subsets: (1) Omni X RSYNTH synthetic amni-modal reasoning evaluation dataset derived from applying Omnify! on the MMLU- Pro (Wang et al., 2024). (2) Omni x RREAL: real-world omni-modaa reasoning evaluation derived from Youtube, which then processed and annotated by human experts.
OmniXHsynt synth
OMNIFY!
Modalities
Categorias
1400 Semples
Text to image. Though there many ways to convert text into images, like using image generation models (e.g., Imagen-3 (Baldridge et al., 2024), DALLE-3 (OpenAI, 2024a)), however, the scemingly appealing text-to- image generation models make it challenging control quality; they cannot ensure the gen- eration contains all the information we need to answer a question. Before figuring out how to judge the quality of and information in the gen- crated images, is not viable to use image gen- erators to scale up the mapping from text to im- ages. Since our main goal is to evaluate models' approach in this work: rendering a canvas and then write the words on it. Given the images in- put, we expect the models can achieve the same performance as they read text in this ideal scenario, where extra noises, information losses, or variations introduced by the text-to-image mapping process. Specifically, we use PIL' to create new image with a white background and the text is drawn onto the image with black color. The engineering details/efforts can be found in Appendix I.
leat, Imaig Audio Minno Vndoord dio.
Mart, ii syoos, Chemmsyy Compuseer de eine...
193 Exemples eoch Eragwy 1400
OmniX Real
20%
25%
Turs, Imagg, Audio, V den, #'tee Sumptas 400
Chemistry
Physice
30%
25%
Figure 2:
The overview of Omnix Raysr and Omnix
reasoning capability, we start from the simplest
Text to Audio We initially attempted to use Google Text-to-Speech" (TTS) for text-to-audio con- version. However, we encountered challenges with the mathematical equations. To address this, developed two-step process. First, convert the original text, if it contains mathematical equations, into a format that is easy to speak orally. The details for the conversion could found in Table 7. Then, use TTS engine generate the audio, which contains the full information of the original text question.
Text to Video Like text-to-image generation models, there exist Sora (Brooks et al., 2024) and Veo (Google, 2024) we could leverage to map text to videos. However, they would incur the same problems as described in the text to image: quality control, time consumption, and computational cost. The main objective with videos here is to evaluate a model's capabilities a video input, which is a series of images from model's view, and then reasoning to solve the problems. We fulfill this objective again using simple approach to generating the video data from text as follows. Based on our image generation process, we render a series of images where each
understanding
'attps: http///ttttttteeeeeeeeeteeeeeset cwm/ wwtp//tttooooeeeeeeeeeee wtt.:annppllew//illee.. "ntppa: //ell/dd goog ie. we..e.....m/.e.mm.......oo...... www.oootttttteeeeoooooooo ttx.ttttttpppppooo

---

Published as a conference paper at ICLR 2025
1) Guage soluee whtch repiliaaa
B
DUAUIRI dueeee disccti
a0 aa Gju
2/w, then 21 c(
1 an aM/e 2000
50
50,00 Clpu
64
34 26
tu x'yur'' = (AA
380 oye
280
cPP solval
concition froor forwerd rection
aermaaing tetooutee and acromses
(a) Coding
(b) Physics
(c) Calculus
(d) Chemistry
Figure 4: Visualization of Examples in the Omnix REEAL. set.
image contains one or several words from the text. We ensure that the information in the text is fully translated to the video. The input text is split into individual words first. Then OpenCV to create video writer object with specified frame rate, i.e.. 1 FPS, and frame size (300x100 pixels). Each word is converted into an image using the text-to-image method. Finally, these images are combined sequentially to create video frames.
2.2 OMNTX OMNTX&SVNTH SCALABLE SYNTHETIC OMINT-MODAL. REASONING EVALUATION
Our initial choices of the text benchmark for Omnify! are Are-Challenge (Clark et al., 2018) and GSM8K (Cobbe al., 2021), but we identify the potential data contamination problems on these two benchmarks as Gemini-1.5 1.5-pro (Gemini-Teane 2024a) can achieve over 99% on GSM8K (results are shown in Table I). is very likely that contaminated OLMs just capture the part of the information they need from the video/audio questions and use their 'memory" to give correct answers, which cannot reflect the actual reasoning ability of the models. Thus, we choose MMLU-Pro (Wang al., 2024), which is augmented from MMLU with ten options per question and released in June after the 1.5rri--.--PPro0001 release, as the text benchmark to Omni fy!. In this way, we minimize the contamination influence, enabling more accurate study of OLMs' omni-reasoning. We randomly sample 100 questions from each of the 14 categories in MMLU-Pro to construct Omni RSYNT- Some examples for Audio and Video modalities are available*.
2.3 OMNIX R REAL: HIGH-QUALITY REAL- WORLD OMINI-MODAL REASONING EVALUATION
We crawl the video data from youtube and then transcribe it into different modalities to develop realistic set as a valuable addition to the OmnixR.
Video: We select four categories that require dense reasoning in real-world scenarios: Mathematics, Coding, Physics, and Chemistry. Videos are sourced from popular educational channels, such as MIT OpenCourse. Two human annotators, spend approximately 30 hours each to review 100 videos (200 total) and identify those containing non-trivial questions that demand substantial reasoning to solve. From these, 100 videos are carefully selected to construct high-quality set, Omni x REEAL Each video clip is curated based on the following criteria: (I) it must contain one or more key frames that provide all the necessary information to solve the question; (2) the clip should exclude the answer to maintain the challenge; (3) some misleading or irrelevant frames are intentionally included to assess the model's 's robustness in reasoning. Image: We manually find the key frame(s) which contain the question information. It should be noted that in some cases, there might be several frames containing the relevant information, where we will crawl two three frames and merge them together into one image. Text: Five human annotators transcribe the text from the video with the help of the tools, c.g.. Gemini. All the open-ended generation questions are transferred into multiple choice questions to make the benchmark casy-to-use. Audio: The original audio will be checked first, which extracted from the video we crawled. to answer the question, then we will just keep and use it. However, there are many cases where the audio does not contain the enough information for answering the questions, e.g., the instructor shows a slide and asks "solve the problems in the slide", where the problem is shown in image. In that scenario, we will use the same method in Omnify! !y! to transfer the transribed text into audio by Google TTS.
contains all the information for OLMs
aattp:://cloud www.eeoowwwooooo...oooooo com/vertex-aa wwtgg//eaagieeeeeeeeeeeeee htt.e//tieeeeeee//e///////eeeees intspa://anonymott 4opan. act.nceeeeeeee...ee raneee///mmm xR- Rxamples-7961 79679
4

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTS AND FINDINGS
3.1 EXPERIMENT SETUP
Models. We mainly test three series of models: Gemini GGemii-TTaam 2024a), i.e., Gemini- 1.5-Pro. and Gemini-1.5-Flash. OpenAI-GPT (OpenAI, 2024c), i.e., GPT-4o and GPT-4o-mini, Anthropic-Clane Claude (Anthropic, 2024), i.e., Claude-3-Opus, Claude-3-Sonnee Claude-3-Haiku, More details about the test models are shown in Appendix E.
CoT Prompting. The standard setting in MMLU-Pro (Wang et al., 2024) is to use Chain-of- Tho)ght(CoT) prompting to elicit the reasoning ability of the OLMs for more comprehensive evaluations. Following them, we use CoT with O-shot, as our standard setting, i.e., the prompt used for evaluation is "Think step by step then output the answer in the format of "The answer (X)" at the end."
Extract-Then-Ansee (ETA) Prompting. In addition, we employ Extract-Then-Aee Then-Answer (ETA) prompting, leveraging the benchmark's 's inherent structure. This method involves first extracting the textual content and then using the OLMs' language capabilities for reasoning to provide answers based on the transcriptions. To prevent potential hackings Omni xR, we transparently demon- strate this approach in our benchmark, aiming for comprehensive evaluation of OLMs. Specifi- cally, the prompt Pleaase extract the text from image/audio/videos' instructs the OLMs to function as text extractors. The extracted text from this initial step subsequently fed back into the same OLM with Chain-of-Thouus (CoTT) prompting obtain the final answer. Consequently, the model's performance reflects two key abilities: OCR/Transcription and Text Reasoning."
Video/Audio/Imaage We first process the video to 1-fps to meet the requirements for both the Gemini and GPT models. For testing with Claude, we used the API available before August 10th, which only supported maximum of image inputs, video evaluations GPT-4o API supports 250 images input at the maximum, so any additional frames were dropped in the evaluation. In contrast, Gemini had issues with the video modality and could handle all frames as input. Image processing is the modality that all models support most effectively, allowing comprehensive testing across all OLMs. Notably, Gemini the only model supporting audio input.
conducted. The
Answer Extraction: We use the model to extract the answers. Since the regex parsing may affect the performance, we sacrifice the API cost to trade in the excellent extraction.
x Table I: Results on OmnixRsvvtH show different mixed modalities evaluations, including text, image, audio, video. Each modality (Image/Audio/Video) combines two input sources: the 'Ques- tion' provided by the respective image, audio, or video modality, and the 'CoT instruction' provided by the text The numbers in red font, following the downward arrows, shows the drops compared to the pure text input.
Gemini 1.5
Claude
GPT
Pro
Flash
Opus Perf.
Sonnet
Haiku A
4o
4o-mini Perf. A
Modality
Perf.
A
Perf.
A
A
Perf.
Perf.
Perf.
A
Text
77.5
69.9
77.7
77.4
72.5
71.5
72.6
Image
57.3 20.2], 36.3 33.6].
26.9 50.8
18.8 58.61.
9.9 62.64
60.1
11.44
48.5 24.1)
Audio
56.6 20.9]
53.9 16.0j
Video
36.3
41.2), 15.1 54.81
53.1
18.4]
18.6 54.00
Extract-Then-Ansee
(ETA) Prompting
Image
73.5 4.0.) 68.1
1.81
62.6 15.11 48.1
29.31
43.2
29.3
66.7
4.8L 58.4
14.2/
Audio
69.9
7.6L
63.6
6.3.)
Video
48.6
28.91 42.8
27.11
25.0
46.5 59.3 13.3
3.2 MAIN RESULTS ON OMNIX
We show the main experimental results ominified MMLU-Pro in Table 1.
5

---

Published as a conference paper at ICLR 2025
Table 2: Results on Omni RREAl shows similar behaviour discrapancy of OLMs as indicated in results the AnnixXRysTTr Interestingly, we also observe that simple prompting strategy (ETA prompting) is not as effective as it was on Omnix RANNTH possibly due to the natural noise and redundancy in real-world image, video, and audio data.
Gemini 1,5
Claude
GPT
Pro
Flash Perf.
Opus
Sonnet Perf. A
Haiku
4o
4o-mini
Modality
Perf.
A
A
Perf.
A
Perf. A
Perf.
A
Perf. A
Text
86
80
78
66
65
85
75
Image
78
81
65
151
41
344
39
274 33
84
79
64
63
12]
Audio
71
154
64
144
Vidco
22
53
274
73
12]
66
94
Extract-Then-Answee (ETA) Prompting
Image
79
74
65
15|
63
154
52
144
51
144
79
6.
70
54
Audio
314
291
Video
71
154
73
74
66
194
63
124
Model Comparison. Gemini-1.5-PP demonstrates the most versatile performance across all modalities, showing results in text, image, audio, video tasks. Claude models struggle with image tasks and lack audio and video capabilities. GPT models show a balanced performance, with GPT-4o performing particularly well direct image and video compare to Gemini and Claude. Generally, larger models outperform their smaller counterparts across modalities, e.g., Pro >> Flash, Opus > Haiku). But interestingly, EPT-4o-mini outperforms GPT-4o in text and video with ETA prompting. For video tasks using ETA prompting, GPT-4o's performance inconsistencies led us to examine the model's 's responses to the extraction, we found that over 46.8% test samples, the detailed analy- sis can be found in Appendix H, GPT-series models cannot extract the text from video, which identify as the primary cause for the significant performance drop compared to CoT prompting. Re- garding the text modality, two possible explanations emerge: first, MMLU-Pro GPT-4o- mini, suggesting that OAI might have optimized for it. Second, since our dataset uses a subset sampled from MMLU-Pro inherent biases may have influenced the results.
released before
Modality Analysis. Text is the most mature modality across all models, with consistently high scores (ranging from 69.9% 77.7%). Image modality shows significant variability, with direct task performance ranging from 9.9% (Claude Haiku) to 60.1% (GPT-4o). However, ETA prompt- ing on image generally improves performance for all models, particularly for Claude (e.g.. Opus improves from 18.8% to 62.6%) The improvement justifies the inclusion of ETA prompting as standard in our benchmark to prevent potential manipulation. Audio modality, only available for Gemini models, shows moderate performance with notable improvement via ETA prompting, Video modality presents the most challenges, especially for the small models, i.e., Semini--15--Fie and GPT-4o-mini.
There are also additional results on Arc-Challenge and GSMSk benchmarks shown in Table with different modality input, i.e., text, image, audio, video. Though the models are likely to be data contaminated on these benchmarks, the performance drops are still significant on image/video/audee compared to the pure text.
3.3 MAIN RESULTS ON OMNIX
The results on the realistic set generally align with those from the synthetic set, showing significant drops in performance across audio, image, and video tasks compared to the text. One difference here is that performance on video does not drop a large margin compared to that in the synthetic set. Though the video is noisy than it is in the synthetic data, we can still capture one key frame and answer the question according to that key frame which largely reduces the difficulties, compared to the synthetic scenario, the model can find the main frame in the video. Another interesting finding is that ETA prompting does not consistently improve performance; for example, there are performance drops in audio tasks with ETA prompting compared to CoT on both Gemini- Femini--Flash and Gemini-Pro These findings confirm that our synthetic effectively simulates real-world scenar- ios scalable, cost-efficient way, serving as a valuable sanity check for OLMs' omni mani-modality reasoning capabilities.

---

Published as a conference paper at ICLR 2025
Key Takeaways. We summarize the following interesting takeaways from experiments:
1. Multi-modal capabilities vary significantly across models, with Gemini 1.5 Pro showing the most broad support and balanced performance across all modalities.
Gaps still exists on other modalities compared to the text modality even just in such easy per- ception test scenarios. Significant room for improvement exists in video processing across all models, presenting opportunities for future development.
ETA prompting generally improves performance on Omni but OLMs can longer solely rely on it for Omni Reeal, indicating the necessity of the further alignment on omni- modality.
4. There's 's clear trade-off between model size and performance, but smaller models (e.g.. GPT-4o- mini) can sometimes outperform larger counterparts specific tasks.
5. Our Omni x could be good simulating set for the real-world scenarios, as the results on ORni match the results the Omni x RSYNT--
4 MIXED MODALITIES
Table 3: The results of more complex mixed modalities on OmnixXssvr We use the A to denote the performance drops from the text modality.
Input Modality
Gemini-Pre
Gemini-
Questicn
CoT Prompt
Perf.
A
Perf.
Text
Text
77.5
69.9
Text
Video
76.1
1.44
66.8
3.14
Text
Audio
74.1
3.44
68.3
1.6
Text
Image
74.1
3.44
66.9
3.04
Image Audio
Text
61.8
15.74
49.1
20.84
Vidco + Audio
Text
40.1
37.44
25.9
44.01
Text to Mixed Modalities. In addition to the types of the Omniffy described in Section 2.1, method could also be applied to generating interleaved modalities to better simulate more complex real-world scenarios, where the information is included in different modalities and requires model to reason across the modalities to solve a problem. For example, an instructor can write down an equation on the blackboard and say "compute the derivative" in Calculus lecture. Scenarios like this example require a model to jointly use image perception and audio understanding process the question, reason across the visual and audio modalities, and then provide response. Using our Omnify!! Ey!, ,, we seamlessly integrate different modalities and create test samples with interleaved modalities, i.e., "Video + Audio". and "Image + Audio", Omni XR which captures a more authentic user experience where multiple senses are engaged simultaneously. To be specific, We transfer the question into video and all the options are transferred for Audio, to get the modality, "Video Audio", while CoT prompting remains in text form to maintain the model's reasoning ability across different modalities.
Transferring CoT prompt to other modalities. All the CoT' prompting is in text for all the previous test cases. Here, we convert the CoT prompt into different modalities while keeping the others, i.e. questions and options in MMLU-Pro intact.
Results. As shown in Table 3, there noticeable decline in performance when transitioning from text to mixed-modality tasks. For example, both the Pro and Flash models perform significantly worse the "Video + Audio" scenario, achieving scores of 40. and 25.9, respectively. This in- dicates that handling mixed modalities presents a significant challenge, likely due to the increased complexity of integrating video and audio information. For Audio//mage/ /mage/Video CoT, the model generally treats these inputs noise irrelevant context, having minimal impact the final sults, as performance approaches that observed with text-based COT. We focus on evaluating the Gemini-series models since only Gemini supports audio inputs.

---

Published as a conference paper at ICLR 2025
5 ANALYSIS
5.1
OMNI-MODALITY REASONING BEHAVIOUR ANALYSIS
After investigating the responses, we find that in omni-modality cases, Gemini-1.5-Flast models can only output very short answers though prompted to CoT before giving the answers, which is quite different from the reasoning behaviour in the pure-text. An example in Figure shows the different behaviours among modalities, which intrigues us to have quantitative analysis of the reasoning paths. We write simple regex, detecting if the model output starts with "the answer/response (*.)", with the rule, the total number of words should be less than 40, to evaluating whether the models' output contain the reasoning path. The results are shown Table
Table 4: The percentage of the model outputs containing the reasoning paths on OmnixRsyNTH RSYNTU.
Path(S)
Gemini 1.5
Claude
GPT
Modality
Pro
Flash
Sonnet
Opus
Haiku
40
4o-mini
Text
98.9
89.1
100
100
98.6
100
100
Image
93.2
54.3
100
100
72.8
100
100
Video
91.3
23.4
99.1
95.7
Audio
94.0
82.3
Our analysis reveals that smaller models tend to produce reasoning paths less frequently for image, video, and audio inputs. Notably, for complex modalities like video, Gemini-1.5-Flash generates reasoning paths for only 23.4% of test examples, substantially lower than Gemini-1.5-PPo Among the modalities, audio inputs elicit reasoning paths most similarly text, while video inputs show the lowest rate of reasoning path generation. GPT-series models demonstrate excellent performance in producing reasoning paths across available modalities. However, these results underscore the signif- icant challenges remaining in cross- modal reasoning. Given that models are expected to exhibit rea- soning abilities, they should ideally output reasoning paths consistently across input modalities.
5.2
VISUAL/VIDEO FORMATS INFLUENCES PERCEPTION PRECISION
5.2. IMAGE
We first analyze how formats affect the performance on images. We show images with two different text formats in Figure 5. The lower image has a compact format, where the options are not spaced out; instead, they are presented in continuous, inline format separated by periods. Compared to it, each option in the upper image is listed separately, making it easy to read, with letters (A to J) clearly aligned before each option. The results of CoT and ETA prompting with two different formats of images are shown in Table 6. The overall trend here is that with better format, we could significantly improve the performance across all the tested models. ETA prompting also boosts the performance for the both formats in general. For all the other models, the performance can be significantly improved when comparing BF with ETA, only the GPT-4o being an outlier.
We further analyze transcription accuracy using the Character Eror Rate (CER), standard metric for assessing text recognition performance, especially in OCR tasks. A CER of indicates perfect accuracy, with higher values reflecting more errors. Details of the CER calculation Appendix J, and results are shown Table The results reveal that GPT-4o's OCR performance is largely inmeat-iddeeendent, whereas other models exhibit considerable format sensitivity, explain- ing the pronounced improvements seen with ETA prompting for all models except GPT-44 when format is enhanced.
provided in
5.2.2 VIDEO
We create different types of videos, one word per frame, several words per frame, etc. Our ablations reveal that increasing the number of words per frame generally leads to improved performance for both Gemini- Flash and Gemini-Pro models under both testing promptings. CoT and ETA prompting. This trend suggests that providing more context within each frame aids in the models' understanding and processing of the video content and narrow the gaps between images and videos.
8

---

Published as a conference paper at ICLR 2025
Table 5: The Character Error Rate, the metric for evaluating the OCR, of different models on two different formats images.
Gemini 1.5
Claude
GPT
Pro
Flash
Opus
Sonnet
Haiku
40
4o-mini
Image
0.11
0.10
0.19
0.28
0.34
0.11
0.12
Better Image
0.06
0.03
0.05
0.18
0.26
0.11
0.11
Questiio: The relurives of a grop of pelicers from the some specisss that separated anrd from each other becoueee of urouctreedfll migration are reuniied 150 yours later and find that they ana unable to produce oflapring. This i an snampla
Oprions: A. gens flow
9. temporal ispuation.
C. disrupthve setection.
founder effect. genctio drit.
F seauel selection. sympatii apeciaaioo H. habitar togmemtttoon. bottleneck effect. allopattio
Question: relatives group of pelicans species tomporal disruptive selection. H.hobitor frogmenztiinn bottonock offect. allopettic speciction
unsuccesshil sympotri speciation.
they gonetie
Figure 5: We include two figures to illustrate which is better format image. The upper one is the image with better format. The lower one the image with the original format.
RELATED WORK
Large Foundational Models. GPT-4o (OpenAI, 2024b), Gemini (Gemini-Team, 2024a) both claim their models having omni-modality capabilities, but actually OAI's model does not support audio(no audio access via APls)/videooonly 250 frames and the videos should be separated manually be- fore feeding into the model) while Gemini can take very long videos and has good Audio support. Claude (Anthropic, 2024) can be viewed as a vision- lision-language model (Bordes et al., 2024) since it has capabilites take image but no audio or video support. There are also other open-sourced vi- sion language models, but they are mostly supporting only two modalities, e.g., the vision-languag,
Flash CoT
Flash ETA
Pro CoT
ETA
4
58.7
50.3
50.3
68.22
Performance (%)
60
48.0
46.3
50
42.8
36 3
40
31.1
30
17.5
23.9
15.1
20
10
Words Per Frame
Figure 6: Video ablation study: Model performance with different words per frame. Pro and Flash denotes Demmini-1.5-Pro-000 and Gemini--15--Ai F.5-hlas-0000, respectively.

---

Published as a conference paper at ICLR 2025
Table 6: The ablations: image with better format. BF: better format. The blue font denotes the performance gain of the better image compared the original image format.
Gemini 1.5
Claude
GPT
Prompt
Pro
Flash
Opus
Sonnet
Haiku
4o
4o-mini
Text
Coll'
77.5
69.9
77.7
77.4
76.5
71.5
72.6
Image
CoT
57.3
36.3
26.9
18.8
9.9
60.1
48.5
Better Image
CoT
64.6 7.31
43.6 7.3
33.5
6.6t
28.9
10.1
19.1
9.21
65.5 5,4
52.1 3.6
Image
ETA
68.7
61.3
36.4
26.6
24.9
66.7
58.4
Better Image
ETA
73.5 4.8f
68.1
6.8t
62.6
26.21
48.1
21.5t
43.2
18.31
66.9
0.2
61.7 3.3
models like LLaMA-3.1 and 3.2 (Meta, 2024), Pixtral (Mistral, 2024), LLaVA (Liu al., 2023b;a); Audio-LLM like GAMA (Ghosh et al., 2024), LTU (Gong et al., 2023b;a), and SALMONN (Tang al., 2024). It hard to judge them on our benchmark, since the main idea behind our evaluations are that we expect the model has cross-modalit, reasoning and would like to encourage the model improving their cross-modal reasoning, only vision/audio/vides would not get a comprehensive re- sults. We would expect the open- community to release real OLMs in the future and we will update the results accordingly.
Video/Audio/Imagg Evaluation benchmarks. Omnibench (Li et al., 2024b) specifically aimed at evaluating OLMs' tri-modal, i.e., text, vision, and audio, processing capabilities with human- annotated tasks. Compared to it, OmnixR emphasizes the omni-modality reasoning evaluations with both human-annotated realistic set and scalable synthetic set. MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b). CMMMU (Ge et al., 2024), and IsoBench (Fu et al., 2024) focus on evaluating vision-language models across various college-level disciplines with highly hetero- geneous image types, emphasizing expert-level perception and reasoning across text-image pairs while LMSYS- Vision (Ims, 2024) evaluates the instruction-followii of the large vision-language models (Liu et al., 2023a; Chen et al., 2023; 2024; Yang al., 2024a). Compared them, OmnixR has larger scope on evaluating OLMs on cross cross-mo reasoning, not only vision input, but audio, video, and mixed modalities such as image audio. AiShell- AiShell-2 (Du et al., 2018), Clotho- AQA (Lipping et al., 2022) are audio understanding benchmarks, providing extensive and high- quality real-world audio data for Mandarin ASR and audio question answering. MVBench (Li al., 2024a) focuses on temporal reasoning across 20 challenging video tasks, Video-Bench (Ning et al., 2023) assesses Video-LLMs across video-exclusive, knowledge-based and decision-making tasks, while MMBeMcM Video (Fang et al., 2024) offers long-form, multi-shot evaluation of LVLMs with 609 videos and 2,000
QA pairs across 26 fine- aeeeggraied capabilities. In OmnixR, we also include long video in both synthetic and realistic scenarios and we also have mixed modalitty evals including video audio.
CONCLUSION
In this paper, we introduced Omnify!, a scalable and cost-efficient approach for generating multi- modal data from text, facilitating the construction of diverse and challenging test scenarios for omni-modal language models (OLMs). Using this method, we developed Omnix a syn- thetic omni-modal reasoning evaluation dataset derived from MMLU-Pro. as well as OmmixXRRaa a real-world omni-modal reasoning dataset based on YouTube content. Our comprehensive evalu- ations reveal that OLMs experience substantial performance drops when confronted with complex multi-modal inputs, particularly in tasks that demand cross-modality reasoning. Notably, we ob- served that smaller models, e.g., Gemini-1.5-Flase are less adept at producing reasoning paths for image, video, and audio inputs compared to text, underscoring the inherent challenges in cross- modal reasoning. The evaluation results underscore the necessity for enhanced training strategies to address the complexities of omni-modal tasks. To sum up, OmnixR stands as critical benchmark for guiding future advancements in OLMs, providing a foundation for measuring progress toward more human-aligned and truly omni-modal Al systems.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENT
We thank Quoc V. Le and Xinyun Chen for the discussions on the CoT reasoning and how LLM reasons with irrevelant context. We thank Prajit Ramachandran, Yiling Huang, Yichong Xu for insights and future works on synthetic audio evals/benchmarks. We thank Micheal Chang, Yao Fu on discussions of the real-world applications of OmnixR. We thank Jean Sann-Bapisst Alayrac, Fangyu Liu on Helpful discussions for video/image evals and possibility to include the data into pretrain corpus. We thank Yong Cheng, Ruibo Liu, on helpful discussion on synthetic data vs. real-world data, and how to reduce data contamination on evaluations.
Lichang Chen and Heng Huang were partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617.
REFERENCES
Multimodal arena. 2024. URL
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: http//pppnneeeeeeeeeeeeee net / forum? id-RzYrn625bus.
Anthropic. Claude: An ai assistant by anthropic, 2024. URL.ttts://wwweenhrroree com/
claude. Accessed: 2024-09-21.
Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, et al. Imagen 3. arXiv preprint arXiv:2408.07000 2024.
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, al. An introduction to vision-language modeling. arXiv preprint 14t.::27....2477 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL
lideo--eneraaioon meseerattiinmmmmleseessse models -steeesssss-woooos as-world simulators.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.1422 14238, 2023.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404 16821, 2024.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wengi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama Advancing spatial-temporal modeling and audio understanding in video-lims, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 0803.800350555 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. 14168, 2021.
Jiayu Du Xingyu Na, Xuechen Liu, and Hui Bu. Aishell-2: Transforming mandarin asr research into industrial scale, 2018.
11

---

Published as a conference paper at ICLR 2025
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406. 14515, 2024.
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomor- phie representations, 2024.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large- scale adversarial training for dision-and-lanuuee representation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 6616-6628. Curran Associates, Inc., 2020. URL. https: http::///ooceeeeeseeeees neu... neurips cc wwcppaper_fiie www.ffllelppe/////// Pacsss-apperppp..
Peng Gao, Renrui Zhang. Chris Liu, Longtian Qiu, Siyuan Huang. Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang. Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ArXiv, abs/2402.05935. 2024. URL https: //api. semant i cacholar org/ CorppsstD : 267547619
Zhang Ge, Du Xinrun, Chen Bei, Liang Yiming, Luo Tongxu, Zheng Tianyu, Zhu Kang, Cheng Yuyang. Xu Chunpu, Guo Shuyue. Zhang Haoran, Qu Xingwei, Wang Junjie, Yuan Ruibin, Li Yizhi, Wang Zekun, Liu Yudong, Tsai Yu-Hsuan, Zhang Fengji. Lin Chenghua. Huang Wen- hao, and Fu Jie. Cmmmu: A chinese massive multi-discipline multimodal understanding bench- mark. arXiv preprint arXiv:2401 20844, 2024.
Gemini-Team. Gemini 1.5: Unlocking multimodal understanding across millions of
tokens of context, 2024a. URL https: httpss://bboo..googe///eeeeee gcogle-gemini--ee www.-eeeeeeeiioooonooooeee wtw-mmmeeeeffuurr----a-m------- wtwggoop/eeeecoollggg.aaatt
Gemini-Team. Gemini: A family of highly capable multimodal models, 2024b. URL https:
//arxxv.org///////./tt//o///tl/t httgg/ab/223333 11805.
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities, 2024. URL https:// arxiv. org. abs 2406. 11768.
Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, December 2023a. doi: 10.1 24110999smuS79944333.000999 http://dx doi. org 10. 100.01199AAAUUS /AA//SSSRS59669 2023 10389742.
URL
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305.10700 2023b.
Google. Veo: Google's most capable generative video model. 2024. URL /deepp::///eeemminn google/ @ooogle/tan eetccno og es /veo/.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), volume abs/2204. 14198, pp. 22195-22200 IEEE, June 2024a. doi: 10. 109/cvp552733. 2024.02095. URL http::/ dx. doi .rr.or////// 100 1tt.:11...111/11 1119/110P222233..... 2024 02095.
Yizhi Li, Ge Zhang. Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang. Zhenzhu Yang. Xiangzhou Wang, Zhaoxiang Zhang. Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. Omnibench: Towards the future of universal omni-language models, 2024b.
Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-
aqa: A crowdsourced dataset for audio question answering, 2022.
12

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks, 2019. URL. btttpp:://rrxvv.rrggg
1.988 02265.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations (ICLR), 2024.
Meta. The llama herd of models, 2024. URL UtL.tttsss///xxx...or///////////
Mistral. Pixtral, 2024. URL /tt.pptpse//ooc...ittaaaaaaaaaaa
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint urXiv:2311.16103, 2023.
OpenAI. Dalle 3. https://opeaaa.co//nneee/aallll
2024a.
OpenAl. Hello gpt-4o! OpenAl Research, 2024b. URL https://ppeeaiccoo/// hello-gpt---eeee
Accessed: 2024-09-21.
OpenAl. Gpt-4o mini: advancing cost-efficient intelligence, 2024c. URL https::///pp/ itt.rioooooooo.iiiiiiiiieieeeeee
com/ com/index i/wiinn//////// x gpppgpptpp 1ao-p-4o-miin --t---o---ii-eeee
Ac-
cessed: 2024-09-21.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / /poenrevveww www.ee////feooo forum? 1=114rd
Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' A'tummee Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models, 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Ii, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li. Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei. et al. Qwen2. technical report. arXiv preprint 1515 2024b.
13

---

Published as a conference paper at ICLR 2025
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning Computer Vision and Pattern Recognition (CVPR), volume 32, pp. 9556-9567. IEEE, June 2024a. doi: 10. 10.1 2024.00913. URL http: dx. dei. orgioorg/100 10. 1w1.11...111111 10t.001111////222220
benchmark for expert agi. In 2024 IEEE/CVF Conference
2024. 00913.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark, 2024b.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjac Yu, Jae Sung Park. Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models, 2021.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wa Wei Chang, Peng Gao, et al. Mathverse: Does your multi multi-modal Ilm truly see the diagrams visual math problems? arXiv preprint arXiv:2403.14624 2024.
14

---

Published as a conference paper at ICLR 2025
A AUTHOR CONTRIBUTIONS
Lichang devotes revising the idea and constructing the O finishing the code writing and most of the experiments.
Hexiang proposed the initial idea for Omni-Eval via the observations on image reasoning behavior inconsistency of Gemini-Flash models.
Hexiang. Boqing, Yiwen, Zifeng, Mingda, and Yandong contributed to the Omni x RREAL-
Hexiang, Mingda, Yandong, Boqing, and Tianyi attend the discussion regularly and pro- vide useful feedback/suggestion for the project.
Pranav contributes to the ablation study: images with better format and video ablations.
Heng provided the university tuition support Lichang.
Ming-Hsuan and Boging are project leads @ @Google Deepmind.
B CONVERT MATH INTO SPOKEN VERSION
For the math equations in the questions, we prompt Semmni--15--Pp to convert them into the version which can be spoken orally. The prompt we used is detailed in Table 7. We also show an example to explain the transformation: the TTS is hard to read the original question in Table 8 but it can handle the converted text.
Table 7: The oral conversion prompt designed for Text-to-Audio transfer.
[Prompt] Please transform all the equations in the text into the format that is easy to speak out orally. [Original text] Please first output single line of the text in the format "The transformed text is xxx"
Table 8: An example of the conversion from the original question into the easily spoken text.
[Original Question] For what values of is it true that x:² - 5x < 10? Express your answer in interval notation.
[Converted Text] The spoken version: For what values of is squared minus five x minus four less than or equal to ten? express your answer in interval notation.
C RESULTS ON OPEN-SOURCED MODELS
We expanded our evaluation to include additional OLMs: Reka Flash (Team et al., 2024), Qwen- 7B (Yang al., 2024b), Qwen-72B (Yang et al., 2024a) and VideoLLaMA-2 (Cheng al., 2024) on both the synthetic set (OmnixR-Synth -Sythh) and the realistic set (OmnixR-Real) The results are shown in table and table 10. The results of the three newly evaluated models align well with the findings in the paper:
1. Gaps in Modalities Beyond Text:Text modality consistently outperforms others across all models in both OmnixR-Synth and OmnixR-Real. For instance, on OmnixR- Reka- Flash achieves 62.5 in text but only 9.4 image, 6.6 in video, and 16.3 in audio.
2. Significant Room for Improvement Video: Video performance remains low across both synthetic and realistic datasets. For example, in OmnixR-Symi -SSthh the highest video score is 11.0 (Qwen-72B), and in OmnixR-Rest
only 0.27.
D CATEGORIES IN MMLU-PRO
There are 14 categories in MMLU-Pro, including Math, Physics, Chemistry, Law, Engineering, Other, Economies, Health, History, Psychology, Business, Biology, Philosophy, Computer Science.
15

---

Published as a conference paper at ICLR 2025
Table 9: Open-soureed models on OmnixR-Synth. We benchmark open-sourced models on our synthetic subset.
Modality
Reka-Flash
Qwen 7B
Qwen 72B
VideoLLaMA2
Text
62.5
46.5
70.1
45.2
Image
9.4
38.8
63.5
4.1
Video
6.6
7.2
11.0
3.9
Audio
16.3
Table 10: Open-sourced models on OmnixR-Real, We benchmark open-sourced models on our
realistic subset.
Modality
Reka-Flash
Qwen 7B
Qwen 72B
VideoLLaMA2
Text
0.66
0.58
0.79
0.52
Image
0.30
0.47
0.52
0.19
Video
0.19
0.22
0.27
0.15
Audio
0.23
E MODEL SETTINGS/DETAILS
The version of the Geminis used this paper are Gemini-I.5-Pro-OU0 and Gemini-1.5-Flas 001. The version of the OpenAl models we used are gpt-4o-2024005-13 and gpt-4o-mini-2024
07-18. The verison of the Claude models used are opus @ 20240229, claude-3-haiku 20240307.
20240229, claude-3-
The Gemini safety settings we used for video, audio, images are shown the following:
Safety Setting
generative_modells SafetySetting
1ARM_CATEGORY
rach www.rhhreeeeooo....o.oooo. H_IGGH
)
categozy=generatiive mwdels..doeeooeooooooooooo
Saroeels..ammetttgrrry.BB.B HARM CATEGORY HARASSMENT,
ttrreshold-generatiiee..co modelss htt.e...aaalllcccheeeooo........ CNLY_HIGH,
11
generative sener
15
HARM BEOP__CAAEGGOY_SEXAALL2
SLLL HIGH,
BLOCK_ONIA HIGH the loosest setting we can use public Gemini APIs video, audio, and images. BLOCK_ONDE NONE is the loosest setting use for text, change all the Safety Settings for language into BLOCK.ONLY_NONE..
For response generation, follow the commonly used settings, temperature=0.7, top-p=0.9, and output.leggth=1024 for all the models, i.e., Gemini, Claude, GPT models.
F RESULTS ON
& GSM8K
We also evaluate Gemini models on ARC-Challenge dataset and GSM8K test set. The results are shown in Table 11.
16

---

Published as a conference paper at ICLR 2025
Table 11: Performance of Gemini Models Across Different Modalities on ARC-Challenge and GSM8K Benchmarks
Accuracy (%)
Benchmark
Gemini-1.5-Pro Semmnn--1.5-Flan
ARC-Challenge
Text
95.5
92.3
Image
79.5
75.0
Audio
91.1
88.0
Video
63.6
40.3
GSM8K
Text
99.1
96.3
Image
92.5
87.9
Audio
86.8
90.8
Vidco
80.3
63.1
G OMNIXR STATISTICS
We show the statistics of Omni x RReAi in table 12.
Table 12: Statistics for Video and Audio on the Omni RAYNTH. Frames, s: seconds.
Min
Max
Mean
Video
28F
552F
117.2F
Audio
7.2s
251.3s
32.3s
Table 13: Statistics for Video and Audio the Omnix RREAL F; Frames, seconds.
Min
Max
Mean
Video
30f
1326f 1326s
255.6f
Audio
10s
139.7s
H ANALYZE THE EXTRACTION
We manually check the data first, and then find the patterns that the extraction failure have mostly "unable to process", "can't extract", "I'm sorry", and "unable to extract". So we use these four patterns to check if the answers contain one of them, and calculate the percentage of the model answers which do not output the extractions when prompted as "Please extract the text from video."
17

---

"Au2225": r"S\paralle1s", Published as a conference paper at ICLR 2025
I DETAILS OF THE TEXI-TO-IMAGE CONVERSION
We use the Python Imaging Library (PIL) to create a new image with a white background and the text drawn onto the image with black color. The tricky part here that the most commonly used font "times.ttf" does not support the Unicode well and will encounter the error when we try to convert the Unicode text, e-g., special mathematical symbols such oc, 2, II, A. Thus, our solution here is to have look-up-tabel to replace these Unicode text with latex code before generating. The details about the look-up-table shown Appendix I.1.
1.1 LOOK-UP-TABLE FOR UNICODE CONVERSION
We show parts of look-up-table here due to the display issues. The full details about the look-up- table could be referred to our code.
\u03b1'; S'alphas',
Alpha
Au03622 r Abbeaa''
Beta
Au03b3'' : r/ Agaammas
Gamma
Au03b44' r AAdeltas,
Delta
Au03300' $\pis',
Pi
\u03e3' r' $\sigmas',
Sigma
Au03c6'; r' A'phig',
Phi
\u03c9' r' Nomegas',
Omega
\u2211' r' $Asums'
Summation
\u220f' r' Aprodss'
Product
11
Au222b' r' Sints'
Integral
12
\u0394' r' ADeltta''
Capital Delta
\u03a3' r' SSiimma''
Capital Sigma
\u03a6' r' $APhiS',
Capital Phi
\u03a9' r' V\mmeaa',
Capital Omega # Left arrow em dash
16
Au2190" r' #\lettarrow"'
Au2014": r"-",
\u03a0" r"SAPis", \u0302" r"$"""
# Capital Pi circumflex accent En dash En dash
20
\u2228"; r"Slvees",
21
"A22013: r"--",
"u2026": r"S\idotss",
# Ellipsis
"u22283" r"$Asupsets"
Superset
24
"Au201c"; r"!""
Left double quotation mark
25
"\u03c1"; r"S\rhos", "\u03b8": r"SAthetas",
Rho
36
Theta
"\u03bb": r"$\lambdas",
Lambda
Parallel
29
"\u2022": 1"SVioullets", Bullet
J
CER CALCULATION
The CER is calculated by
CER S+I+D
Where S = Number substitutions, / Number insertion, D= Number of deletions, Total number of characters in the ground truth text. Since the model output will contain some addition text, e.g. starting with "The text in the image reads:", which will affect final results, we will detect the "question:" by regex to climinate the influence by that irrelevant text.
18

---

Published as a conference paper at ICLR 2025
Table 14: Examples of different reasoning behaviour across modalities on GPT-4o. We use red font mark the key errors in the reasoning path. The correct answer here is F. GPT-4o only respond correctly on the text modality input.
[Question] Question: Find 4(3/4) % of $24.85. Options: A.$1.50 B.S1.10 C.$0.99 D.$1.25 E.$1.35 F$$1.18 G.$1.00 H.S0.93 I.$1.20 J.$1.05"
[Prompt] Think step by step then output the answer in the format of "The answer is (X)" at the end.
[Response(to Text)] To find 4 (3/4) % of $24.85, we will follow these steps:
Convert the mixed number (3/4) into an improper fraction: (3/4) 4 (3/4) = (4 * 4/4)
+ (3/4) (16/4) + (3/4) 19/4.
2. Convert the fraction to a percentage: - (19/4) % (19/4)* * (I/100) 19/400.
3. Now, calculate 19/400 of $24.85: First, find the value of $24.85 (19/400): $24.85
(19/400) = $24.85 0.0475 (since 19/400 0.0475).
4. Perform the multiplication: $24.85 0.0475 = S1.179875.
Round to two decimal places: $1. 179875 rounds to $1.18.
Now, check the options provided: A. $1.50 B. $1.10 C. $0.99 D.. $1.25 $1.35 $1.18 G. $1.00 H. $0.93 I. $1.20 J. $1.05 The closest match is F. $1.18. Therefore, the answer is F. [Response(to Video)] 1. The question asking to find 4/3 of $24.85.
2. First, understand the mathematical expression: (4/3) $24.85.
Calculate the multiplication: Divide $24.85 by 3= $24.85 3 S8.2833 (approx) Multiply the result by 4 $8.2833.333 4 $33.1333 (approx)
4. The closest option to $33.13 13 among the given options $33.13. The answer is S33.13.
[Response(to Image)] To find 3/4 of 24.85, follow these steps:
Calculate of 24.85: $ x 24.85 = 3x21.85
3 X 24.85 74.55, Now divide by 4: 71.55 4
18.6375, Round the nearest cent: This gives us approximately 18.64. However, since we need to find 4 of that value, we can directly multiply: 4
24.85 18.6375, None of these options seem match the calculated value directly. The final answer considering the options is "The answer is not found among the options provided.'
19


Reviews:
[{"review_id": "GtkdyPxzCz", "reviewer": "Reviewer_89Zj", "summary": "This paper proposes an evaluation suite for multi-modal language models - Omni×R, which can integrate various modalities such as text, vision, and audio. The design of Omni×R aims to assess the models' capabilities in cross-modal reasoning, especially when the user's messages contain multiple modalities, the models need to be able to understand and reason comprehensively to complete the task. The paper introduces two evaluation variants: synthetic data Omni×RSYNTH and real-world data Omni×RREAL. These evaluations reveal the existing problems of the state-of-the-art models.", "questions": "- Why is the performance of ETA Prompting also poor in Table 2, especially on the audio modality where it performs worse than the normal evaluation?\n- I would like to know if the authors have cleaned the constructed data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of this paper is very clear, the problem proposed is valid, and the benchmark proposed is distinctly different from others.\n- The evaluation experiments in this paper are comprehensive, exploring the experimental results under different modality combinations and different prompt scenarios, and providing a detailed analysis.\n- The pipeline of data construction is inspiring, providing a possible solution for the subsequent generation of cross-modal datasets.", "weaknesses": "- The synthesis method for audio data is common, but the direct rendering approach for images and videos seems overly simplistic and overly reliant on the model's OCR capabilities.\n- While the paper points out the challenges models face in cross-modal tasks, it does not provide specific model optimization strategies or directions for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpNwkkoDAU", "reviewer": "Reviewer_5iEv", "summary": "Authors propose the Omni x R benchmark intended to evaluate Omni LLM, presenting three modalities: text, vision and audio. Two variants are proposed: Omni x R_synth, which is synthetic built, and Omni x R_real that has been human curated. The sources for those subsets are respectively MMLU-Pro and Youtube. \n\nAlong those benchmarks, the authors introduce the Omnify method, which offers to create synthetic omni-modality evaluation data by 'translating' across modalities data sample (e.g. video to image and audio, or audio and text, etc).", "questions": "- What is the license for your benchmarks? It is probably safe to assume that MMLU-Pro can be used freely by the community. However, scraping YouTube videos might be restricting the use of Omni x R. What is the license? Does it allow Commercial use, if say another company wants to use that benchmark to evaluate their internal OLM?  \n- L204: Frame merging is mentioned. How realistic is that? Can you show a not-cherry-picked example?\n- Is there are reason why the evaluation was limited to closed-source models (Gemini, Claud and GPT-4o)? Why not evaluating the rest of the omni-LLM models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written. Data sources, models and experiments are clearly stated with examples. Takeaways are given in page 7, that help the reader understanding the scope of the contributions. \n- Good point on the authors to spot the data contamination with Gemini on GSM8K.", "weaknesses": "- One big weakness of this work seems that the whole paper doesn't show examples of Omni x R_synth or Omni x R_real. It is unclear for the readers how exactly those benchmark look like in reality? One could assume that Omni x R_synth is probably a MCQ-type of benchmark, but this isn't stated in the paper. What Omni x R_real is looking in practice is even less clear in those settings. As those two benchmarks are the main contribution of this work, it is a bit unclear why the authors have not shown them in first place. Arguably Figure 1 should be not-cherry-picked examples of Omni x R_synth and Omni x R_real. \n- The current Figure 1 is not very informative. The source of that example isn't mentioned. The terminology \"OmniLM\" in the figure seems to conflict with the rest of the paper (\"omni-LM\"?). The figure mentions \"Gemini or GPT-4o\". The 'OR' alone is misleading. There are actually many omni-LLM in the community, starting with Reka-Core, UnifiedIO2, AnyGPT, MIO-Instruct, and many others. They all seem to be ignored by this paper. Finally, the legend mentions using 'Gemini'. Arguably, Figure 1 should be removed altogether and replaced with actual examples from Omni x R_synth and Omni x R_real.    \n- The omnify method doesn't seem to be evaluated. \n- Terminology: The paper mixes freely \"omnimodal\" and \"omni-modal\". Please pick one and keep the terminology consistent. \n- Terminology: Figure 2 uses \"OmniXR_{real}\" while the rest of the paper uses \"Omni x R_{real}\". Why not using the same consistent terminology?\n- Nitpicks: L092 s/Youtube/YouTube/, L126 s/Youtube/YouTube/, L192 s/youtube/YouTube/. The only correct is L529. \n- Figure 2: The color code is misleading. Blue seems to be 'Chemistry' in the pie chart. The YouTube source is blue too. However, in L197 it is mentioned that 4 categories are used: Math, Coding, Physics and Chemistry. Same remark for the string '14 Categories' with a blue box above. This figure 2 seems completely inaccurate. Use color code when needed and accurately. \n- As an evaluation paper that proposes a benchmark, the model evaluated are quite lacking. The work focused on Gemini, Claude and GPT-4o, putting aside many omni-LLM (Reka, AnyGPT, Video-SALMONN, UnifiedIO2, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vO9tgdXkqm", "reviewer": "Reviewer_aEZD", "summary": "This paper proposes a novel evaluation suite called Omni x R for omni-modality language models (OLMs) evaluation. They propose an automatic data synthetic method to convert natural language questions into other modalities, i.e., audio, image, video. Besides, they also annotate 400 samples from real videos to  other modalities. With Omni x R, they conduct experiments of several OLMs and provide insights to the develop of future OLMs", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a novel evaluating method called Omni x R for Omni-modality Language Models (OLMs) to assess OLMs over a diverse mix of modalities. Unlike other benchmarks that focus on one or two modalities, Omni x R evolves up to six modalities that embed information, presenting new challenges for OLMs.\n2. Omni x R can effectively measure the reasoning behavior discrepancy in different modalities. Besides, this paper demonstrates several key findings regarding this discrepancy. For example, they find extracting information and then conducting reasoning could improve model performance on modalities other than text.", "weaknesses": "1. The automatic data synthesis method is somehow oversimplified. For example, rendering text onto a white background image or using tts to translate text to audio may be used in synthetic training data generation. These kinds of data could vary largely from the data distribution in the real world, which makes the OMNI-R-Synth data unreliable in reflecting the model performance in real life.\n2. The benchmark size is small. There are only 400 real samples and 400 synthetic samples. Besides, Demonstrations of some examples of OMNI-R-Synth and OMNI-R-Real could help readers gain an intuitive impression of the data format.\n3. I’m curious about how humans perform in different modalities. A hypnosis is that audio could cause difficulty for humans to perform reasoning. Are the authors planning to add a human upper bound for Omni x R?\n4. When conducting ETA prompting, did the author measure the precision of the intermediate textual transcripts? Does the performance drop come from failure to catch the key information or failure to conduct reasoning when some modalities exist?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KB9qOw6V8u", "reviewer": "Reviewer_JM5H", "summary": "The paper introduces *Omni×R*, a benchmark suite for evaluating omni-modality language models (OLMs) like GPT-4o and Gemini. And its key contributions include:\n1. **Datasets**: *Omni×RSYNTH*, a synthetic multi-modal dataset, and *Omni×RREAL*, a real-world, expert-curated dataset for cross-modal reasoning.\n2.  **Pipeline**: It proposes *Omnify!* to create the synthetic omni-modality evaluation data from the original text benchmarks.\n3. **Prompting Strategies**: Tests of \"Extract-Then-Answer\" (ETA) prompting indicate it helps with synthetic data but is less effective with real-world data.\n\n*Omni×R* establishes a rigorous standard for assessing OLMs, advancing omni-modal AI development.", "questions": "Kindly find below additional questions. \n\n1. Can you provide the results of video+audio on the *Omni×Real* dataset?\n\n2. For Claude and GPT-4o, you only tested video and image or just the image modality. So, why not test more open-sourced models? like LLaVA, QWen, etc. They can receive videos and images as input.\n\n3. Why not test some open-source OLMs, such as Macaw-LLM, Next-GPT, or VideoLLaMA2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. **Gap Addressed**: The benchmark addresses a crucial gap in evaluating omni-modality language models, offering a framework for comprehensive cross-modal reasoning that goes beyond single or dual-modality assessments.\ns for advancing omni-modal AI systems.\n\n2. **Methodological Strength**: The methodological approach is robust, with a well-structured benchmark that includes both synthetic and real-world datasets, along with clear evaluation protocols and analysis.\n\n3. **Clarity and Organization**: The paper is well-organized, providing clear explanations and thoughtful insights into the challenges and current limitations of existing omni-modality language models.", "weaknesses": "Here are some weaknesses I observe:\n\n1. **Motivation of the Benchmark**: I understand the authors' goal of establishing a unified and comprehensive multi-modal benchmark to test OLMs' capabilities for holistic understanding and reasoning across modalities. I think the *Omni×RREAL* dataset achieves this goal to some degree. However, for the *Omni×RSYNTH* dataset, I don’t think it effectively assesses an OLM's cross-modal reasoning capabilities. Since *Omni×RSYNTH* is derived by translating from pure text, the images and videos it contains are entirely text-based, which means it primarily tests OLMs' OCR abilities rather than other critical capabilities like temporal, spatial, and grounding capabilities. In real-world scenarios, when faced with this type of data (e.g., slides), it would be more practical to use expert models to extract text information from other modalities, and then pass the task to LLMs. A more meaningful scenario might involve using films, where audio and video from movies are input to OLMs to answer plot- and character-related questions. I believe this setup would provide a more valuable test for OLMs.\n\n\n2. **Evaluation**: In the \"Extract-Then-Answer\" (ETA) prompting method, the authors first prompt the OLMs to generate corresponding OCR results and then answer questions based on the extracted text. As I mentioned above, given this approach, why do we even need OLMs for such tasks? We could just use expert models to handle the first step. In real-world scenarios, \"Extract-Then-Answer\" is impractical, so I feel that this evaluation method also lacks significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)