ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments?
Authors: Liu Yang
Huiyu Duan
Ran Tao
Juntao Cheng
Sijing Wu
Yunhao Li
Jing Liu
Xiongkuo Min
Guangtao Zhai
Venue: ICLR 2026 Poster (2026)
Keywords: omnidirectional image
benchmark
virtual reality
Primary area: datasets and benchmarks

Abstract: Omnidirectional images (ODIs) provide full 360$^{\circ} \times$ 180$^{\circ}$ view which are widely adopted in VR, AR and embodied intelligence applications. While multi-modal large language models (MLLMs) have demonstrated remarkable performance on conventional 2D image and video understanding benchmarks, their ability to comprehend the immersive environments captured by ODIs remains largely unexplored. To address this gap, we first present ODI-Bench, a novel comprehensive benchmark specifically designed for omnidirectional image understanding. ODI-Bench contains 2,000 high-quality omnidirectional images and over 4,000 manually annotated question-answering (QA) pairs across 10 fine-grained tasks, covering both general-level and spatial-level ODI understanding. Extensive experiments are conducted to benchmark 20 representative MLLMs, including proprietary and open-source models, under both close-ended and open-ended settings. Experimental results reveal that current MLLMs still struggle to capture the immersive context provided by ODIs. To this end, we further introduce Omni-CoT, a training-free method which significantly enhances MLLMs’ comprehension ability in the omnidirectional environment through chain-of-thought reasoning across both textual information and visual cues. Both the benchmark and the code will be released upon the publication.

Published as a conference paper at ICLR 2026
ODI-BENCH: CAN MLLMS UNDERSTAND IMMER- SIVE OMNIDIRECTIONAL ENVIRONMENTS?
Liu Yang": ; Huiyu Duan' 1.. Ran Tao?, Juntao Cheng', Sijing Wu', Yunhao Li', Jing Liu
Xiongkuo Min' Guangtao Zhai Shhnggaai Jiao Tong University
2Xinjiang University
Tianjin University
(a) Examples ODI-Bench
(b) Benehmark Results of MLLMs
Auuman Attribute
Object Attribute
ODI-Bench
GFI-4z A Gasmoddi--aas Grand Trait
GFT-Ay Gantn-221Blale
Grond
RAu
Anadir
Paintng
Fidig ragh
Palistog she dear,
vy right meating evang and jouns"
Blacz & While Black Rel
Counting
Egocentriic View Oriextation
thor?
Wilere tuke dowe?
I.B.B.G
GFT-de C
Close-ended
and
Open-ended
Grond Teatr
(c) Illustration Omoi CoT
Reletive Direction
Whrre the reppect ned oar Ayt??
Reasoning Ar they viling in line?
Onami COT
G-T- Yoi
@ more people are thre? CC
C Backy Eeft
GPT-4x
Connat Torth A
Suep: Yissporet- Aunworieg
View Orientation
(1 ditig in the par reft, vitere it paate Frosiviphic Preesight: Beckinghi Rock-lff
ca?? L. &. Right Bock Frast
of
Siep,Greundveg Refieiog rap
images helpful!"
GPT-4c: D
BI Gesisi-2.661 Greadd
Sopsl: Regporre Refivesent
"DR
Figure (a) We introduce ODI-Bench, a comprehensive benchmark for omnidirectional im- age understanding, covering 10 diverse tasks. (b) 20 leading MLLMs are benchmarked with both close-ended and open-ended evaluation. (c) To further improve model performance, propose Omni-CoT, a chain-of-thought framework that enhances MLLMs comprehension on omnidirec- tional images via step-by-step reasoning.
ABSTRACT
Omnidirectional images (ODIs) provide full 360 x 180 view which are widely adopted in VR, AR and embodied intelligence applications. While multi-modal large language models (MI.LMs) have demonstrated remarkable performance on conventional 2D image and video understanding benchmarks, their ability to com- prehend the immersive environments captured by ODIs remains largely unex- plored. To address this gap, first present ODI-Bench, novel comprehen- sive benchmark specifically designed for omnidirectional image understanding. ODI-Bench contains 2,000 high-quality omnidirectional images and over 4,000 manually annotated guestion-answering (QA) pairs across 10 fine-grained tasks, covering both general-level and spatial-level ODI understanding. Extensive periments are conducted to benchmark 20 representative MLLMs, including pro- prietary and open-source models. under both close-ended and open-ended tings. Experimental results reveal that current MILMs still struggle to capture the immersive context provided by ODIs. To this end, we further introduce Omni- CoT, training-fiee free method which significantly enhances MLLMs' comprehen- sion ability the omnidirectional environment through chain -f-hhughhtt reason-
"qqual contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
Table I: Comparison between widely adopted general benchmarks, omnidirectional benchmarks, and our ODI-Bench. The first row group presents commonly used image benchmarks, the middle row group includes two spatial benchmarks, and the last row group lists ODI benchmarks.
Berchnook
Kanges eQA Pais
(Quesion Type
Visial Modallt
Max Reso
Real Sceces
Open
Clow
Genrra
Diniision
Spaiil
ON Scncce
<IK
Marzal
MBV- ct st. 2023)
20 image
ViewSpuital Beach (Li et al, 2025)
1,00
5,700
5
30 Scaae
<IK
x
K
Aato
BealB (Yarg std. 20255)
29
5,000
NFOV Video
<IK
/
x
/
Alto
SSiBBack (Lix 2025)
789
750
2D Image
/
/
Asto
360
1000)
Indeor ODI
Beach (Dongfaug al. 2025)
153,00
Indoor ODI
Alto
Dene Bench (2hos 3025)
1,279
6,000
Indiookkitiiine ODI
Asto
ODt Bos-Besch (Oun)
2,000
4.254
Indoor&Ousiioe ODI
12K
Mianaall Juto
ing across both textual information and visual cues. Both the benchmark and the code will be released at https://githu..cmm/// http://www.oom/lllyll/u///----o-
INTRODUCTION
360 omnidirectional images (ODIs) have gained increasing attention nowadays. Unlike conven- tional 2D images with limited field of views (FoVs), ODIs provide a full 180 360 FoV with rich scene information, enabling fully immersive viewing. Thus, ODIs are widely used in virtual reality (VR), augumented reality (AR), spatial navigation, and hold great potential for embodied intelligence (Yang et al., 2025b; Zheng et al., 2025). Although recent advances in multi-modas large language models (MLLMs) have led to significant progress in conventional image understanding across various benchmarks (Liu et al., 2024b; Yu et al., 2023), their ability to comprehend ODIs has not been comprehensively evaluated, with existing benchmarks remaining insufficient. Compared to conventional images, ODIs capture substantially richer visual information from omnidirectional scenes and require higher-level spatial reasoning in immersive environments beyond a single front- view perspective. These unique characteristics make ODI understanding a distinct and difficult research challenge, highlighting the necessity to systematically evaluate MLLMs on this task.
Though a limited number of ODI understanding benchmarks have been proposed as shown in Ta- ble 1, they generally suffer from one more of the following issues: (1) Low resolution: since many applications (such as VR, autonomous driving) requires high-resolution ODIs to provide im- mersive viewing experience 360 details, benchmarks with low resolution are impractical for real-world application (Chou et al., 2020; Dongfang et al., 2025); (2) Limited scene diversity: some benchmarks are developed with the assistance of 3D-annotated indoor datasets (Chou et al., 2020: Dongfang et al., 2025), focousing only on indoor environments, or even unrealistic synthetic indoor scenes with blurry top-bottom views; (3) Constrained question domains: Existing bench- marks are automatically annotated, either leveraging existing 3D datasets or curated pipelines for question generation, thus tend to exhibit strong textual biases and provide relatively narrow or sim- plistic question types (Zhou et al., 2025): (4) Viewpoint limitation: for spatial understanding, exist- ing ODI benchmarks are primarily designed from an egocentric perspective, neglecting allocentric viewpoints and the simulation of user interactions. As a result, they fall short in evaluating the em- bodied aspects of spatial understanding, which are critical for advancing embodied intelligence and interactive multimodal systems.
To address these gaps, we introduce ODI-Bench, a novel ODI-oriented benchmark designed to com- prehensively evaluate both the general-level and spatial-level understanding capabilities of MI.MMs. The question-answer pairs are derived from two complementary sources: (1) rigorously designed automated pipeline that generates reliable instance-level QA pairs, which are further checked and refined by human experts, and (2) high-quality human annotations produced by three domain ex- perts, whose works are carefully cross-checked to ensure reliability. The final benchmark contains 2,000 high quality real-life omnidirectional images, covering diverse indoor and outdoor scenes. 10 representative tasks are proposed to facilitate fine-grained and pulti-perspeettve evaluation of the performance of MILMS under ODI settings, partially illustrated in Figure 1.
Unlike previous benchmarks that restrict the evaluation of each task to either a close-ended (multiple-chocce or true/false) or an open-ended QA setting, ODI-Bench evaluates every task un- der both settings. This dual-format design enables a comprehensive and comparative assessment, capturing both the recognition accuracy under constrained choice conditions and the model's gen- erative reasoning ability in unconstrained scenarios. Experimental results demonstrate that MLLMs still struggle to comprehend the immersive environment presented by ODIs. To this end, we fur- ther propose a training-fre free chain-of of-uhht framework, termed Omni-CoT, to improve MLLMs' understanding capabilities ODIs through step-by-step reasoning with viewpoint-guided scene
2

---

Published as a conference paper at ICLR 2026
interpretation and visual cue based refinement. This approach significantly enhances MLLMs' com- prehension on ODIs across both general and spatial-level tasks. Our contributions are summarized as follows:
We introduce ODI-Bench, comprehensive benchmark for evaluating MLLMs on om- nidirectional image understanding, which consists of 2,000 high-quality omnidirectional images and over 4,000 QA pairs across 10 fine-grained tasks, covering both general and spatial-level ODI understanding.
We conduct in-depth study evaluate the ODI comprehension ability of 20 leading MILLS on our ODI-Bench, using both close-ended and open-ended settings to make comprehensive and comparative analysis. Experimental results reveal the challenges of MLLMs in understanding immersive ODI scenes.
We propose Omni-CoT, a training free strategy to enhance MLLMs' comprehension ca- pabilities omnidirectional scenes through chain-of-thought reasoning. Experimental results demonstrate the effectiveness of the proposed framework on both propritary and open-sourced models.
RELATED WORKS
2.1 GENERAL UNDERSTANDING BENCHMARKS
With the advancement of MLLMs, there is increasing need for comprehensive and systematic evaluation of their visual understanding capabilities. number of benchmarks have been developed to assess the general-level comprehension ability of MLLMs (Liu et al., 2024b; Yu et al., 2023; Duan et al., 2025). However, as the performance of MLLMS has significantly improved, such gen- eral benchmarks marks are proposed to evaluate the spatial understanding ability of MLLMs (Yang et al., 2025a; Liu et al., 2025), presenting new challenges for MILMs spatial understanding tasks. However, while most of these benchmarks focus on 2D images or NFoV videos, the benchmarks specifically designed for omnidirectional images are still scarce. Given their unique format and application sce- narios, the ability to understand ODIs holds great potential for advancing not only MLLMs but also vision- waann--nnngggaatiiinnn (VLA) models.
longer sufficient for thorough ability assessment. More recently, new bench-
2.2 OMNIDIREECTIONAL IMAGE UNDERSTANDING BENCHMARKS
A limited number of ODI understanding benchmarks are proposed evaluate MILMs' understand- ing of this unique type of images. Dense360-Bente (Zhou et al., 2025) introduces QA curation pipeline and further constructs benchmark for general-level grounding and captioning tasks on ODIs, but such tasks remain superficial and fall short in adequately evaluating spatial understanding abilities. VQA 360 (Chou et al., 2020) constructs a benchmark for simple ODI understanding tasks, but the image resolution is too low (1024 x 512), constraining its applicability in real-world sce- narios. OSR-Bende (Dongfang et al., 2025) develops a pipeline for generating ODI spatial compre- hension QA pairs from 3D datasets, yet it focuses solely synthetic oow-resolution indoor scenes, limiting its applicability. In contrast, our ODI-Bench is the first to comprehensively benchmark both the general-level and spatial understanding capabilities of MLLMs on ODIs, with carefully manu- ally curated QA-pairs assisted by an automatic annotation pipeline, encompassing both high-quality indoor and outdoor scenes.
ODI-BENCH
3.1
OVERVIEW OF ODI-BENCH
In this section, we introduce ODI-Bench, a benehmark for comprehensive evaluation of MLLMs on undersstandinggODI-eenccchh
omnidirectional image understanding. ODI-Bench consists of 2,000 real-world omnidirectional im- ages covering diverse indoor and outdoor scenes, along with 4.254 question-answering pairs across 10 ine-grained tasks, offering both general-level and leat ODI understanding evaluation.
3.2
IMAGE COLLECTION
The images in our benchmark primarily web-crawled from Flickr and carefully selected to sure both quality and diversity. The distribution of images is presented in Figure 2 (a) and (b). Com- pared with existing omnidirectional image understanding benchmarks (Dongfang al., 2025; Chou
3

---

Published as a conference paper at ICLR 2026
Inkkr Soco, Dtotttutce
14.2%
7.0%
1280
*K
Beatsrce
6%
Eppomentt Orania Kon
Indeor
Cutdoor
172
rounring
19.5%
4k-6k
2000 ODTs
ObI-Bench
Humon
4000+ QA
Simukntion
Oclio Sune Diobetis
Qk-bk
10 Tasks
Albuaetricc Video
Girintatien
6K-8K
47 47
52.6%
24.%%
OUR
ost Reaiceaty
Der perection
Relirted
500
11,074
*BK
.AK
27%
7.2%
(a) Seene Distribution
(b) Resolution Distribution
(c) Task Distribution
Figure 2: Data distribution in ODI-Bench.
et al., 2020) which are largely restricted to indoor environments and predominantly depict house- like scenes rendered from 3D datasets, our benchmark covers diverse indoor and outdoor scenes, ranging from human activity to natural landscapes, hereby enabling a comprehensive evaluation across diverse scenarios. In addition, some exsiting benchmarks suffer from low image quality. For instance, VQA 360 (Chou et al., 2020) consists of ODIs with blurry top and bottom views and the image resolutions are limted to IK, which restricts their practical value in real-world applications. In contrast, as presented in Figure 2 (b), our benchmark is high quality, with sufficient resolution to ensure both practical applicability and reliable benchmarking.
3.3 FINE-GRAINED TASK DEFINATION
Unlike previous works that evaluate MILMs' comprehension abilities on ODIs either the general level (Zhou et al., 2025) or the spatial level (Dongfang et al., 2025), or through simple combination of the two (Chou al., 2020), we carefully design 10 fine-grained tasks tailored for comprehensive ODI understanding, covering both general-level and spatial-level aspects, as shown in Figure 2.
General-level ODI understanding. Inspired by conventional 2D image understanding tasks, we propose five main general-level tasks to evaluate MILMs' comprehension on common ODI sce- narios. These tasks typically impose relatively low spatial reasoning requirements, while the main challenges arise from the massive amount of visual information and distorted projections inherent ODIs. Among them, we define instance-level tasks, i.e., object-attribute and human--ttribute, to assess the models' ability to accurately localize instances and extract visual information across wide fields of view. In addition, we introduce counting and existence tasks to measure the models' global perception capabilities ommidirectional images. Finally, define omnidirectional OCR task to evaluate the models' capability to extract textual information under distorted perspectives and high-resolution conditions, including cross-view scenarios.
Spatial-level ODI understanding. Omnidirectional images project immersive 3D scenes onto a 2D plane, leading substantial differences in spatial perception compared conventional 2D im- ages. In 2D images, the viewer is constrained to a single front-facing perspective, while ODIs provide a full 360 field of view encompassing front, back, left, right, top, and bottom perspec- tives. To evaluate the capability of MLLMs in handling such unique spatial characteristics, we design dedicated spatial- ODI understanding tasks. These include egocentric view orientation and relative direction tasks, which adopt the viewer's own perspective, as well as allocentric view orientation and scene simulation tasks, which involve perspective-taking from another agent or a virtual viewpoint. Furthermore, due to the equirectangular projection (ERP) format of ODIs, spa- tial relationships and motion trajectories often become distorted and confusing. To address this, we introduce an ODI-reasonig task, specifically designed to assess MILMS ability to understand and interpret ERP-related spatial properties in ODIs, as illustrated in Figure
3.4 QUESTION-ANSWERING ANNOTATION
Instead of conventional phrasing in 2D images benchmarks (e.g., "What [A] on the right side of the image doing?"). adopt an immersive question design tailored to the characteristies of ommidirectional images, i.e., "What is [A] on my right doing?". This first-person phrasing not only aligns with the natural immersive viewing experience of ODIs but also serves to evaluate the ability
4

---

Published as a conference paper at ICLR 2026
Imuge Collection
(c) Curation Pipeline
Wrong Format
Back
@) Instance Segmentation
Blurry Content
Viewport
Grnandel
Split
SAM
Diverse Socne
Web-crawled
images
Solection
Hlgh Quality
(R) Deseription Generalion
Doaris the instarae the
"A A weoring red hoodse dank poits, and while mook."
(b) Manual Annotation
Labd-Man
"Whr is the mumber of train"
(iil) OA Generation
March
"What color the hoodle the man mask boak
(Iv) Choice Generation
Uniqe Refareno
210
Asewe Ancuracy
Expert Amnotation
Cross Check
"Ral"
Choioe
Figure 3: Construction procedures of ODI-Bench. (a) The benchmark images carefully selected to ensure quality and diversity. (b) The majority tasks are manually annotated by human experts. (c) Object Attribute and Human Attribute QA pairs are generated through dedicated annotation pipeline with human verification to guarantee quality.
of MLLMs understand and utilize ODIs in interactive environments. For the QA construction process, we adopt both automatic pipelines and manual annotation process for different tasks.
For instance-level QA generation, i.e., object attribute and human attribute, automatic pipeline is adopted as presented in Figure The ERP-formatted image first paooemap-proected into non-overlapping viewpoints for lower distortion. After that, GroundedSAM (Ren al., 2024) is adopted to segment instances in each view, producing both segmentation masks and instance labels. To ensure precise instance segmentation, instances spanning multiple views are filtered out. The re- maining instances are cropped based on the segmentation masks and fed into Dwen2.55VL-77. al., 2025b) for detailed caption. To ensure reliable instance selection, only those instances whose predicted categories by GroundedSAM are consistent with the descriptions from Ww-nn- are retained. These captions are further utilized by GPT-4o to generate QA pairs. All the generated QAs manually refined to ensure (i) unique reference, that each question clearly targets single instance; (ii) answer accuracy, guaranteeing the correctness of the provided answers.
(Bai
For more complex tasks including counting, view-orientation and relative direction, etc., automatic generation is not trust-worthy enough, thus manual annotation is employed for precision. The anno- tation process took month in VR environments with three expert participants, whose annotations were cross-checked to guarantee accuracy. To construct close-ended evaluation, GPT-4o is adopted generate 3 distractor options based on the QA pairs and the corresponding omnidirectional im- ages. Each multiple-choice question is accompanied by three distractors, which are further assessed by human annotators to ensure their plausibility and to avoid semantic overlap with the correct answer. To mitigate model bias, the options are randomly shuffled, thereby ensuring a balanced distribution of correct answers across choices A to
EXPERIMENT
4.1 EVALUATION SETUP
We conduct comprehensive experiments on 20 leading MLLMs with different architectures and parameter scales on our ODI-Bench. The models can be categorized into two groups: (1) proprietary models, including GPT-4o (Hurst et al., 2024), o3 (OpenAI, 2025), Gemini (Comanici et al., 2025), etc. (2) open-sourced models, including Intern VL series (Zhu et al., 2025), Qwen- VL series (Bai et al., 2025b), LLa (Li et al., 2024b), LLava-One are evaluated using the same prompt template provided the Appendix.
(Li et al., 2024a), etc. All models
We believe that model performance may vary under different evaluation settings. iee, close-ended and open-ended conditions. Unlike prior benchmarks, we benchmark all models across all tasks us- ing both close-ended formats (multiple-choice yes/no) and open-ended formats, providing com- prehensive and comparrative assessment of their capabilities. For close-ended benchmark, model performances measured by their accuracy multi-choice yes/no questions. For open-ended benchmark, we adopt the L.M-aased evaluator (Yu et al., 2023), as detailed the Appendix E.3. Finally, we report the average score for each task.
5

---

Published as a conference paper at ICLR 2026
Table 2: Benchmark results for MLLMs under the close-ended evaluation setting. The tasks are de- fined as follows: OA (Object Attribute), HA (Human Attribute), Exist. (Existence), Count. (Count- ing), EVO (Egocentric View Orientation), AVO (Allocentric View Orientation), SS (Scene Simu- lation), RD (Relative Direction), OR (ODI Reasoning). For each task, the best-performance is indicated in bold and the second-best are underlined.
Ovvall
Genyrll
Spattal
OA
HA
Exist.
Count.
OCK
EVO
AVO
SS
RD
OR
Proprictury MLLMs
GPT-4a (Hurst al., 2024)
55.79
74.43
67.76
65.50
49,42
74.38
43.24
32. $2.49
39.60
57.55
$3.50
Qwen- PL. (Bai et al., 201250
52.85
74.51
68.75
43.00
52.35
67.77
46.13
29.98
34,80
50.61
49.00
(lo-inl--200fftss (Comanici 2025)
57.12
75.03
69.41
66.55
52.35
90.16
48.10
32.91
40.20
61.13
$4,00
o3 (OpenAI, 2025)
62.62
75,82
74.01
75,00
57.56
77.69
56.20
39.62
46.60
70.20
59,50
Open-wureed ML.LASs
vL (Liu 2024a)
45.04
51.44
43.86
25.93
19.60
24.08
LLaVA-uv-. (Li al., 2024g)
44.25
61.21
61.26
42,00
44,77
27.27
30.61
29.77
35,40
50.20
32,00
idefies3-81 SB (Laurençon et al., 2024)
49.89
68.45
65.32
62.50
50.00
57.85
36.71
28.79
30.20
49.39
49.50
2Composer2 (Dong
51.84
75.57
26.32
44.00
54,07
25.02
46.56
27.20
31. 31.84
46,00
Drepsock VL-1.36 (Lu et al., 2024)
42.02
$5.66
50,00
52.50
39,55
29.75
39.02
18
28.00
23.27
49,00
LLava-Next-7B (Li at. 2024b)
45.91
64.84
54,93
29.77
40.42
27.04
21.20
32.24
53.50
1Lava-Next334B (1. ot 2024b)
52.24
70.57
62,17
44.50
45.95
45.45
47.42
26.42
38.00
45.71
57.50
glm-4v-98 (GI at. 2024)
53.20
70.41
64.14
69.00
54.65
44.29
33.91
30 30.80
43.27
57.50
MiniCPM-V (Yao etal. 2024)
53.71
51.74
74.38
42.58
33.00
49.39
$1,00
IntenV1 2.5-V/ (Chen al., 2024)
$2.76
68.52
70.07
60.00
$1.74
66. 12
45.23
31. 1..24
33.00
44.08
58.00
Qwen2.5-VL_3B (Bai al. 2025b)
20255)
52.88
33.00
45.71
$3.50
Qwen2.5.VL VL- (Bai et 2025b)
56.70
74.89
67.76
60.00
57.56
72.73
45.97
35.01
38.20
59.59
$4.50
Qwen2.5-V1.-22 (Bai
20255)
56.91
27.38
68.75
52,00
58.14
75.21
16.75
32.08
38,
60.00
$0.00
IntemVL33 3SB (Zhu al., 2025)
57.91
75.57
76.32
69.00
51,07
76.96
46.56
32.29
40,40
55.95
$6.50
IntenVL3-788 (Zha et al., 2025)
59.43
79.18
77.30
66.30
59.30
80.99
46.01
31.67
40.40
60.82
58.50
Inten-sl 3025a)
42.37
63.93
40.94
33.06
20.15
08.08
45.
43,00
Randine
Blind GPTT-4
36.39
58.93
42.45
17.00
17.44
29.75
21.96
29.14
31.60
29.
25.50
Random Choice
26.93
25.00
25.00
25,00
25.00
25,00
25.0
35.00
25,00
25
Table 3: Benchmark results for MILMS under the open-ended evaluation setting. For each task, the best- rest-perforrance is indicated in bold and the second-best are underlined.
General
Spotial SS
Overal!
OA
HA
Exist.
Connt.
CCR
EVO
AVO
RD
CR
Proprictory MLLMI
GPT-4o Cllunt el 2024)
42.91
52.62
45.35
32.27
27.35
30.30
Quen-VLL-Pls (Bai ot al. 2025k)
39.87
44.30
39.35
67.50
47.00
36.36
34.39
20.10
27.65
51.09
46,20
Gemini-2.0-naal 2.0-Dash (Comanici et al., 2025)
36.42
37.82
28.55
48.26
50.00
56.50
30.86
23.89
31.00
55.51
42.17
o3 (OpenAL, 2025)
49,33
$5.49
45,36
69.50
50,00
52.89
45.40
34.59
39,60
62,04
59.10
Open-rounced MTT.My
LL (l.i et 20246)
32.29
32.07
56.50
9.504
28.28
26.60
45.
LLaVA-ov-0.8B (La et 2024g)
13.90
28.75
11.12
42.00
19.19
3.719
6.196
5.765
8.300
29.18
32.20
idefes3-88 (Lanrengon et al., 2024)
23.93
$1.50
28.16
65.00
43.60
27.27
16.75
15.41
20.60
37.35
37,90
XCompase2 (Dang
28.36
30.12
34.88
25.79
23.00
42.65
43.15
Deepseck-VL. VL-1.38 (Lu et al., 2024)
27.80
31.33
18.22
$3.50
33.14
6.198
20:98
19.92
23.10
42.24
44.15
1aavaNexx-788
20144)
30.52
36.58
60.50
40.12
12.81
21.04
22.01
17.60
38.
1Lvv-Next-344 al., 2024bb
35.25
41.57
29.28
49.50 50
41.52
7.025
35.40
20.21
24.60
45.51
52.42
glm- (GLM etal. 2024)
35.79
38.50
26.12
60.50
42.44
32.23
36.38
25.26
26.50
41.43
42.95
MisiCPM-V 4.0 et 2024)
32.52
36.00
46
IniemVL2.55 SD (Chen et 202.)
30.86
33.52
22.34
62.00
39.53
34.71
21.96
25.68
20.40
38, 98
$1.55
Qwen25-VL-38 (Bai at.l.
38.91
39.64
41.86
41.32
35.72
25.09
26.80
47.35
36.20
Owen2.5.VLL3328 (Bai et 2025h)
37.67
39.56
26.34
62.00
42.45
30.91
36.47
30.21
30.45
50.41
44.25
Qwen2.5 VL-722 (Bai et 2025b)
39.49
45.39
35.14
63.00
42.44
39.55
34,91
24.10
27.60
54.39
47.75
IntenVL33-388 (Zha al., 2025)
40.96
40.94
33.43
34,66
29.77
27.60
$4.08
52.62
IntenV1 3-788 (7hn et at., 2005)
42.52
47.05
43.91
62.00
57.56
48.76
35.58
20:77
29.85 02.82
53.27
$3.05
Inter-sl 2005p)
42.12
63,95
34.72
31,87
52.00
4.2 MAIN RESULTS
Close-ended and open-ended performances of all models across all tasks are reported in Table 2 and Table 3, respectively. For the close-ended evaluation, we additionally include GPT-4o without image input (Blind GPT-4o) and chance- accuracy as baselines for comparison.
4.2.1 OVERALL PERFORMANCE
As illustrated in Table 2, proprietary models achieve the strongest overall performance under both the close-ended and open-ended evaluation settings, where ChatGPT o3 attaining the top overall score of 62.62 and 49.53, respectively. Open-source models also show competitive results, in which Qwen2.5-VL-722 and VL-77B even outperforming GPT-4o. However, the results are still far from satisfactory, revealing that current MLLMS still struggle to comprehend the immersive environments presented by omnidirectional images. Besides, the best-performing model (o3) ex- ceeds the Blind GPT-4o bascline by less than 30% accuracy under close-ended setting, suggesting that MI.LMs still struggle to comprehend the rich visual information conveyed by ODIs.

---

Published as a conference paper at ICLR 2026
4.2.2 TASK-WISE PERFORMANCE
From both the close-ended and open-ended evaluations, it is evident that ODI spatial understand- ing is substantially more challenging than general understanding. For general-level tasks such as attribute recognition, existence verification, and OCR, the complexity of immersive environments increases task difficulty; nevertheless, models can still partially interpret ERP images from 2D per- spective and thereby produce correct answers, which aligns with their already strong capabilities in 2D general-level understanding. However, for tasks more closely related spatial comprehension, i.e. counting, model performance drops significantly (by about 20% compared with attribute recog- nition tasks).
The challenge becomes even more pronounced for tasks that fully rely on immersive spatial compre- hension. Since current MLLMs are primarily trained 2D data, their spatial reasoning capabilities are inherently limited. These limitations become especially obvious when it comes to omnidirec- tional comprehension requiring immersive spatial understanding, where model performance drops greatly compared to general-lever tasks, only slightly above the random choice baseline. This issue particularly evident in non-egocentric spatial reasoning tasks. i.e., allocentrie view orientation and scene simulation, which are already difficult in conventional 2D images (Li et al., 2025). In the ODI setting, these tasks pose an even greater challenge, with model performance only marginally surpassing (or even falling below) that of the Blind GPT-4o baseline, suggesting that current models still fall short capturing the spatial information conveyed by omnidirectional images.
4.3 CLOSED VERSUS OPEN EVALUATION
Comparing Table and Table 3, we observe that model performance differs greatly between closed and open-ended QA settings. The findings demonstrate the necessity for conducting both closed and open-ended benchmarks. For tasks with a unique ground truth, Le.. counting and OCR, we can directly compare their performance across the two tables. The performance of these two tasks generally drops, indicating that (he choices may provide a hini for the MI..Ms. Interestingly, not all answer choices produce a positive effect. We observe cases where model provides correct response in the open-ended setting but selects the wrong option in the close-ended format. This discrepancy suggests that the presence of predetined options may sometimes introduce interference, and further reflects a potential difference between the model's generative reasoning in open-ended tasks and its discriminative reasoning in multiple-choice tasks.
We further observe that the model performance divergence is more evident in the open-ended set- ting. especially on spatial-level tasks, where models exhibit significantly larger variations. Un- like multiple-choice questions, where the given options constrain the answer space, open-ended responses can better reveal the differences between MI.LMss' reasoning and that of humans. For example, in the egocentric view orientation task, when no explicit constraints are imposed, models rarely produce ego orientation terms. Instead, they tend to describe orientations in relative terms. However, when explicitly instructed to output absolute orientations, the models' performance remains unsatisfactory. This suggests that MLLMS do not naturally reason about immersive ODI scenes in human-like manner, i.e., by first engaging in perspective-taking and then conducting relative spatial analysis. Instead, their reasoning still resembles processing a warped 2D image.
5 OMNI-COT: IMPROVING MLLMS UNDERSTANDING OF ODIs
In this section, we propose Omni-CoT, training-free free framework for improving MLLMs under- standing capabilities on ommidirectional images by leveraging a human-like step-by-step chain-of- thought reasoning strategy, including viewpoint-guided answering, crop cue grounding and refining. and response refinement.
5.1
FRAMEWORK OVERVIEW
5.1.1 VIEWPOINT GUIDED ANSWERING
Unlike conventional images, ODI comprehension requires MLLMs to extract viewpoint cognition from the projected ERP-format images. However, as analysed in Section 4.3, MLLMs often per- ceive ODIs merely as warped 2D images rather than reasoning within the immersive full-view set- ting, which poses a significant challenge even for proprietary models. In 2D image comprehension, spatial understanding is often enhanced either through large-scale training or by incorporating prior information generated from 3D models (Yang et al., 2025a; Li et al., 2025). However, training-base approaches are intencceinteeivve and may overfit to the training data, While 3e-devived features
7

---

Published as a conference paper at ICLR 2026
Direct Answering
Omni-CoT
Step Fieupoint-Giin Guided Auswering
Grounding and Refining of Cues
Chep:
are ghas onssilvacionall raage.. queniiee
asore grossslng
DXE yP 040, 3.R
Rerrnbe te dikragornn laage
keage donts hadren wit
Rofier
positioned side
goozioo??
by side?
You air glin condinationl aoog and strad naseanber:
A: Yes. (Ground Truth)
Hight viight desrppaxe>
Step 3: Response Refinement
NNo.
Your <.l Bused redhint <RIAEES
PUR
We introduce Omni-CoT, The framework enhances VLMs' comprehension of omnidi- rectional images via chain-of of-thought reasoning through three steps: diewpoint-guidee answering, grounding and refinement of crop cues, and response refinement. Compared with direct answering, Omni-CoT achieves notable performance improvements.
Figure 4:
can provide useful cues, reliance on external models is often insufficient and not widely applicable. To this end, We aim explore training-frre free approaches to enhance MLLMs' understanding ODIs by leveraging internal scene information through step-by-stey reasoning.
straightforward approach to feed ODIs along with the multi-view images splitted from them into MLLMs, effectively guiding the models to view the ODIs in human-like way by incorporat- ing viewpoint information. However, this approach not practically feasible, as omnidirectional images inherently have high resolution, combining them with high-resolution multi-view inputs can easily exceed the model's maximum input capacity, leading to failure. Moreover, high-resolution multi-view inputs generate large number of image tokens, most of which are redundant or ir- relevant, potentially hindering the model's ability to focus on the critical information within the omnidirectional images.
To this end, we propose a more efficient approach by guiding MLLMs to explore the immersive environment presented by ODIs using compact textual prompts rather than additional image inputs, as presented in Figure 4. Specifically, multi-view images are first extracted from the inverse sphere projection to generate six perspective views, i.e., top, bottom, right, left, front and back. Subse- quently, we use the MLLM to generate captions for each of the six viewpoints, capturing the key information contained in each view. By integrating these captions with the corresponding orien- tation information, the model can acquire a coarse understanding of the surrounding environment, thereby enhancing its global perception of the omnidirectional scene.
5.1.2 CROP CUE GROUNDING AND REFINEMENT
Directly extracting visual information from the full warped ERP image be challenging for MLLMs. To address this, we propose crop cue projection strategy. The MLLM is tasked with iden- tifying the most relevant image crops, from which narrow-FoV crops extracted as low-distortion visual cues to aid ODI comprehension. For a grounding box (x V1, T2 U2) where (#1, V1) and (12, 3/2 represent the normalized coordinates of the top-left and bottom-right corners, respectively, the spherical parameters of the narrow-FoV crop, i.e. the spherical coordinates of the narrow-FoV cue's center (9, Φ) and the approximate FoV for are computed as:
Cz 360 . = 90 Cy 180%,
= 110 +
(1)
W
H
foo (x2 - xn) ) 360 Jov, = (y2 - y1) 180?,
(2)
for clip max max((fow.,fooa) + margin, 30 120%)
where (Cz, Cy) is the center of the crop, W and H are the width and height of the ERP image, respectively, margin is an optional angular expansion to avoid overly tight cropping, and clip(.) limits the V to reasonable range.
However, since grounding may not always be accurate, relying solely on it can introduce unnec- essary distractors. Therefore, introduce refinement mechanism, where the MILLM is queried
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance of Omni-CoT on ODI-Bench, better performances over baseline are bolded.
Overai
Generul
Spatial
OA
IIA
Exts.
Coum.
OCR
EVO
AVO
SS
RD
OR
62.62
74.11
75.00
57.56
77.69
56.20
70.20
Viewpout Guiding
68.78
75.57
75.66
78,00
35.1
77.69
51.23
41.93
54.00
66.57
62.00
A(T)
46.16
-0.25
+1.5
+3.00
-1.75
4000
-25.03
42.31
48.20
-163
+2.50
(w/ Omini-Coo)
70.03
75.6
A(t)
+7.41
+0.33
+1.65
+6.00
46.97
4082
43640
43.36
48.00
4082
+2.5
GPT
35.79
74.43
67.76
63.50
19.42
74.38
43.24
32.49
39.00
57.55
53.30
(w/ Viewpoint Guiding)
61.67
73.60
66.26
74.00
76.03
70.06
37.31
39.80
53.65
56.50
A(t)
+3.88
0,74
2,48
+5.50
+4.65
+1.65
+26.8
+4.82
+0.20
3.67
+3.00
Ommi--CoT)
62.08
68.42
75.00
76.03
71.33
37.60
$2.65
A(1)
+6.17
0.66
+0.66
+9.50
+1.65
+28.29
+5.45
4,90
+5.00
Gemini-2.--tssh
57.12
75.00
89.41
66.50
52.33
80.16
48.10
32.91
40.20
61.13
$4.00
Viewpoint Guiding)
75.50
72.64
A(t)
+5.83
+0.57
-0.66
+9.00
+2,00
+3.31
-34.54
+2.94
40.8
+1.32
+0.00
(w/ Culn
63.89
73.77
69.41
76.90
57.56
$4.30
74.36
36,06
42.20
62.45
55.50
+6.77
+26.26
Q11-255V1-733
56.91
77.38
68.75
52,00
56.14
75.21
46.75
32.08
38.40
60,00
50.00
(wl Viewponit Guiding)
64.51
76.39
7302
66.50
55.40
77.87
76.07
37:31
46.61
$4.89
51.00
A(t)
(w/ Ommi--CoT)
65.41
76.00
74011
66.50
54.32
77.87
$n.12
37.54
45.20
56.33
51.50
A(t)
43.50
45.6
+14.50
+2.66
+33.37
+6.80
-367
VL2551
52.76
70.07
45.23
33,00
44.18
Owl Viewpomn Guiding)
35.76
68.77
7169
63.00
19.1
90.17
$2.39
34.98
34.00
48.95
60.30
A(t)
+3.00
1262
+3.00
+14.55
07.16
+1.40
+4.00
(w/ Omoi-CCoo))
38.04
71,48
72.99
63.50
5233
86.99
56.28
35.69
34.30
49.89
61,50
A(t)
+3.28
+2.96
+2.62
+3.50
+0.59
+1397
+3305
+4.40
+1.40
+3.31
+3.50
Table 5: Ablation studies of Omni-CoT on ODI-Bench
Model
Stratogy
Performace
Viewpoint Guiding
Crop Grounding Crop
General
Spatial
Gemini 2.0-llsh
57.12
70.49
45.05
63.07
72.08
54.94
62.79
71.79
54.67
70.05
Inter V1.2..8
66.33
55.76
67.82
44.88
53:71
65.79
58.04
69.81
47.43
again to label the relevance of each crop with respect to the question as "yes" or "no", and only the crops deemed relevant are fed back to the model assist the response refinement step.
5.1.3 RESPONSE REFINEMET
Finally, the ODI, along with the viewpoint captions, as well the crop cues with their orientation information derived from the spherical coordinates, is fed back to assist in refining the response. The model provided with previous answer and prompted to rethink the answer based on the crop cues. As illustrated in Figure 4.
5.2
OMNI-COT PERFORMACE
5.2.1
PERFORMANCE IMPROVEMENT ON ODI-BENCH
We conduct close-ended experiments on o3, GPT-4o, Gemini-2.0-flase Swwe2.55-VL-723 and Inte-nVL.2.588B validate the effectiveness of our framework both proprietary and open-sourced MILMM,, the results are presented in Table 4. As indicated in the table, viewpoint guiding serves as key component for performance improvement, especially on spatial-related tasks. Moreover, the models performance could be overall further improved with the aid of crop cues.
5.2.2 ABLATION STUDIES
We conduct ablation experiments to validate the effectiveness of each step in Omni-Con with results presented in Table 5. The results indicate that relying on direct grounding and cropping introduces unnecessary crops, which degrade model performance. However, the proposed crop refine-

---

Published as a conference paper at ICLR 2026
Table 6: bolded.
More hyperparameter ablation of Omni-CoT on ODI-Bench, best performances are
Mechol
Ovemal
General
Spottal
Exist
Count.
OCR
55
RD
GPT-40
55.79
74.33
67.76
65.50
89.42
74.38
43.24
32.49
39.60
57.55
53.50
Gom ind RoV)
60
06.30
6600
24.40
51.51
77527
73.30
38.00
57.65
5a500
Ouub-CoT (100" Ful)
60.71
67.29
67.11
74.50
54,07
73.55
72.27
57.73
40,40
55.10
61.50
Omai CVT 190" PoY (Ours)
62.08
75.77
75.00
54,07
76.03
37.94
37,00
52.65
Table 7: Performance comparison of multi-view input, video input and proposed Omni-CoT, best performances are bolded.
General
Spatial
QA
TEA
Esist
Count.
OCR
EVO
AVO
SS
RD
OR
GPT--o
55.79
74.43
67.76
65.50
49.42
74.38
43.24
32.49
39.60
57.55
53,50
(multi-viow ínpur)
56.01
14.10
67.11
71.00
$3.49
76.03
45.15
32.91
37 60
53.24
54.00
(w/ Omnii-CoT)
62,08
73.77
68.42
75.00
54.07
76.03
71.53
37.94
57.60
52.65
58.50
Gemsini-2A-faahh
57.12
73.03
60.41
66:50
52.33
80.16
48.10
12.91
40.20
61.13
54.00
(nuit--view input)
72.95
55.81
63.64
55.92
55.30
(video imput)
55.34
70.66
67.11
67.00
48.84
67.77
48.59
30.40
36.00
66.12
52.50
(w/ Ommi-CuT)
63.89
73.77
69.41
76.50
57.56
84.30
74.36
36.06
42.20
62.45
55.50
IL1e5mVL2..-SB
52.76
68.52
70.07
60.00
51.74
66.12
45.23
31.24
33.00
44.08
58,00
(multi-view inpur)
71.38
62,50
47.67
76.86
39.14
22.22
33.60
(video angut)
50,63
68.85
71.28
64.00
43.60
61.16
36.32
25.27
34.00
53.88
00:00
(w/ Ommi-CoT)
58.04
71.48
72.69
63.50
52.33
80.99
58.28
35.64
34,40
49.39
61.50
ment step filters out irrelevant crops, thereby mitigating this negative effect and further boosting performance beyond the baseline with viewpoint guiding. Besides, viewpoint guiding serves as the fundamental component to improve model comprehension of spatial understanding, while crop grounding supplies the model with important visual cues for comprehensive understanding.
We further conduct hyperparameter ablations GPT-4o evaluate the effectiveness of the per- spective FoV settings in Omni-Co'.. As shown in Table 6, using a field of view of 90 leads to the best overall performance, supporting the rationality of our chosen configuration.
5.2.3 COMPARISON EXPERIMENTS
We provide additional baseline comparisons using both multi-view images and video-based inputs. For the multi-view setting, each omnidirectional image is projected into 12 perspective views (front, front-right, right, right- right-back, back, left-back, left, left-front, top-front, top-back, Nottom-front, and bottom-back) and fed into the VLMSs along with their orientation information. For the video set- ting, we convert each omnidirectional image into a 12-second, 60-FPS video that smoothly rotates through the front, right, back, left, and front views, followed by the top and bottom views. Besides, in Table 7 in the supplementary material, we compare the performance of Zero-shot CoT with our proposed Omni-CoT.
The results are presented in Table 7. Though multi-view and video-based inputs offer improve- ments on relative-direction and ODI reasoning tasks, they do not yield clear overall performance gains. Besides, their effectiveness is further limited by the models' inherent constraints in han- dling multi-image or video inputs. While Zero-shot CoT provides minimal improvements on model performance, no clear improvement is observed in spatial-level tasks. In contrast, Omni-CoT con- sistently achieves the best results across all evaluated models, demonstrating its effectiveness and superiority over other inference pipelines. The experiment further highlights the necessity of our dedicated reasoning strategies tailored for omnidirectional image understanding.
6 CONCLUSION
In this work, we introduce ODI-Bench, a comprehensive benchmark for evaluating MLLMs' ability to understand immersive environments presented by omnidirectional images. ODI-Bench consists of 2,000 high-quality omnidirectional images and over 4,000 question-answering pairs spanning 10 fine-grained tasks, covering both general-level and spatial-level understanding. We benchmark 20 leading MLLMS using both close-ended and open-ended evaluation settings. Our in-depth analysis of the experimental results reveals that current MLLMs still underperform on omnidirectional image understanding To address this, we further propose Omni-CoT, a training-fri free approach to enhance omnidirectional images through chain-of tf-tuuugttt reasoning. Overall,
MLLMs' comprehension ODI-Bench provides a rigorous yardstick for both evaluating and improving MLLMs on immersive environment understanding presented by ODIs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under Grants 62401365, 62225112, 62271312, 62132006, U24A20220, and in part by the China Postdoctoral Science Foundation under Grants BX20250411, 2025M773473.
ETHICS STATEMENT
Our work adheres to the ICLR Code of Ethics. This work introduces ODI-Bench, a benchmark for evaluating MLLMs understanding omnidirectional images, and Omni-CoT, training-free immersive environments. All images in ODI-Bench are
framework to enhance comprehension sourced from publicly available web resources under permissible use, with personally identifiable information or sensitive data are included. Annotation was conducted by domain experts using VR devices, and only consensually verified QA pairs were retained ensure annotation reliability, Our benchmark is intended purely for academic research and evaluation, and does not involve animal subjects, medical data, or applications in high-risk domains. We carefully considered potential societal impacts, including risks of misuse, and aim to provide the community with transparent and rigorous tool for assessing VLMs' capabilities in immersive environments, while promoting responsible stewardship of Al research.
REPRODUCIBILITY STATEMENT
We have made efforts to ensure the reproducibility of our work. The main paper provides a detailed description of the construction process of ODI-Bench (Section 3), including task definitions and evaluation settings (Section 4.1), and clearly outlines the design of the proposed Omni-CoT frame- work (Section 5.1). In Appendix E, we provide the core prompts used in our experiments to facilitate replication. The dataset construction pipeline, annotation protocol, and evaluation methodology are all described in detail in the main text and supplementary materials. Together, these resources aim to provide sufficient transparency and guidance to reproduce our experimental results. We also plan release the dataset and code upon publication.
REFERENCES
Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, et Intern-sl: scientific multimodal foun- dation model. arXiv preprint arXiv:2508.15700 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rXi::2502.13225, 2025b.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Shih-Har Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun, and Ming-Hsuan Yang. Visual question answering 360deg images. In Proceedings of the IEEE/CVF Winter Conference Applica- tions of Computer Vision (WACV), pp. 1607-1616, 2020.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06266 2025.
Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm xxcoposee: Mastering free- form text-image composition and comprehension in vasion-language large model. arXiv preprint arXiv:2401. 16420, 2024.

---

Published as a conference paper at ICLR 2026
Zihao Dongfang, Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Danda Pani Paudel, Luc Van Gool, Kailun Yang, and Xuming Hu. Are multimodal large language models ready for omnidirectional spatial reasoning? arXiv preprint arXiv:2555.1190, 2025.
Huiyu Duan, Qiang Hu, Jiarui Wang, Liu Yang. Zitong Xu, Lu Liu, Xiongkuo Min, Chunlei Cai, Tianxiao Ye, Xiaoyun Zhang, et al. Finevq: Fine-grainet user generated content video quality assessment. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025.
pp.
Team GLM, Aohan Zeng. Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang. Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang. Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang. Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- glm-130b to glm-4 all tools, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127. 2024.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions., 2024.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03332 03326. 2024a.
Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, al. Viewspatial-bench: Evaluating multi- perspective spatial localization in vision-language models. arXiv preprint 25050 2025.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-nntelleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint 1ixXi:24407.078955 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296-26306, 2024a.
Yang Liu, Ming Ma, Xiaomin Yu, Pengxiang Ding, Han Zhao, Mingyang Sun, Siteng Huang, and Donglin Wang. Ssr: Enhancing depth perception in vision-language models via rationale-guided spatial reasoning. arXiv preprint arXiv:2505. 124488 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? In European Conference Computer Vision (ECCV), pp. 216-233. Springer, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseck-vl: towards real-world vision- languaee understanding. arXiv preprint arXiv:2403.05522 2024.
OpenAl.
o3 and 04-mini system
card. 7789-800b--7758930 pdf, 2025. Accessed: 2025-08-28.
https://cot
openai wttp///p////21//50oooo -o--ando--miii-yyyttee
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin. Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024.
12

---

Published as a conference paper at ICLR 2026
Jihan Yang, Shusheng Yang. Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 10632-10643, 2025a.
Liu Yang, Huiyu Duan, Yucheng Zhu, Xiaohong Liu, Lu Liu, Zitong Xu, Guangji Ma, Xiongkuo Min, Guangtao Zhai, and Patrick Le Callet. Omni 2: Unifying omnidirectional image generation and editing in an omni model. arXiv preprint arXiv:2504.11337, 2025b.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 2023.
Xu Zheng, Chenfei Liao, Ziqiao Weng, Kaiyu Lei, Zihao Dongfang, Haocong He, Yuanhuiyi Lyu, Lutao Jiang. Lu Qi Li Chen, al. Panorama: The rise of omnidirectional vision in the embodied al era. arXiv preprint (xXi:2559.129880 2025.
Yikang Zhou, Tao Zhang, Dizhe Zhang. Shunping Ji, Xiangtai Li, and Lu Qi. Dense360: Dense understanding from omnidirectional panoramas. arXiv preprint arXiv:2506.1444 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 250::25044.04799 2025.
13


Reviews:
[{"review_id": "YnR8ZR4h8b", "reviewer": "Reviewer_jfZu", "summary": "This paper introduces ODI-Bench, a benchmark designed to evaluate the spatial and reasoning capabilities of multimodal large language models (MLLMs) in immersive omnidirectional environments. The benchmark covers 10 fine-grained tasks across 2,000 images with over 4,200 QA pairs. The authors further propose Omni-CoT, a training-free chain-of-thought prompting framework that decomposes reasoning into multiple stages. Experiments on a wide range of MLLMs are performed.", "questions": "The questions are listed above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed benchmark for ODI is timely.\n2.\tThe paper is generally easy to follow and polished.\n3.\tThe results are promising with the proposed Omni-CoT.", "weaknesses": "1.\tThe dataset scale is somewhat limited. Could the diversity of ODI-Bench cover the real-world scenes? \n2.\tThe benchmark relies heavily on automatic template-based question synthesis, which may restrict linguistic diversity and introduce annotation bias.\n3.\tCould the authors provide ablation studies to show the effects of viewpoint, crop, and refinement stages? It is suggested to provide more hyperparameter ablation to provide more insights.\n4.\tThe comparison focuses only on MLLMs. Could the authors compare with the method that first reconstructs 3D, followed by evaluation by 3D-aware LLM methods?\n5.\tThe authors should clarify the data licenses. \n6.\tFigure 1 is confusing, especially the upper right figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o90mKGJ2zM", "reviewer": "Reviewer_RHjD", "summary": "This paper addresses the gap in evaluating the ability of Multi-modal Large Language Models (MLLMs) to understand Omnidirectional Images (ODIs) and constructs ODI-Bench, the first comprehensive benchmark for this task. The benchmark comprises 2,000 high-quality omnidirectional images and over 4,000 manually annotated question-answering (QA) pairs, covering 10 fine-grained tasks. It supports both close-ended and open-ended evaluations, enabling a thorough assessment of MLLMs’ general-level and spatial-level understanding of ODIs. Experiments on 20 representative MLLMs reveal significant shortcomings in current models’ ability to comprehend immersive ODI environments. To tackle this, the paper proposes Omni-CoT, a training-free framework that enhances MLLMs’ ODI understanding through step-by-step reasoning—including viewpoint-guided answering, crop cue grounding and refinement, and response refinement—with its effectiveness validated across multiple models.", "questions": "See weaknesses \"Evaluation of Reasoning Efficiency Is Lacking\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Benchmark Construction Fills a Critical Domain Gap: ODI-Bench addresses key flaws of existing ODI benchmarks (e.g., low resolution, limited scene diversity, constrained question domains) by providing high-resolution images, covering both indoor and outdoor scenes, and designing diverse tasks. It adopts a hybrid annotation approach (automated pipeline + human verification) to ensure data quality, serving as a unified, reliable benchmark for evaluating MLLMs’ ODI understanding and promoting standardized research in this field.\n\n2. Comprehensive Evaluation Dimensions and Rigorous Experimental Design: For the first time, the paper employs both close-ended (multiple-choice/yes-no) and open-ended evaluation settings. This dual design not only assesses models’ recognition accuracy under constrained options but also measures their generative reasoning ability in unconstrained scenarios. Experiments cover 20 MLLMs of varying types (proprietary/open-source) and parameter scales, with additional baselines (Blind GPT-4o, random choice) for comparison. In-depth result analysis effectively reveals the challenges MLLMs face in ODI understanding.\n\n3. Innovative and Practical Training-Free Enhancement Framework: Omni-CoT targets MLLMs’ insufficient comprehension of immersive ODI environments by introducing a human-like step-by-step chain-of-thought strategy. It guides models to interpret ODI scenes via compact textual prompts (instead of additional image inputs) and refines reasoning using crop cues, avoiding the high resource consumption of training-based methods. The framework demonstrates strong versatility, achieving performance improvements on both proprietary and open-source models.", "weaknesses": "1. Stepwise Ablation of Omni-CoT’s Reasoning Stages Is Insufficient: Existing experiments validate the overall effectiveness of Omni-CoT but fail to disassemble and analyze the individual contributions of its three core steps (viewpoint-guided answering, crop cue grounding and refinement, response refinement). For example, it remains unclear how much each step independently improves performance on spatial-level tasks, or whether crop refinement (a key sub-step) effectively filters out irrelevant cues. Supplementing stepwise ablation experiments will help clarify the role of each component and strengthen the framework’s interpretability.\n\n2. Evaluation of Reasoning Efficiency Is Lacking: Omni-CoT enhances performance through multi-step reasoning but does not report the increase in inference time compared to direct answering. It is recommended to add quantitative analysis of inference efficiency—such as comparing Omni-CoT with direct answering and Zero-shot CoT in terms of average reasoning time per sample—to balance performance gains against time costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d88b5lrQey", "reviewer": "Reviewer_FDum", "summary": "This paper makes two main contributions:\n\n1.\tODI-Bench, an omnidirectional image question-answering (QA) benchmark for multimodal large language models (MLLMs), consisting of 2,000 omnidirectional images and over 4,000 manually annotated QA pairs.\n2.\tOmni-CoT, a training-free method designed to enhance MLLMs’ comprehension ability on omnidirectional image QA tasks.\n\nThe authors demonstrate that both open-source and proprietary MLLMs still struggle with reasoning and understanding in omnidirectional settings. The proposed Omni-CoT method improves performance by cropping and wrapping ODI images from multiple viewpoints before feeding them into the models. Experiments show that this approach consistently enhances MLLM performance across different architectures.", "questions": "* Line 355: Please clarify what is meant by “absolute directions”. Does it refer to directions with respect to a global (earth-fixed) frame, or relative to the ego’s orientation in the scene?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is clearly written and well structured, making it easy to follow.\n* The dataset is carefully designed and systematically organized.\n* The evaluation of MLLMs is comprehensive and covers a wide range of models.\n* The related work section provides a thorough and insightful overview of prior research.\n* The proposed Omni-CoT method is simple yet effective, demonstrating consistent improvements across various models.", "weaknesses": "* Limited technical novelty of Omni-CoT: While Omni-CoT is effective, its core idea mainly involves viewpoint decomposition and prompt-based aggregation, which may be seen as a straightforward extension of existing multi-view prompting techniques.\n    * Clarification: In lines 396–402, the authors discuss the drawbacks of directly splitting ODIs and feeding them into the model. Could the authors clarify how this baseline differs from Omni-CoT? Is Omni-CoT’s improvement primarily due to its CoT reasoning structure, or due to the view cropping itself? An ablation that isolates these effects would significantly strengthen the paper.\n* Dataset scale and reliability: With only 2,000 images and ~4,000 QA pairs, the benchmark is relatively small for evaluating large-scale models. The results may be statistically unstable\n    * I would suggest reporting mean ± standard deviation over multiple runs or random seeds to quantify evaluation noise and ensure reproducibility.\n* Broader impact and future directions: The paper could benefit from a brief discussion on how ODI-Bench might be used for training, not just evaluation — for instance, as a pretraining or fine-tuning resource for spatial reasoning in immersive environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qS7jTmdrxJ", "reviewer": "Reviewer_Jjv6", "summary": "This paper introduces ODI-Bench, a 360°×180° omnidirectional-image benchmark for VR/AR/embodied settings: 2,000 ODIs, 4,000+ human-annotated QA pairs, 10 fine-grained tasks covering general semantics and spatial reasoning, with both close- and open-ended evaluation. Benchmarking 20 proprietary/open-source MLLMs shows they underperform on ODIs despite strong 2D results, indicating poor immersive/spatial understanding. The authors propose OmniCoT, a training-free chain-of-thought procedure that fuses textual and visual cues, yielding substantial accuracy gains across ODI tasks. Benchmark and code will be released upon publication to catalyze research in panoramic scene understanding.", "questions": "See weaknesses.\n\nBesides, authors claim the benchmark are high-resolution images. However, in recent paper [1, 2], the researchers discussed that in most general scenarios, even simple resizing can achieve strong performance, do not need such high resolution image. How do the authors view this issue? I look forward to some discussion on this point.\n\n\n[1] VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning\n\n[2] Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The topic is timely and relevant.\n2. The manuscript is clearly written and well structured.\n3. The proposed method Omni-COT delivers consistent, meaningful improvements.", "weaknesses": "1. In Table 1, the authors name the proposed benchmark \"360Bench.\" For consistency with the main paper, I recommend renaming it to \"ODI-Bench.\"\n\n2. ODI-Bench focuses on 360-degree views, which the authors claim benefits AR/VR. I’m not fully convinced. Human field of view is roughly ~180 degrees; in AR/VR, users still turn their heads/bodies to see other views. Is it necessary to require models to reason directly on a single 360-degree image?\n\n3. The ODI-Bench images are 360-degree panoramas re-projected to 2D. A concern is that most current models were not trained on such projections and may treat them as \"wrapped images,\" causing train–inference mismatch and errors.\n\n4. As a human evaluator, my answer in Figure 4 would also be \"No.\" The image seems confusing for both VLMs and people. Without seeing Appendix Figure 1 (which reveals that the far left and far right edges are adjacent—i.e., the \"back\" region), I would likely misinterpret it.\n\n5. When benchmarking, consider giving VLMs minimal necessary priors, e.g., \"This is a 360-degree panoramic (pano) view image,\" to reduce avoidable misunderstandings. I have asked GPT-4o for the Appendix Fig. 6 question: \"This is a 360 degree pano view image. Standing under the shelter facing the railway tracks, where is the train in relation to me? A. Behind;B. Right;C. Left;D. Front\" The model could correctly solve this question.\n\n6. I’m curious whether performance improves if the panorama is split into multi-view images or converted into a short continuous video and then fed to the VLM, followed by inference on the benchmark questions (not viewpoint guiding—just direct inference).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)