Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning
Authors: Minheng Ni
YuTao Fan
Lei Zhang
Wangmeng Zuo
Venue: ICLR 2025 Poster (2025)
Keywords: Understanding ambiguous instructions
large multimodal model
chain-of-thoughts
multimodal
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose Visual-O1, a multi-modal multi-turn reasoning framework that enhances high-intelligent and general-intelligent models' understanding of ambiguous instructions in multi-modal tasks by simulating human reasoning.

Abstract: As large-scale models evolve, language instructions are increasingly utilized in multi-modal tasks. Due to human language habits, these instructions often contain ambiguities in real-world scenarios, necessitating the integration of visual context or common sense for accurate interpretation. However, even highly intelligent large models exhibit observable performance limitations on ambiguous instructions, where weak reasoning abilities of disambiguation can lead to catastrophic errors. To address this issue, this paper proposes Visual-O1, a multi-modal multi-turn chain-of-thought reasoning framework. It simulates human multi-modal multi-turn reasoning, providing instantial experience for highly intelligent models or empirical experience for generally intelligent models to understand ambiguous instructions. Unlike traditional methods that require models to possess high intelligence to understand long texts or perform lengthy complex reasoning, our framework does not notably increase computational overhead and is more general and effective, even for generally intelligent models. Experiments show that our method not only enhances the performance of models of different intelligence levels on ambiguous instructions but also improves their performance on general datasets. Our work highlights the potential of artificial intelligence to work like humans in real-world scenarios with uncertainty and ambiguity. We release our data and code at https://github.com/kodenii/Visual-O1.

Published as a conference paper at ICLR 2025
VISUAL-01: UNDERSTANDING AMBIGUOUS IN- STRUCTIONS VIA MULTI-MODAL MULTI-TURN CHAIN-OF-THOUGHE REASONING
Minheng Nit:2, Yutao Fan", Lei Zhangle & Wangmeng Z DDpartment of Computing, Hong Kong Polytechnic University minheng. ni@connect polyu.hk csizhang@ pol.u..polu..omm edu. hk Faculty of Computing. Harbin Institute of Technology *Pengchnng Laboratory fanyutacsstu. cn wmzuo@hit .ddu. en
ABSTRACT
As large-scale models evolve, language instructions are increasingly utilized in multi-modal tasks. Due to human language habits, these instructions often con- tain ambiguities in real-world scenarios, necessitating the integration of visual context or common sense for accurate interpretation. However, even highly in- telligent large models exhibit observable performance limitations on ambiguous instructions, where weak reasoning abilities of disambiguation can lead to catas- trophic errors. To address this issue, this paper proposes VISUAL-O1, a multi- modal multi-turn dhain-of-thought reasoning framework. It simulates human multi- modal multi-turn reasoning, providing instantial experience for highly in- telligent models empirical experience for generally intelligent models to derstand ambiguous instructions. Unlike traditional methods that require models possess high intelligence to understand long texts perform lengthy complex reasoning, our framework does not notably increase computational overhead and is more general and effective, even for generally intelligent models. Experiments show that our method not only enhances the performance of models of different intelligence levels on ambiguous instructions but also improves their performance general datasets. Our work highlights the potential of artificial intelligence to work like humans in real-world scenarios with uncertainty and ambiguity. We release our data and code at http:///thhbbcoom////dd//////all
INTRODUCTION
With the advancement of deep learning, increasing attention is being paid multi-modal scenarios that are more relevant to reality, such as visual question answering (VQA) (Antol et al. 2015 Zhou
al. 2020 Teney et al.] 2018), referring image segmentation (RIS) (Lai al. 2023 Ni al.
2023
Yang ct al. (023). ) and vision-and-languaee navigation (VLN) & Bansal 2023 Hong et al. 2021
Feng al. 2022). In recent years, more manipulable language instructions are gradually being
introduced into these tasks to better align with human interaction habits. Combined with large-scale
language models (OpenAI, 2024 Chen al. 2023a Liu al. 2024), artificial intelligence (AI)
beginning to use language instructions closer to real- weal-world scenarios to perform tasks, broadening
the range Al applications..
As shown in Figure [ due to the inherent ambiguity of natural language and the excellent analytical abilities of humans, the language instructions used by humans often contain vagueness and ambi- guity. Additionally, human language and vision closely related and often require combining visual context or common sense to accurately understand their meanings. Therefore, ambiguous instructions. which are common in reality, differ from meticulously designed accurate instructions (Antol et al. (015)), and learning to understand them directly becomes challenging in the absence of corresponding task data.
Recently, ch of-thhhghtts (CoT) reasoning has greatly enhanced the understanding and analytical capabilities of high-intelligent large models, such GPT-40. However, its application to scenar-

---

Published as a conference paper at ICLR 2025
Homan
High -inelliieegen Modal
General-ineeliger Model
Ficoso find that one itaan!!!
Mult-errdel Muli-tur Colaa
29997
that.on Aamm imgiine on sigeen ther notin ciase pricerity
uction
Instantaal Exp
Anewer
let chect the mage Hounta achimosttaicceee and
group
Muttimoodil Muti-turn Calls
The Indlaton ELAS nor some whool insiireet
Ecamples coseonooee
Erepiical Fat
Biaal a opar to the dittament stsohans So. the olenbent
hnrration
Answaar
(a) Ambiguuus Instruction
Ambiguity
(d) Ambiguity Understanding by Visual-O1
Understanding ambiguous instruction. The Al model may not be able to execute in- structions normally when encountering ambiguous instructions. However, humans can usually cor- rectly analyze the actual meaning of ambiguous instructions by combining visual context and can accurately interpret ambiguous instructions. Based on this, propose VISUAL-O1, which sim- ulates human multi-modal multi-turn reasoning to gain instantial (for high-intelligent models) or empirical (for general-intelligntt models) experience in order to understand ambiguous instructions.
Figure 1:
involving multi-modal ambiguity understanding has yet to be explored. Additionally, general- intelligence models used in multi-modal tasks, like LLAVA, often lack the data and parameter ca- pacity to perform chain reasoning and analysis, making it difficult to apply CoT methods to enhance their understanding of ambiguous instructions.
To address these challenges, we propose VISUAL-O1, a multi-modal multi-turn chain-of-thought reasoning method that simulates human multi-modal multi-turn reasoning. VISUAL-O1 builds instance-specific experience during inference for high-intelligence models or creates general ex- perience for any ambiguous instructions through one-time optimization with several examples for general-intellignne models. This helps models correctly understand ambiguous instructions and synthesize the final answer.
We also construct a dataset containing various types of ambiguous instructions, including ellip- sis, colloquialism, subjectivity, relativity, and other, validate performance across different multi- modal scenarios. Experiments show that our method improves the performance of models with varying intelligence levels on the ambiguous instruction dataset and enhances their performance on general datasets. Ablation studies demonstrate that VISUAL-OI can be easily applied to different multi-modal models and tasks.
Our contributions are three-fold:
We reveal the capabilities of multi-modal models in analyzing and executing ambiguous instructions by setting up novel benchmark for understanding ambiguous instructions in various multi-modal tasks.
We propose VISUAL-O1. a multi-modal multi-turn ofain-of-thought reasoning method, to build instantial or empirical experience for high-intelliignt general-intelligent models, enabling them to correctly understand ambiguous instructions.
Experimental results show that our method improves the performance of models with vary- ing intelligence levels on ambiguous instruction datasets and enhances their performance on general datasets.
2 RELATED WORK
Language Instruction Understanding in Multi-modal Tasks Language instructions were often used as conditions in multi-moda tasks, serving as an essential medium for user-Al interaction. They had been applied in wide range of highly reality-related tasks (Reed et al., 2016 Bigham ct al. 2010; Zellers et al. 2019). Traditionally, language instructions were precise and unambiguous. Although some works had managed to understand and execute complex language instructions by combining large multi-modaa models (LMMs) (Lai et al. 2023 Yang et al. 2023), 3,, they still lacked practical understanding of ambiguous instructions. Recently, few works had noticed the presence of ambiguity in language within specific multi-mon tasks, In image classification, WAFFLECLIP
(Roth et al. 2023) and FUDD (Esfandiarpoor & Bach 2023) had pointed out the issue of polysemy
2

---

Published as a conference paper at ICLR 2025
in classification texts, using LLMs to generate commonalities differences within categories to enhance image classification tasks. In visual question answering. Prasad et al.] (2023) proposed REPHRASE repeatedly utilize LLMs to mine image information through manually pre-designed prompts to enhance language understanding.
However, existing methods required models engage in extensive interactions with samples based on predefined rules to mine information, requiring extremely long inference times and relying on specific tasks and models, making difficult generalize various tasks and models with different intelligence levels.
Complex Reasoning with Large Multi-modal Models In recent years, large language models (LLMs) have expanded into multi-modal scenarios (OpenAl 2024 Liu et al. 2024 Lai et al. 2023),
showcasing the potential of large multi-modal models (LMMs) (iMMsinn in handling multi-modal tasks. How- ever, challenges remain in understanding complex language instructions within visual contexts in multi-modal tasks. Recently, methods to enhance the reasoning capabilities of large multi-modal models have attracted some research attention. On one hand, some works have proposed training- based methods. Dai et al. (2024) proposed a two-stage training method that aligned pre- pre-trained models with images and texts, enhancing the capabilities of LMMs. Similarly, Chen et al. (202bb); Wang et al.] (2024) achieved impressive results in modai tasks using data generated by GPT. On the other hand, some works have proposed non-training methods, such as the chain-of-the -futn-oft (CoT) (Wei et al. [022]] Yao et al. 2024) approach, which enhances model understanding by simu- lating human reasoning.
However, existing methods either relied large amount of real or model-generater data to opti- mize model parameters, which is challenging to scale cost-effcctively to any task, or they required the model to have strong reasoning abilities, making challenging to
models.
VISUAL-O1: MULTI-MODAL MULTI-TURN CHAIN-OF-THOUGHTS REASONING FRAMEWORK
Isference Stage
One-dime Optimizatizn Stage
Inforence Stage
Reassring
Ressoring
Regpants
Zlaa
Systhoo
High-inddiiper
Seflccion
Mali-tare Cafe
Gemer rndiggert Moddl
Roflcctinn
Pulli roold
Mus
Genanal-ttilligatt
Reppanss
Who ditadd the sarre?
Whathha et
STAR
Sythhoo
Anneer
(n) Vituat-O1 for High-intelligent Model with Instantial Experience
(b) Visual-D3 for General-inbellggen Model with Empirical Experience
Figure 2: The overview of VISUAL-O1. VISUAL-O1 introduces multi-modal multi-turn chain- of-thoughts to understand ambiguity with (a) instantial experience for high-intelligent models to generate the correct answer directly, and (b) empirical experience for general-inteligent models to transform ambiguous instructions into clear instructions and then generate the correct answer. Instantial and empirical experience develops during inference and one-time optimization stage.
3.1 OVERVIEW OF PROPOSED FRAMEWORK
Given a task that requires both a visual context x, and an ambiguous instruction Xa. can be performed by a multi-modal model f. The result of the task is denoted as f(xa, x, 0y), where f represents the parameters of the task mode!" Due to the ambiguous or unclear information contained in the text instruction X,, the performance of the model f can be largely affected. If this model is general-intellignnt model, this can have catastrophic effects due to its weaker reasoning abilitics, making it nearly impossible for the model to produce the correct result. Even for high- intelligent large multi-modal models, still observe performance limitations (see Tables and for more details).
Since do optimize 0,, we omit it in the following text for brevity
3

---

Published as a conference paper at ICLR 2025
To address this issue, shown in Figure 2 we propose VISUAL-O1, multi-modal multi-turn chain-of-thon -fthuuhhs reasoning framework. VISUAL-O1 aims to help models build experience in dis- ambiguating ambiguous instructions, enabling them to correctly understand the instructions and pro- duce the right answers. Unlike traditional chain-of-thougg methods that require models to possess high intelligence for understanding long texts and performing complex reasoning, framework is models.
more general and effective even for
Specifically, for high-intelligent models, VISUAL-O1 establishes sample-level disambiguation expe- rience during inference by leveraging the ambiguous instructions themselves. For general-intelligen models, VISUAL-O1 builds empirical disambiguation experience during one-time optimization us- ing several examples. VISUAL-OI synthesizes responses by either summarizing the sample-level experience into the final answer or using the empirical experience to transform ambiguous instruc- tions into clear instructions. This clear instruction is then combined with the original ambiguous instruction to summarize the final answer. Despite the slight differences in Visual-O1 for high- intelligent and general-intelligent models, their structures are completely consistent. Therefore, we can formulate them within a unified framework.
3.2 REASONING AND REFLECTION
Given multi-modal model f. by utilizing the visual context and logical reasoning, we can supple- ment vague information, enabling the model to understand the precise meaning of the ambiguous instruction. To achieve this, perform reasoning step under the guidance of pre-defined prompt Pran to attempt to understand the ambiguous instruction:
Xyen
f(xa X.
Preni A),
where Xysn the reasoning result, and A is special prompt that contains the disambiguation expe- rience. Our goal is to obtain this special prompt A to help f understand the ambiguities.
Assuming we complete one reasoning step, next, the model reflects on the reasoning result X Assuming Pra is the predefined prompt, we obtain the reflection result x,a by:
Xxfl f(A, Xran Pra).
By utilizing reflection, we summarize the issues in the reasoning process to improve the disambigua- tion experience A. By repeatedly performing the reasoning-reflection process, we gradually refine the reasoning results, transforming it into an iterative form. We rewrite the reasoning and reflection steps in iterative form follows:
f(xa; Ny Prsm; A (0),
(3)
:a f(A(1, x(() Pea).
(3)
(4)
where i is the i-th iteration. Assuming our budget is N, we directly apply the iterations formed by Eq. (3(4) and an update function of A by N times to obtain the final A == A Next, we introduce the instantial and empirical experience of A and their updating mechanism in Visual-O1 for different intelligence levels of the model.
Reasoning and Reflection for Instantial Experience For high-intelligent models, in each iter- ation, VISUAL-O0 continues to analyze the ambiguous instruction based on the previous disam- biguation process, then reflects on its analysis and generates feedback. These analyses and feedback form the reasoning process for the entire problem. Therefore, for the i-th iteration, we update Ains by aggregating all information:
(6+1)
where is the concatenation function. Obviously, this process can directly work on the inference stage. Assuming our budget is Nins. we directly apply the iterations formed by Eq. (3 Nins times during the inference stage to obtain complete disambiguation process.
Reasoning and Reflection for Empirical Experience For general-intelligent models, in each it- eration, VISUAL-O1 uses the existing experience to continue attempting transform ambiguous
4

---

Published as a conference paper at ICLR 2025
instructions into clear ones, then reflects on its conversion process and updates the experience. Due to the limited understanding capabilities general intelligent models for long texts, we only retain the new reflection information the update for Aempi
(1+1)
(6)
Similar to the process of nigh-intelliggen model, we apply the iterations formed by Eq. Nemp times to obtain the final disambiguation experience. The difference is that we change one sample in each iteration, which means a total of Nemp samples, to prevent a learning experience that is specific to certain samples rather than generalizable experience. Fortunately, we only need to obtain Aemp once in the one-stage optimization stage; for the same task, we no longer need to repeatedly obtain Aemp-
3.3 RESPONSE SYNTHESTS
Although experience not the final answer, it contains the model's understanding of the instruc- tions and is important information for solving the problem. Next, obtain the final answer y through response synthesis.
Response Synthesis by Instantial Experience For high-intelligent models, Ains already encom- passes all the reasoning, and the model possesses strong comprehension abilities. Therefore, we simply summarize the entire reasoning process with a pre-defined response synthesis prompt Payn to directly obtain the final answer:
(Aiins
Psyn);
Response Synthesis by Empirical Experience For ceneral-intelligenn models, considering their limited reasoning capabilities, first the disambiguation experience Aemp learned by the model during the one-time optimization process to transform the ambiguous instruction Xa into a clear instruction x. based the visual context X,:
x = sxxa; X2 Aemp).
Then, we utilize the pre-defined response synthesis prompt Psyn, allowing the model to summarize the original ambiguous instruction and the clear instruction together to obtain the final answer:
y fXe: Na.
Psyn)-
Finally, regardless of the model's level of intelligence, obtain final answer y that well- understands the ambiguous instruction x, based on visual context X
3.4 IMPLEMENTATION DETAILS
We experiments 10 random seeds obtain average results. We set budgets Nims and Nemp to 10 and 3 for general-intelligen intelligent and high tighlintelligeett models respectively. Please note that due to the difficulty of performing reflection with general-intelligent models, in practice, a high-intelligent model at this step. Since the reflection for general-intlligent models is only con- ducted during one-time optimization, our final inference still relies solely on the general-intelligenn model itself without introducing additional models or computational overhead.
temporarily
For more details of implementation, please refer the Appendix A
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
To verify the effectiveness of our method, conduct series of experiments. We apply VISUAL- O1 to the state-of-the high-intelligence model and general-inteliggnne model on two typical multi-modal tasks: referring image segmentation (RIS) and visual question answering (VQA). All comparisons are divided into ambiguous instructions and general instructions to comprehensively evaluate the models' performance ambiguous and non-ambiguous instructions.
5

---

Published as a conference paper at ICLR 2025
Table 1: Overall results on RIS. VISUAL-O1 outperforms other methods and achieves notable improvements on both high-intelligent and general-intelligee models. We report the average results on different random seeds. The improvement statistically significant with p < 0.01 under t-test.
AMBIGUOUS
GENERAL
MODEL
GloU
cloU
GloU
cloU
LISA LALET AL.. 2023 SOM
0.0237
0.0272
0.4654
0.4721
ET AL.,
2023
0.0752
0.1158
0.3507
0.4154
CHAIN-OF-THOUGIH (WEI ET AL.
2022)
0.0746
0.0982
0.4183
0.4326
FtDD
[ESFANIARPOOR & BACH
2023
0.0718
0.0928
0.3416
0.3534
VISUAL-OT (LIIA))
0.1131
0.1215
0.4738
0.4980
VISUAL-O1 (SoM)
0.1304
0.1756
0.3686
0.4508
Table 2: Overall results VQA. VISUAL-O1 outperforms other methods and achieves notable improvements on both ambigous and general datasets. We report the average results on different random seeds. The improvement is statistically significant with 0.01 under t-test.
AMBIGUOUS
GENERAL
MODEL
Acc
BLEU-1
Acc
BLEU-1
LLAVA (LIU ET AL. 2024
8.58
0.4760
54.19
0.7194
GPT-40 OPENAT 2024)
14.10
0.5960
61.25
0.8192
CHAIN-ON SAAIN-OF-THEE THOUGHTS (WELET AL.
2022
15.03
0.3372
39.78
0.5113
FUDD (ESPANDIARPOOR BACH
2023
19.18
0.4841
48.37
0.64554
VISUAL-O1 (LLAVA)
22.13
0.5095
57.58
0.7295
VISUAL-O1 (GPT-40)
22.78
0.6640
63.14
0.8482
Baselines and Evaluation Metries. We select series of typical methods as baselines. We choose
the original, unprocessed ingg-intellieence SoM Yang et al. 2023) and GPT-4o (openAl|20224 as well as the ceneral-intelligennne LISA (Lai et al. 2023) and LLAVA (Liu et al., 2024) as comparison
models for RIS and VQA. Then, we apply CHAIN-OF-THOUGHTS (Wei al., (022) reasoning and FUDD (Estandiarpoor & Bach 2023) language explanation methods on LISA and LLAVA for further comparisons. To ensure fairness in the comparisons, we ensure that all models are under the same settings. Following previous work (Bigham et al. 2010 Kazemzadeh et al. 2014, Ni et al.
2023 for RIS, we compare gloU and cloU, and for VQA, we compare accuracy and BLEU-1.
Datasets. In addition to the general datasets, we set up extra ambiguous datasets for RIS and VQA, which only contain ambiguous instructions filtered or refined manually. We categorize the ambiguous instructions into five types based the cause of ambiguity: ellipsis, colloquialism, subjectivity, relativity, and other. Ellipsis indicates that the ambiguity stems from omitted content; colloquialism indicates that the ambiguity arises from the of informal or imprecise expressions; subjectivity indicates that the ambiguity is due to subjective judgments; relativity indicates that the ambiguity comes from implied comparisons, and other represents other types of ambiguity.
For more details and analysis of the dataset, please refer to the Appendix B and C
4.2 OVERALL RESULTS
Ambiguous instructions understanding. As shown in the left part of Tables and 2 we observe that the original model performs poorly on the distinctly different tasks of VQA and RIS. This indicates that existing methods are susceptible to ambiguous instructions and do not perform well when correct visual context needs to be integrated to understand instructions. After integrating VISUAL-O1, the performance of all tasks shows a notable improvement. This means that existing methods have the potential to understand ambiguous instructions but require proper guidance. It worth mentioning that we observe that using VISUAL-O1, general-intellige intelllgentt model, LLAVA, and LISA capable of achieving results comparable to even better than high-intelligent models without introducing any additional models or data during inference. This further demonstrates the value of VISUAI.-O1.

---

Published as a conference paper at ICLR 2025
Table Ambiguous instruction results on different intelligence level. VISUAL-O1 has demon- strated powerful flexibility across different levels of intelligence.
MODEL
VISUAL-O1
GloU
CloU
LISA EA AL.
2023)
0.0237
0.0272
VISUAL-OT (LISA)
EMPIRIALL
0.1131
0.1215
VISUAL-O1" (LISA)
INSTANTIAL
0.1447
0.1351
SOM (YANG ET AL.
2023)
0.0237
0.0272
VISUAL-OT (SOUA
EMPIRICAL
0.1143
0.1530
VISUAL-OI (SoM)
INSTANTIAL
0.1304
0.1756
General instructions understanding. While the ability to understand ambiguous instructions is crucial, the performance the model on general datasets is also important. As shown in the right part of Tables [ and 2 we also find that VISUAl-OI enhances the understanding of ambiguous instructions and greatly improves performance on general datasets. This is because improving the ability to understand instructions is beneficial even for those that do not contain ambiguity. What even more noteworthy is that traditional COT and description- -based synthesis like FuDD greatly degrade the model's 's performance on general datasets. This is because ceneral-intelligen models like LLAVA find it challenging to maintain the logical chains and long-text comprehension akin to high- intelligent large models like GPT-4o. Complex reasoning tends to impair the model's understanding capabilities.
For more analysis on improvement, please refer to the Appendix D
4.3
GENERALIZABILITT STUDIES
4.3.1
FLEXIBILITY TO INTELLIGENCE LEVELS
Based on the strong long-text comprehension and reasoning abilities of GPT-4v's SoM, we use VISUAL--11 for instantial experience during the inference stage for each sample. In contrast, due to LISA'S weaker reasoning abilities, we utilize VISUAL.-O1 for empirical experience during the one-time optimization phase, avoiding complex reasoning for each sample during the inference stage. However, this does not mean that VISUAL-OI cannol perform empirical experience for high- intelligence models or instantial experience for general-inter ceneral-ntelliience models. To test VISUAL-O1 adaptability to different intelligence levels, design additional models: VISUAL- SEDUALO11 -1 (LISAA)) us- ing instantial experience and VISUAL-O1 (SoM) using empirical experience.
As shown in Table the variants of VISUAL-O1 thy that since LISA cannot comprehend and generate long texts in logical chains, we use GPT-4V as its disambiguation model during the inference stage to supplement its capabilities. Whether applied to high-intelligent models or general-intellggent models, the variants of Visual-O1 largely enhance performance on ambiguous instructions.
effective across different models. is notewor-
For more analysis on intelligence levels, please refer to the Appendix E
4.3.2 ADAPTABILITY TO MODELS
Can VISUAL-O1 generalize to different models? We select various models on RIS to verify this, and as shown in Table regardless of the model used, we observe stable improvements. This highlights the generalizability of VISUAL-O1 to different models and suggests the potential for broader applications of our method.
For more analysis adaptability different models and tasks, please refer to the Appendix
4.4
ABLATION STUDIES
4.4.1
EFFECTIVENESS OF COMPONENTS
To further explore the effectiveness of VISUAL- 01, we conduct ablation experiments on the general VQA data in Table Reasoning and reflection are crucial for the model correctly understand

---

Published as a conference paper at ICLR 2025
Table 4: Generalization studies on different models. VISUAL-O1 is effective across different
models. "-" indicates values not reported in the original paper.
MODEL
GlOU
cloU
OVSEG LIANG ET AL
2023
0.0418
0.0778
REF-DIFF (NT ET AL.
2023)
0.3301
0.3140
UNIFIED-IO LL
2022
0.4015
INSTRUCTDIFFUSION GENG ET AL.
2023
0.3904
SOM YANG ET AL.
2023
0.3507
0.4154
LISA LAT AL.
2023
0,4654
0.4721
GPT-40 OPENAL
2024
0.5728
0,5152
QWEN- V L-22 WANG
2024
0.4417
0.3220
VISUAL-OI (SOM)
0.3686
0.4508
VISUAL-OI (LISA)
0.4738
0.4980
VISUAL-O1 (GPT-40)
0.5777
0.5325
VISUAL-O1 (QWEN-VL-2)
0.5161
0.4457
Table 5: Ablation studies. Each module of VISUAL-O1 plays an indispensable role.
MODEL
Acc BLEU-1
VISUAL-O1 (LLAVA)
57.58
0.7295
W/O RESPONSE SYNTHESIS
47.59
0.6589
W/o REASONING AND REFLECTION
54.25
0.7076
VISUAL-O1 (GPT-40)
63.14
0.8482
w/O RESPONSE SYNTHESIS W/O REASONING AND REFLECTION
60.00
0.8254
60.60
0.7943
Table 6: Efficiency of different budgets for VISUAL-O1. We report the average accuracy of 10 experiments, as well as their maximum, minimum, and variance.
AMBIGUOUS
GENERAL
VISUAL--11
AVG
MAX
MIN
VAR
AVG
MAX
MIN
VAR
W/O BUDGET
8.95
10.14
7.89
0.85
54.02
54.90
53.30
0.35
W/ HUMAN EXP
16.88
18.43
14.29
1.61
54.55
56.20
53.15
1.02
W/ BUDGET
17.83
19.73
16.76
0.96
55.38
56.22
54.10
0.45
W/ BUDGET
19.92
21.89
18.77
0.95
56.31
56.92
541.6
0.58
BUDGET
22.13
24.52
20.91
57.58
58.56
56.76
0.35
W/ BUDGET
18.82
20.54
17.30
0.84
55.42
56.34
54.36
0.34
w/ BUDGET
19.85
21.35
17.94
0.95
56.17
57.44
55.14
0.46
ambiguous instructions. The model's performance on ambiguous dataset declines noticeably without reasoning and reflection. Meanwhile, response synthesis is very important for the performance on regular data from non-ambiguous instructions, as the original instruction may contain important information, and response synthesis ensures the complete transmission of this information. Every module of VISUAL-O1 effectively improves the model's performance across different datasets.
For more explorations of the model, please refer the Appendix
4.4.2 EXPLORATION OF REASONING
We design additional validation experiments further prove the effectiveness of VISUAL-O1 rea- soning process. We invite volunteers to manually design disambiguation empirical experiences, which are used as benchmarks for ten tests and averaged. As shown in Table designed prompts are less effective than reasoned empirical experience, demonstrating the rational- ity of VISUAL-O1 O1'ss design.
T shows that manually
We also show the influence of budgets in the reasoning and reflection process of VISUAL-OI on 10 experiments with different random seeds. However, with the reasoning and reflection process, the
8

---

Published as a conference paper at ICLR 2025
Table 7: Comparison with data augmentation. Compared resource-intensive data augmenta-
tion, VISUAL-O1 still achieves best performance.
METHOD
Acc
BLEU-1
LLAVA W/ AMBIGUOUS EXTRA DATA
51.48
0.6863
LLAVA W/ AMBIGUOUS ORIGINAL DATA
45.08
0.6363
LLAVA W/ NOISED ORIGINAL DATA
39.84
0.5912
VISUAL-O1 (LLAVA)
57.58
0.7295
performance gradually grows and then falls again after reaching its peak. We observe that VISUAL- O1 follows a trend similar to deep learning. The performance begins to grow the early stages of learning, but then decreases due to overfitting after reaching a particular stage.
For more explorations of reasoning process and budget > 3. please refer to Appendix Handd]
4.4.3 COMPARISON OF DATA AUGMENTATION
We generate an additional 10,000 data samples for fine-tuning LLAVA, as shown in Table 7 We design three different data augmentation methods: (1) AMBIGUOUS EXTRA DATA: directly using extra 10,000 sets of ambiguous instruction data; (2) AMBIGUOUS ORIGI- NAL DATA: rewriting 10,000 sets of original instructions into ambiguous instructions, then training the model; and (3) NOISED ORIGINAL DATA: randomly deleting or modifying 10,000 sets of orig- inal instructions, then training the model.
LLAVA to synthesize
We find that due to the synthetic data's 's inability effectively simulate instructions, there is large performance drop in VQA. Additionally, the annotation and training costs brought by the 10,000 data samples also very high. Therefore, VISUAL-O1 effectively solves the problem of understanding ambiguous instructions while maintaining performance on the general dataset.
For comparisons of computational cost, please refer to the Appendix J
4.5
CASE STUDIES
How does our method help the model better understand ambiguous instructions? To delve into this issue, we visualize the results of RIS in Figure 3 We find that ambiguous instructions common in the data and often not easily detected by humans, but this can severely affect the model's performance. After introducing VISUAL-O1, the accuracy of the instructions improves noticeably, leading to more effective outcomes.
As shown in the first example, this is typical case of subjectivity ambiguity, requiring the per- spective of human subjective observation for reasonable inference. The original instruction could describe most of the bears in the image, as each bear's angle has some deviation. However, when combined with the image, it is found that only one bear meets the requirements of turning and slight deviation, and VISUAL-O1 successfully infers this and provides an accurate description, enabling the task model to accurately segment the target.
In the second example, this is a typical case of ellipsis ambiguity, where many sentence components are omitted. The segmentation model is misled by the word "readinm ng" and incorrectly segments the person in the image. Meanwhile, ViSUAL.-O1 accurately describes that this bus showing "reading station" sign and even provides additional information to help the segmentation model locate the target.
In the third example, this is typical case of relativity ambiguity, where comparison is used to refer to specific entities. To understand the instruction, the model must first locate the comparison object of the bottle's s lighter color, which is the color of other bottles, that is, the lightest-colored bottle. When combined with VISUAL-O1. 1 the instruction is transformed into a more accurate form, enabling the model to easily locate the target.

---

Published as a conference paper at ICLR 2025
Image
LISA
Visual-01 (LISA)
Answer
Answer
Clear Instruction
White plush with red bow and heart on chest, positioned the right side of the image, surrounded by red plush bears.
Ambiguous Instruction
white bear turned slightly
Answer
Answer
Clear Instruction
The green and blue double-decker bus with "Reading Station" and the number "144" displayed on the front.
Ambiguous Instruction
reading station bus
Answer
Answer
Clear Instruction
The second bottle from the left in the front row, which has amber color compered the adjacent darker brown bottles
lighter
Ambiguous Instruction
botile thats next to main bottle lighter color
Answer
Answer
Clear Instruction
The giraffe positioned to upper right of feeding basket, with its head slightly raised and mouth of hay.
Ambiguous Instruction giraffe at clock
Figure 3: Case studies on RIS. Our approach aids the model in understanding ambiguous instruc- tions by incorporating VISUAL-O1, which enhances the accuracy of instructions, thereby enabling more effective segmentation. Please note that ambiguous instruction may contain typos.
In the fourth example, this is typical of colloquialism ambiguity, using common positional phrases in spoken language. The model cannot understand that "1 of clock" is a position, so cannot select the correct object, but VISUAL-O1 transforms it into clearer description.
For more cases, please refer to the Appendix
CONCLUSION
Even high-intelligent large models exhibited observable performance limitations on ambiguous in- structions, where weak reasoning abilities of disambiguation could lead to catastrophic errors. We proposed VISUAL-O1, a multi-modal multi-turn chain-of-thoughts reasoning framework. It simu- lated human multi-modal multi-turn reasoning, providing instantial experience for high-intelligent models or empirical experience for ceneral-intellignn models to understand ambiguous instructions. Unlike traditional methods that required models possess high intelligence to understand long texts or perform lengthy complex reasoning. VISUAL--01 I did not notably increase computational over- head and was more general and effective, even for general-intellignnce models. We validated our approach across various tasks and models with different intelligence levels. Experimental results demonstrated that ViSUAL-O1 substantially improved the performance of models of varying intel- ligence ambiguous instructions and enhanced their performance general datasets. Our work revealed the potential AI to operate like humans real-world scenarios characterized by uncer- tainty and ambiguity.
For statements of broader impact, limitations, and reproducibility, please refer to the Appendix
M and N
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is partially supported by the Major Key Project of PCL under Grant PCL2024A06.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. auion-and-languaae navigation: Interpreting visually-groundee navigation instructions in real environments. In Proceedings of the IEEE con- ference computer vision and pattern recognition, pp. 3674-3683, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 -2333, 2015.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time an- swers to visual questions. In Proceedings of the 23nd annual ACM symposium User interface software and technology, pp. 333-342, 2010.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for languala multi-task learning. arXiv preprint arXiv:2310.09478, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards ceneral-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Reza Esfandiarpoor and Stephen H Bach. Follow-up differential descriptions: Language models resolve ambiguities for image classification. arXiv preprint arXiv:2311.07525 2023.
Weixi Feng, Tsu-Jui Fu, Yujie Lu, and William Yang Wang. Uln: Towards underspecified vision- and-language navigation. arXiv preprint arXiv:22 10.00000, 2022.
Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al. Instructdiffusion: A generalist modeling interface for vision tasks. arXiv preprint arXiv:2309.03387 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-366 2018.
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opant and Stephen Gould, Vin bert: A recurrent annn-aand- bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 1643-1653, 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09682 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798. 2014.
11

---

Published as a conference paper at ICLR 2025
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. arXiv preprint arXiv:2308.0003 2023.
Jialu Li and Mohit Bansal. Improving vision-and-language navigation by generating future-view image semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10803-10812, 2023.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. In The Eleventh In- ternational Conference on Learning Representations, 2022.
Minheng Ni, Yabo Zhang, Kailai Feng, Xiaoming Li, Yiwen Guo, and Wangmeng Zuo. Ref-diff: Zero-shot referring image segmentation with generative models. arXiv preprint arXiv:2308. 16777, 2023.
OpenAL. Gpt-4o. https://cchtt openai. com 2024.
Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Rephrase, augment, reason: Visual ground- ing of questions for vision-language models. arXiv preprint arXiv:2310.05861, 2023.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15746-15757, 2023.
Damien Teney, Peter Anderson, Xiaodong He, and Anton Van Den Hengel. Tips and tricks for visual question answering: Learnings from the 2017 challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lssion-lanuuae model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. -hain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arX1v:2310.11441, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6720-6731. 2019.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- languge pre- pre-training for image captioning and vqa. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13041-13049, 2020.
12

---

Published as a conference paper at ICLR 2025
This appendix mainly contains:
Supplementary implementation details in Section
Supplementary dataset details Section B
Deeper analysis on dataset in Section C
Deeper analysis on improvement in Section
Further generalizability verification on intelligence Section
Further generalizability verification various tasks in Section F
Extra exploration of the model in Section G
Extra exploration reasoning process in Section
Additional ablation studies in Section
Additional computation studies in Section
Additional case studies in Section
Statement of broader impact Section
Statement of limitations in Section M
Statement of reproducibility in Section
A SUPPLEMENTARY IMPLEMENTATION DETAILS
VISUAL-O1 uses the following prompts for ingh-intelligentt and inference. For each prompt, the upper part is used by the high-intelligent model, while the lower part is used by the general-intelligent model.
models during
Prompt of Reasoning Prsu
are a helpful assistant in normal conversation. ftask description]
Follow these instructions carefully:
1. Read the given question carefully and reset counter between budget
2. Generate detailed, logical step-by by--byy-spppppp solution.
3. Enclose each step of your solution within reasoning tags.
4. You are allowed to use most budget steps (starting
budget), keep track of it by counting down within tags
budget, STOP GENERATING MORE STEPS when hitting 0. You
don't t have to use all of them.
Example format:
starting budget Content of step
remaining budget Content of step 2
remaining budget
Content of step 3 or Content of some previous step
remaining budget
remaining budget
13

---

Published as a conference paper at ICLR 2025
Content of final step
Description:
{ambiguous instruction]
Provide a detailed, step-by-step solution given question.
(experience) (ambiguous instruction)
Since the reasoning process of the high-intelligent model occurs during the inference stage, to ac- celerate this stage, we combine all the prompts, allowing the model complete the entire reasoning process in a single output. To clearly demonstrate, we split the prompt used by the high-intelligent model into Prans Prit, and Payn-
Prompt of Reflection Prf
Follow these instructions carefully:
5. Do a self-reflection when you are unsure about how to proceed; based on the self-reflection and reward, decide whether you need to return to the previous steps.
6. Provide critical, honest, and subjective self-evaluation of your reasoning process within <reflection> and </reflection> tags.
7. Assign quality score to your solution as float between 0.0 (lowest quality) and 1.0 (highest quality), enclosed in <reward> and </reward> tags.
8. If the image or question is not clear enough, you need to reflect and try to get answers from the unclear image question.
Example format:
<reflection> [Evaluation of the solution] </reflection> <reward> [Float between 0.0 and 1.0) </reward>
According to the instruction you generated last time, the annotator has rewritten (ambiguous instruction] instruction). Please correct or rewrite your instruction based on the image situation. The image and the result of the data annotator are only for your evaluation. Please do not include the specific case in the instructions. You need to generate the full instruction even if no change is needed.
{clear
If the annotator cannot find it, please let the annotator guess the one with the highest probability.
Make sure the annotator only responds to the rewritten phrase and does not include any other thing.
Instruction: (experience)
14

---

Published as a conference paper at ICLR 2025
Table A: Distribution of dataset.
ELLIPSIS
COLLOQUIALISM
SUBJECTIVITY
RELATIVITY
OTHER
23.3%
27.3%
3.3%
29.3%
16.7%
Prompt of Response Synthesis Psyn
After completing the solution steps, reorganize and synthesize the steps into the final answer within <answer> and </answer> tags. The final answer cannot be empty.
Disambiquated question: fclear instruction) Original question: {ambiguous instruction]
For FUDD, the model will find the five relevant objects of ambiguous instructions in the image. Then, the model will generate the key distinguishing information for each pair of objects. The model will use this disambiguating information appending to the question to obtain a better answer. For -FANU-OT-THOUGHSS, we use the classic prompt: "Think it step by step." and madi-fed prompt: "Analyze the question based the image. There slight differences in performance of two prompts and we report the higher scores.
B SUPPLEMENTARY DATASET DETAILS
For RIS, we use the REFCOCO+ dataset (Kazemzadeh et al. 2014); for VQA, we use the V1zW1Z dataset (Gurari et al. 2018). For their ambiguous instructions, manually screen and construct subsets of 150, 650, and 106, respectively. Specifically, we use GPT-4 for an initial screening, selecting 2, 000 sets of potentially ambiguous instructions from the original dataset. Then, enlist multiple volunteers to manually screen for instructions that clearly contain ambiguity.
We categorize the ambiguous instructions of RIS into five distinct types based the underlying causes of the ambiguity: ellipsis, colloquialism, subjectivity, relativity, and other. The distribution of ambiguities can be found in Table A Their detailed explanations are follows:
Ellipsis indicates that the ambiguity stems from omitted content, where essential informa- tion is left out, leading to uncertainty about the intended meaning.
Colloquialism refers to ambiguity arising from using informal imprecise expressions that may not be universally understood or may vary in interpretation across different contexts.
Subjectivity indicates ambiguities due to subjective judgments, where personal opinions or individual perspectives cause unclear or varied interpretations.
Relativity indicates that ambiguity comes from implied comparisons, where the meaning depends unstated reference point context, making the instruction open multiple interpretations.
Other encompasses all other types of ambiguity that not fit neatly into the previously mentioned categories, covering broad range of miscellancous sources of confusion.
We will release the data we set under the MIT license.
C DEEPER ANALYSIS ON DATASET
To analyze the difficulty of the ambiguous dataset, we further conduct human experiments.
Our dataset is relatively easy for humans but difficult for Al models. We conduct human experiment by inviting 10 volunteers to answer questions of our ambiguous dataset. The results are shown in the Table
15

---

Published as a conference paper at ICLR 2025
Table B: Human results dataset.
HUMAN
GPT-4o
LLAVA
82.17
14.10
8.58
Table C: Average number of Q&A turns different datasets.
METHOD
AMBIGUOUS
GENERAL
(A) HUMAN-GPT
7.1
3.4
(B) HUUMAN-HUMAR
4.8
2.2
Table D: The improvements on different types of ambiguity.
ELLIPSIS
COLLOQUIALISM
SUBJECTIVITY
RELATIVITY
OTHER
METHOD
GIoU
cloU
GIoU
cloU
GloU
cloU
GloU
cloU
GlOU
cloU
LISA
0.0424 0.0460 0.0107
0.0124
0.0025 0.0081 0,0113 0.0119 0.0490 0.0633
VISUAL-O1
0.1742 0.1704 0.1662
0.1375
0.0826 0.2637 0.1444 0.1207
0.0763 0.0490
We find that humans can correctly answer more than 80% of the questions, indicating that the am- biguous dataset does not pose major challenges for humans. This is because humans have strong disambiguation abilities, enabling them to accurately understand the meaning of ambiguous instruc- tions by combining them with images, capability that AI models lack.
The difficulty of our dataset lies the image-text ambiguity. We design another human experi- ment where volunteers only read original instructions but turn, volunteers are allowed to ask one question and receive one answer from either (a) GPT-40, which can see the images but not the instructions, images but not the instructions. Each sample will go through up to 10 turns of Q& A or until the correct answer is obtained. Volunteers cannot ask questions that have the same meaning as the orig- inal instructions. We conduct the experiment on both the general dataset and the ambiguous dataset and calculate the average number of Q&A turns for samples that successfully obtained the correct answer.
allowed
images. In each
(b) another volunteer who can also the
As shown Table we find that on our ambiguous dataset, the turns of Q&A greatly improved. This is because, for samples with ambiguity. volunteers have difficulty determining what exactly to ask, leading to numerous tentative questions before finally pinpointing the intended question and obtaining the correct answer. In contrast, on the general dataset, volunteers can often quickly identify what needs to be asked, resulting in much lower turns of Q&A. This also demonstrates that text-image ambiguity in our dataset increases the difficulty of reasoning.
D DEEPER ANALYSIS ON IMPROVEMENT
To deeply analyze how VISUAL.-O1 mitigates ambiguity issues, we separately caleulate the original scores and the scores after using VISUAL-O1 for each ambiguity category in RIS, as shown in Table D We find that, in the original case, LISA performs relatively best in ellipsis, because ellipsis the relatively simplest category, while the scores for the other categories are almost all around 1. After using Visual-O1, we observe notable improvements in scores across various categories, particularly in colloquialism and relativity. This is because these two can be more easily converted into clear instructions through visual context. The improvements in other more challenging categories also signify the effectiveness of VISUAL--01.
16

---

Published as a conference paper at ICLR 2025
Table E: General instruction results on different intelligence level.
MODEL
VISUAL-OI
GloU
cloU
LISA (AL ET
2023
0.4654
0.4721
OI (LISA)
EMPIRICAL
0.4738
0.4980
VISUAL-O1" (LISA)
INSTANTIAL
0.4985
0.5188
SoM (YAGG ET AL.
2023)
0.3507
0.4154
VISUAL-OI (SOM)
EMPIRICAL
0.3772
0.4336
VISUAL-O1 (SoM)
INSTANTIAL
0.3686
0.4508
Table F: Generalization studies VLN.
MODEL
SR
SPL
NAVI ERROR
VIN-SIG Li BANSAL
2023)
4.72
4.59
7.95
VISUAL-OI SVIN-SIG)
26.42
22.66
5.99
E FURTHER GENERALIZABILITY VERIFICATION ON INTELLIGENCE
We not only compare the results on ambiguous instructions but also extend the comparison to general instructions better confirm the generalization capability of VISUAL-O1. In Table E. we also see that all variants of VISUAL-OI maintain and even improve performance on general instructions. This indicates that the design of VISUAL-O1 011 is highly flexible and can be broadly applied to various intelligence levels of models, thereby improving the whole system's 's reasoning efficiency.
F FURTHER GENERALIZABILITY VERIFICATION ON EXTRA TASKS
VISUAL-O1 demonstrates remarkable generalizability across various tasks. To further validate this, we apply VISUAL--11 to two complex multi-modal tasks: image synthesis the visual synthesis navigation (VLN) in the robotics field. Since models for these types
field and vision-and-languaar of tasks do not have direct language output capabilities, we use GPT-40 for parts requiring inter- mediate language output. For VLN, we use the valid unseen split of the ROOM-TO-ROOM dataset (Adderoon et al. 2018). As detailed in Table F our observations reveal substantial performance enhancements across all tasks, underscoring the versatility and robustness of VISUAL-O1. We also provide examples in Figure A For image synthesis, we choose the state-of-ther -he-----thhht-n art model DALL-E As shown Figure B we notice that even the most advanced models often misunderstand am- biguous instructions in human interactions, whereas VISUAL--OI substantially alleviates this issue. This broad applicability paves the way for extending our method to a wider array of applications, showcasing its potential to improve performance in diverse contexts.
G EXTRA EXPLORATION OF THE MODEL
G.1 EXPLICIT CLEAR INSTRUCTION
We slightly adjust the prompt to allow the reasoning process of instantial experience to generate clear instructions explicitly. We find that implementing this very simple as it only needs to add sentence to Psyn: Explicitly generate clear instruction> and </clear instruction tags before answering.
within <clear
As shown in the Table G the performance. We explain this as explicit clear instruction will help the model to review and better understand the instruction.
see that explicitly generating clear instructions further enhance
17

---

Published as a conference paper at ICLR 2025
Image
VLN-SIG
Visual-01 (VLN-SIG)
Answer
Arawer
Clear Instruction
Success: no
Success: yes
Move forward towards the open door that you see straight shead. As you pass through the door, you will see staircsse to your right. Ascend the steirs and proceed until you reach area set up for dining. Stop there.
Nav Error: 4.1363
Nav Error: 1.7766
SPL: 0.00
SPL: 1.00
Ambiguous Instruction
Please guide me up the stairs and stop once we reach where we eat.
Answer
Answer
Clear Instruction
Success: no
Success: yes
Move forward, heading towards the deor past the foot of the bed. Once you reach the door, proceed through it to enter the hallway. Look for the nearest place to wash hands, which could bathroom or a sink, and there to wash exact location to wash hands is imm mmmedtltlly visible, continue in the general direction along the hallway and turn right opportunity bathroom or sink.
Nav Error: 4.61
0.0000
SPL: 0.00
SPL: 0.99
hands.
Ambiquous Instruction
Guide the bed to the hall, take wash hands.
Figure A: Cases of Visual-O1 on VLN.
Answer
Answer
Answer
Answer
DALL-E3
Visual-01 (DALL-E
Answer
Answer
Answer
Answer
Ambiquuus Instruction
Ambiqquus Instruction
Ambiquuus Instruction
Ambiquuus Instruction
A long vehicie with multiple connected sections moves along tracks urban setting.
Large animal with long nose sprays water, amall oris nearby.
Bright circte high in sky, people beach look up.
Colored lights sky make patterns, people watch at night.
Figure B: Cases of Visual-O1 on Image Synthesis.
18

---

Published as a conference paper at ICLR 2025
Table G: Instantial experience with explicit clear instruction.
VISUAL-O1
GloU
cloU
EMPIRICAL
0.11443
0.1530
INSTANTIAL
0.1304
0.1756
INSTANTIAL W/ EXPLICIT CLEAR INSTRUCTION
0.1316
0.1864
Table H: Comparisons of joint instruction.
VISUAL-O1
GloU
cloU
W/O JOINT INSTRUCTION
0.1131
0.1215
W/ JOINT INSTRUCTION
0.1245
0.1463
G.2 JOINT INSTRUCTION
We expand the functionality of response synthesis: instead of directly integrating the clear instruc- tion with the original instruction, we designed a simple prompt to use a general-intlligent model to rewrite the clear instruction into joint instruction guided by the purpose of the original instruc- tion, ensuring keeping the original purpose while integrating as much information from the clear instruction possible.
As shown in Ta we are surprised to find that the accuracy further increases. We explain this as the extra step can keep the original purpose by fixing some misalignment between clear and original instructions. This also aligns with the findings in our paper that the response synthesis is important for VISUAL-O1.
H EXTRA EXPLORATION OF REASONING PROCESS
H.1 COMPARISONS OF EXPERIENCE PROMPT
To better demonstrate VISUAL-O1 O1'ss 's reasoning process and reveal the specific content of VISUAL- 01 's reasoning, we show the detailed instantaneous and empirical experiences after each reasoning step of VISUAL-O1. The upper part the instantaneous experience for the highly intelligent model, while the lower part is the empirical experience for the generally intelligent model. The sentences in gray are given by reflection.
Experience Prompt of Ains (Top) and Aump (Bottom)
between two bears and not plate
Budget Identify the objects marked as bears in the image.
Budget
The bears in the image are part of the design on the mugs, which are marked with [7] and [3].
Budget 5
Locate objects that are not plates. In this image, the plate a prominent object, but we need to identify objects that
19

---

Published as a conference paper at ICLR 2025
not plates.
Budget 4
Exclude the objects that are the plate, such as the sandwich marked with [4], the soup bowl marked with [2], and the pickle marked with [8].
Budget
Ident ify any objects that are between the two bears but not on the plate.
Budget 2
It seems that the description provided was to find an object that is situated between the two bears but is not plate itself. The steps taken so far have led to the exclusion of items the plate and the identification of the bears the mugs.
<reward> 0.8 </reward>
Budget 1
Looking at the objects, the mugs marked with [7] and [3] situated on either side of the image. The only object that is directly between them and not on the plate seems to the salt or pepper shaker marked with [5].
Budget 0
The object that fits the description "between two bears and not plate" is the salt or pepper shaker marked with [5]. The process involved a systematic exclusion of objects the plate and identifying the bears on the mugs. The final object that fit the description was found to be between the bears (mugs) and not on the plate. The reasoning was sound, and followed the instructions accurately. <reward> 1.0 </reward>
Budget Repeat the question.
Budget Please review the image provided and use its details to rephrase the ambiguous and blurry question into clear and precise one that can be effectively answered. Ensure that your revised question is directly related to the content of the image.
Please ensure that your revised question is specific and directly references observable elements in the image. Your question should guide the respondent in addressing particular details aspects present in the image clearly despite any blurriness. Avoid general or vague terms and aim for specificity that will elicit precise answer.
Budget 1
Budget When rephrasing the question, focus on the
discernible elements in the image, such text, icons, or specific features visible on the computer monitor. Your question should ask for details about these specific elements, avoiding any reference to the clarity of the picture the physical location, as these not relevant
20

---

Published as a conference paper at ICLR 2025
the content displayed the screen. Aim formulate a question that inquires res about the information or processes shown in the image, which can be answered with the visible data.
As the reasoning progresses, in the instantial experience, VISUAL-O1 begins perform increas- ingly in-depth reasoning. Meanwhile, in the empirical experience, VISUAL-O1 starts to generate more specific requirements, such as generating specific locations and elements, which are essential for accurately interpreting and generating unambiguous instructions. This also proves the effective- ness of VISUAL-OI optimization.
H.2 COMPARISONS OF REASONING TRAJECTORY
Image
SoM
CoT
Visual-O1 (SoM)
Answer
Answer
Answer
STPP
STOP
STOP
STOOP
Ambiguous Instruction)
the eye-catching keep going
Figure C: Common case of VISUAL-O1 with trajectory.
Trajectory of SoM CoT (Top) and VISUAL-O1 (SoM) (Bottom)
The phrase "the catching keep going" suggests finding something that stands out and implies continuity or progression. In the context of this image with numbered elements and the instruction, let's analyze the image step by step:
The most eye-catching element in the image is the stop sign, which is labeled with the number [2].
The phrase "keep going" implies progression or continuity. a sequence, we often continue to the next number. Considering the instruction, on the pole of the stop sign.
look at the number (8), which
Therefore, the id of "the eye-catching keep going" is [8].
<count> 5 </count>
<step> Examine the image for elements that might be described as "eye-catching keep going." </step>
<count> 4 </count>
<step> Identify the most prominent and eye-catching object in the image. The upside-down stop sign (with ID [2]) is the most eye-catching object in the image. </step>
<reflection> The steps taken far correctly identify the most prominent and eye-catching object in the image. Since
21

---

Published as a conference paper at ICLR 2025
Table I: Ablation studies.
MODEL
Acc
BLEU-1
VISUAL-OI (LLAVA)
57.58
0.7295
w/o MULTI-EXAMPLES
55.75
0.7092
W/o MULTE
55.90
0.7143
the stop sign is upside down, it draws attention and might be interpreted orientation. Therefore, it is reasonable to assume that the stop sign is the target object. </reflection> <reward> 1.0 </reward>
"keep going" because it is not in its usual
<count> 3 </count>
<step> Identify the number associated with the most prominent and eye-catching object that contradicts its usual meaning, suggesting "keep going" instead of stopping. </step>
<count> 2 </count>
<step> Confirm that the upside-down stop sign is the focus of the instruction, as it is the prominent object that suggests ironic "keep going" message. </step> <reflection> The steps and process effectively identified the upside-down stop sign the most eye-catching object due to its unusual orientation. This sign draws attention and suggests an ironic "keep going" message. The identification of the stop sign's s ID as [2] was made accurately, and the reasoning aligns well with the given instruction. </reflection> <reward> 0.9 </reward>
<count> 1 </count>
<step> Verify the ID on the upside-dowd down stop sign, which is [2]. </step>
<count> 0 </count>
<answer> [2] </answer>
<reflection> The steps taken effectively identified the most eye-catching object that suggests "keep going" due to its unusual orientation. The interpretation aligns well with the instruction, and the ID [2] has been correctly identified as the target. </reflection> <reward> 1.0 </reward>
ADDITIONAL ABLATION STUDIES
I.1 COMPONENTS
We further verify the effectiveness of VISUAL-O1's one-time optimization design details in Table I We find that without multiple examples, i.e., only allowing optimization for the same sample leads to overfitting and ineffective optimization. Without multiple modalities, i.e., lacking visual contexts, VISUAL-Oi optimization capability also declines, as visual contexts help the optimizer determine the current state of optimization. By avoiding all these issues, it demonstrates excellent capabilities.
22

---

Published as a conference paper at ICLR 2025
Image
LLaVA
Visual-01 (LLaVA)
Answer
Answer
Clear Instruction
Res ponse: unanswerable
Res ponse: usb
What device is currently highlighted on the boot device selection screen?
Ambiguous Instruction
Once again, which device is now you so much for help.
Answer
Answer
Clear Instruction
posso:s: what is product
Res ponse:
fiber
Whet name the product shown the imago?
FIBER One 9 Brose
Ambiquous Instruction
question.
picture. double tap.
Answer
Answer
Clear Instruction
Res ponse: sgh connection
Res ponse: sight connection
Whet green the word "sight" say?
siht CONDECTION
Ambiquous Instruction
What does this shirt say?
Answer
Answer
Clear Instruction
Response:
Response: google
What the content desplayed the computer screen?
Ambiquous Instruction
What is this?
Figure D: Case studies on VQA.
Table Results of different budgets with summarization.
AMBIGUOUS
GENERAL
VISUAL-O1
AvG
MAX
MIN
VAR
AvG
MAX
MIN
VAR
W/O BUDGET
8.95
10.14
7.89
0.85
54.02
54.90
53.30
0.35
W/ HUMAN EXP
16.88
18.43
14.29
1.61
54.55
56.20
53.15
1.02
W/ BUDGET
17.83
19.73
16.76
0.96
55.38
56.22
54.10
0.45
W/ 2 BUDGET
19.92
21.89
18.77
0.95
56.31
56.92
54.16
0.58
W/ 3 BUDGET
22.13
24.52
20.91
0.88
57.58
58.56
56.76
0.35
W/ 4 BUDGET SUMM
22.90
25.13
21.43
0.88
57.60
58.57
56.95
0.25
W/ BUDGET + SUMM
23.00
25.28
21.78
0.69
57.79
58.62
56.84
0.28
I.2 REASONING BUDGET
The performance decrease observed when the budget N is due to the length of empirical experience prompt becoming longer, making it difficult for the current to perform image-text understanding effectively. However, this is an unsolvable problem. By
model
23

---

Published as a conference paper at ICLR 2025
Table K: Computational overhead of Visual-O1.
METHOD
STAGE
TIME
VRAM
LLAVA
LORA FINE-TUNING (HU AL..
2021
25.4439MIN
88797MB
VISUAL-OI (LLAVA)
ONE-TIME OPTIMIZATION
1.5260MIN
16148MB
LLAVA
INFERENCE INFERENCE
0.5103s
16102MB
VISUAL-OI (LLAVA)
0.65475
16370MB
GPT-40
INFERENCE INFERENCE
4.94035
VISUAL-O1 (GPT-40)
6.76148
adding an extra step where the model summarizes and simplifies long prompts, the prompt length be shortened.
As shown in T we find that the accuracy further improves when the budget > This means that if the model understands long texts well, method can achieve increasingly higher results as the budget increases. If the model can only understand shorter texts, a budget of 3 is sufficient to achieve good results.
ADDITIONAL COMPUTATION STUDIES
As shown in Table KJ we analyze the computational overhead of VISUAL--O1. We find that VISUAL- Ol's overhead in both the optimization and disambiguation phases low. Specifically, we also compare the time it takes for the vanilla LLAVA and GPT-40 to complete a VQA task. The com- putational overhead is comparable LLAVA, further proving VISCAL-O1's capability.
K ADDITIONAL CASE STUDIES
K.1
VQA CASE
Similar to RIS, we also visualize examples on VQA. As shown in Figure D VISUAL-OI enhances the clarity of ambiguous instructions, helping the task model produce more accurate and natural results.
K.2 FAILED CASE
To figure out how VISUAL-O1 fails in some cases, we conduct the analysis on failed cases. We find that these errors generally be divided into two categories: (1) errors caused by non-reasoning factors such as calculation, color recognition, and orientation identification errors, and (2) errors caused by excessive reasoning.
Acroverr
intrictionn mulorcycie
Armove
Elese extniction)
tellight light hell whtch nellecingg light.
fer-igh right to the foon rta motorryyles..
Ambiguous Instruction
to tnfrrnt
Ambiguous Instruchinn
reffecting
Figure E: Failed cases of VISUAL-O1.
we find that although ViSUAL-OI helps the model perform reasoning analysis well and generates clear instructions, the model is still limited by its segmentation capabilities, resulting in incorrect outcomes. However, we believe that as the model's 's understanding ability improves, such errors will gradually decrease.
The first type of error is shown in the left part Figure E
For the second type of error, we find that sometimes VISUAL-O1 leads to over- ever-reasoning, As shown in the right part of Figure we find that VISUAL--O1 over-interprets the meaning reffffeleiiiii
24

---

Published as a conference paper at ICLR 2025
light one", thinking it refers to small reflective motorcycle. However, the instruction refers to the entire motorcycle. Despite the presence of exces- sive reasoning in some samples, the overall better performance the general dataset means that, in most cases, VISUAL-01 can effectively reason out the correct meaning of the instructions.
the motorcycle rather than the whole
K.3 CHALLENGING CASE
Image
GPT-40
GPT-4o CoT
Visual-01 (GPT-4o)
Answer
Answer
Answer
Response: C. juice
Response: C. juice
Response: A. water
Ambiquous Instruction
Lisa poured glass water, a porcelain cup of coffee, and prepared a pen, mobile phone, and eraser for her friend's visit. Then, Lisa labeled all the items and left an amusing note. But here's something strange for her friend. What does Lise think it substentielly be?
A. water B. coffee
C. juice
D. eraser E. phone
Figure Challenging case of VISUAL-01 with trajectory.
Inspired by the theory of mind, we design very challenging case in Figure F We compare GPT- 40, GPT-40 + COT, and our VISUAL-O1. We find that GPT-40 and GPT-40 + CoT respond the wrong answer: juice, indicating their weak understanding of ambiguous multimodal scenarios.
By reviewing the trajectory of GPT-40 COT, we note that GPT-40 CoT recognized that Lisa incorrectly labeled the water cup but mistakenly believed that Lisa thought the cup contained juice because she placed the label. Given that the cup transparent in the image, we know this is im- possible, meaning that although CoT enhances reasoning abilities, its effect is limited in ambiguous multimodal scenarios.
As shown in the trajectory of VISUAL-O1, it correctly understood that the question was actually asking what Lisa thought the liquid in the glass cup was. Then, noticing the amusing note left by Lisa the table, it inferred that Lisa intentionally mislabeled the items. Thus, it realized that Lisa knew the cup contained water and gave the correct answer, A. water, correctly interpreting the ambiguous command in the multimodal context.
Trajectory of GPT-40 + CoT (Top) and VISUAL-OI (GPT-40) (Bottom)
Let's s analyze the situation step by step:
1. Items and Labels:
Lisa poured glass of water and labeled it as "juice",
Lisa poured a porcelain cup of coffee and labeled it correctly "coffee".
Lisa prepared a pen and labeled it correctly as "pen". Lisa prepared mobile phone and labeled it correctly "phone".
Lisa prepared eraser and labeled it correctly as "eraser".
2. Correctness of Labels: The "coffee" label is correct.
The "pen" label is correct. The "phone" label is correct.
25

---

Published as a conference paper at ICLR 2025
The "eraser" label is correct.
The "juice" label is incorrect because it is actually glass of water.
Given the labels, the only thing that is substantially incorrect is labeling the glass of water as "juice". Lisa labeled it as "juice" as she thinks it is "juice".
Thus, Lisa thinks it substantially be: C. juice
<count>6</count>
<step>Observe the given image carefully. Notice that Lisa has labeled each item on the table. The labels are "coffee" for the cup of coffee, "eraser" for the eraser, "pen" for the pen, "phone" for the mobile phone, and "juice" for the glass water. </ste> <reflection>Star, by identifying each item on the table and their corresponding labels. s/rell.o/reflections Breward>o.&</rewe
<count>5</count>
<step> Identify the discrepancy between the items and their labels. The glass of water is labeled "juice," which is incorrect. incorrect.</step>
<teflectioonReeconiiie the mismatched label understand the source of confusion or humor in the situation.</refleene (heward>0.9</rent
<count>4</count>
<step>Consider the context of the amusing note left by Lisa, which states, "Hi stranger, just labeled everything! Enjoy it." This implies that the labeling is meant to be humorous or kisleading.<<step
<count>3</count>
<step>Determine the logical conclusion from the context and the incorrect label. The strange thing Lisa might think it substantially be is the mislabeled glass of water.</step> trefleectiosConneec the incorrect label ing to the context to identify what Lisa might. find strange or amusing.</refeetiion <rewardb1.0</rewardd
<count>2</count>
<step>Match the options provided (A. water, B. coffee, C. juice, D. eraser, E. phone) with the conclusion. The glass of water being mislabeled led as "juice" aligns with the question's context.</ster,
<count>1</count>
<step>Choose the correct option based the analysis. The glass of water is the item Lisa thinks is strange due to its incorrect label.</step>
<count>0</count>
<answer>A, water</answer>
26

---

Published as a conference paper at ICLR 2025
<reflectionse solution process involved carefully ly observing ng the image, ident fying the incorrect label, and understanding the context the note. The logical conclusion was drawn based on the humor implied by the note and the mismatch between the item and its label. The final answer correctly identifies the glass of water as the strange item. //ee.///feection> <reward>0, 9</rewards
L BROADER IMPACT
With the development and application AI, language instructions for interacting with Al are being applied in increasing number of scenarios. In reality, humans often issue vague instructions for communication, and due to their natural visual ability, language and visual information complement each other, which further intensifies the ambiguity of instruction information. In this paper, reveal the phenomenon of ambiguous instructions. Our proposed VISUAL-O1 substantially alleviates the pressure of ambiguous instructions Al models, helps Al understand more natural instructions, broadens the application range of AI, and serves more non-professional people.
M LIMITATIONS
Despite VISUAL-O1 demonstrating strong reasoning capabilities and allowing general-intelligent models like LLAVA to perform inference without relying on large-scale high-intelligent models such as GPT-40, it still requires the use of high-intelligent models during the one- one-time optimization phase. In the future, we will explore methods that do not rely on high-intelligent models even during optimization, in order to better extend our approach to scenarios with more limited resources.
N REPRODUCIBILITY STATEMENT
We place high emphasis the reproducibility of our work. To facilitate this, we have provided comprehensive set implementation details and prompts the appendix. Additionally, to further enhance the reproducibility of our results, we will release the source code and data.
27


Reviews:
[{"review_id": "FjwvAaU9Rb", "reviewer": "Reviewer_SvAq", "summary": "This paper proposes Visual-O1, an approach that improves VLM's reasoning performance given ambiguous language instructions. For VLMs with stronger reasoning ability, the paper proposes an \"instantial experience\" approach where multi-turn multi-modal CoTs are generated to reason about the instruction, and the model subsequently directly outputs final answer conditioned on such reasonings. For VLMs with weaker reasoning ability, the paper proposes an \"empirical experience\" approach where after multi-modal CoTs are generated, explicit and clear instructions are generated before the model outputs the final answer. The authors demonstrate the effectiveness of their approach in referring image segmentation and visual question answering benchmarks.", "questions": "- For stronger VLMs (e.g., GPT-4o), I don't quite get the motivation for authors to adopt the \"instantial experience\" approach instead of the \"empirical experience\" approach. Intuitively, allowing VLMs to reason and explicitly output the clear instructions with ambiguities removed should almost always improve their performance, regardless of the ability of the VLMs themselves. I also didn't find an ablation study in the paper to support why we need the \"instantial experience\" approach for stronger VLMs.\n- Additionally, per my comments in the review, I'd encourage authors design better prompts such that the generated instructions in their \"empirical experience\" approach do not defeat the purpose of an actual \"instruction\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Context and instruction ambiguity is very prevalent in language and vision-language applications. Handling these cases is an important research problem. The proposed approach presents a promising way to improve the accuracy of VLM's \"best guess reasoning\" under ambiguous instructions.\n- The paper is overall well-written, with clear methodology and extensive experiments.", "weaknesses": "- Table 6 indicates that the effectiveness of the proposed approach and the results of the paper may be vulnerable to noise, and the improvements could lack statistical significance. Specifically, as the budget limit increases from 1 to 5, the success rates are \"55.87, 54.16, 57.38, 55.27, 55.56,\" showing no clear trend of performance improvement with respect to budget limit. For instance, the fact that \"budget limit = 3\" outperforms \"budget limit = 2\" by 3% and \"budget limit = 4\" by 2% could simply reflect evaluation noise.\n- The generated \"clear instruction\" in Figure 3 (which correspond to authors' \"empirical experience\" approach) appears to directly answer the initial question, rather than serving as an actual \"instruction\" that enables the LLM to explore various objects, select the relevant one, and then provide the final answer. Thus, authors' approach seems to undermine the purpose of an \"instruction.\" For instance, for the ambiguous prompt `white bear turned slightly`, a more suitable instruction with ambiguity removed would be, `identify the bear that is predominantly white with subtle color variations from white`.\n\nEdit: The authors have sufficiently addressed my concerns and I'm increasing my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dldmNLbdKH", "reviewer": "Reviewer_B9dk", "summary": "This paper proposes Visual-O1, a multimodal multi-turn Chain-of-Thought reasoning framework to deal with ambiguous natural language instructions in two typical vision-language tasks: referring image segmentation and visual question answering. Visual-O1 builds\ninstance-specific experience for high-intelligence models or creates general experience for any ambiguous instructions for general-intelligence models. Experiments show that Visual-O1 improves the performance on both the ambiguous datasets and the general datasets.", "questions": "1. Could you provide some further information to explain the difficulty of the ambiguous dataset? For example, how humans (might) perform on this dataset?\n2. How are the Chain-of-Though and FuDD baselines implemented?\n3. An example of $\\mathcal{A}_\\text{ins}$ is provided in the appendix. If I understand the method correctly, it should be composed of the alternating appearance of reasoning and reflection. Are the sentences in gray given by reflection? If so, is there an example of the full trajectory?\n4. Analysis of Visual-O1's failure modes will help a lot, as discussed in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposed a prompting method that seamlessly applies to both high-intelligence and general-intelligence models within the same framework. This adaptability indicates that Visual-O1 is not limited to a specific model type and can scale across different levels of model intelligence, which enhances its utility for a wider range of applications and users.\n2. The authors presented extensive experimental results that include a variety of ablation studies and model comparisons. These results strengthen the credibility of their claims and provide thorough evidence of the framework's effectiveness. Notably, Visual-O1 not only improves performance on ambiguous instructions datasets but also enhances results on general datasets.", "weaknesses": "1. Although Visual-O1 shows clear improvements over baselines, the absolute success rates and IoU scores (on the ambiguity dataset) are still not entirely satisfactory. This gap suggests that there are underlying limitations to the model’s ability to handle certain types of ambiguous instructions. The paper could be strengthened if the authors provided a more detailed examination of failure cases (e.g., identifying specific patterns in instructions that remain challenging to disambiguate). Without this analysis, Visual-O1's readiness for real-world applications remains in question.\n2. The paper tends to overemphasize its successes by frequently using terms like \"significant,\" even when sometimes the improvements are marginal (e.g., success rate gains of less than 3%). This could be misleading, especially considering that the ambiguous instructions dataset is not large enough to robustly support such claims. Furthermore, the results do not report averages over different random seeds, which raises questions about the stability and generalizability of the improvements. \n3. Minor issues or typos: The term $x_\\text{rsn}$ should be $x_\\text{rfl}$ in Eq. (6). The word \"botlle\" should be \"bottle\" in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FaAD9GLeNV", "reviewer": "Reviewer_FebY", "summary": "The paper proposes a method for visual o-1 that is similar to language o1 that increases the computation budget on test-time. The work is proposed to highlight the potential of artificial intelligence to work like humans in the real world, especially when the instruction is ambiguous for the instructional model. This is achieved by multi-turn reasoning with chain of thought. \n\nThey have separated the multi-round reasoning process for high-intelligence model and general intelligent model, where it tends to have more round interaction when the model has a higher intelligence level.\n\nThey have evaluated their result on visual-QA, and RIS in comparison with the state-of-the-art model and shows reasonable improved on both segemtantion and visual-QA.", "questions": "1. Can you write the exact GPU settings for Table M?\n2. How does the author interpret that the performance saturates quickly with the budget?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's proposed direction of Visual-O1 towards instructional QA is an interesting direction.\n2. By integrating the chain of thought into the pipeline, the model can improve the segmentation result of referring image segmentation, and the visual QA result. The performance gain on visual QA is larger than RIS, and the performance gain on ambiguous cases is higher than general cases, which indicates that the multi-round and multimodal inference needs more on the harder case.\n3. The additional overhead of the visual chain of thought is limited for both training and evaluation stages.", "weaknesses": "1. To be completely honest, although the performance gain shows statistically significant improvement, however, the improvement is marginal, which indicates that the chain-of-thought works.\n2. The author lacks examples when we actually need visual CoT (As the cipher example in o1).\n3. As shown in Table.6, I actually think the budget curve is confusing, the performance fluctuates for Acc and BLEU, and saturate w/ 3 budget, this indicates that the designed algorithm is not sophisticated enough for scaling up.\n4. They propose the empirical solution for less-intelligent model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)