PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments
Authors: Weijie Zhou
Xuantang Xiong
Yi Peng
Manli Tao
Chaoyang Zhao
Honghui Dong
Ming Tang
Jinqiao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Active Visual Reasoning
MLLM
Embodied AI
Visual Reasoning
Agent
Primary area: deep_learning

Abstract: Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in passive, static settings, limiting their effectiveness in real-world physical environments where an embodied agent must contend with incomplete information due to occlusion or a limited field of view. Humans, in contrast, leverage their embodiment to actively explore and interact with their environment—moving, examining, and manipulating objects—to gather information through a closed-loop process integrating perception, reasoning, and action. Inspired by this capability, we introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to a paradigm of embodied interaction in partially observable environments. AVR necessitates embodied agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive environments designed to assess both reasoning correctness and information-gathering efficiency. We present AVR-152k, a large-scale dataset that offers rich Chain-of-Thought (CoT) annotations detailing iterative reasoning for uncertainty identification, action-conditioned information gain prediction, and information-maximizing action selection, crucial for training agents in a higher-order Markov Decision Process. Building on this, we develop PhysVLM-AVR, an embodied MLLM achieving state-of-the-art performance on CLEVR-AVR, embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath, Geometry30K). Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities.

Phys VLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments
Weijie Zhou 1,2,3, 1.2,,
Xuantang Xiong?*
Yi Peng'
Manli Tao
Chaoyang Zhao 34,1
Honghui Dong'
Ming Tang*
Jinqiao Wang 34
Beijing Jiaotong University
Tencent Robotics X & Futian Laboratory, Shenzhen
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences ObjectEye Inc.
Abstract
Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings, limiting their effectiveness real-world environments where information is often incomplete due occlusion or limited field of view. Humans, in contrast, actively explore and interact with their environment-movinng examining, and manipulating objects- gather in- formation through a closed-loop process integrating perception, reasoning, and action. Inspired by this human capability, we introduce the Active Visual Rea- soning (AVR) task, extending visual reasoning to partially observable, interactive environments. AVR necessitates agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVE simulation benchmark featuring multi-round interactive environments designed assess both reasoning correctness and rrffommattonggateerrriii efficiency. We present AVR-152K, a large-scale dataset offers rich Saain-of-Thougee (CoT) annotations detailing iterative reasoning for uncertainty identification, action-condittoned information gain prediction, and information-maxinizinn action selection, crucial for train- ing agents in higher-order Markov Decision Process. Building this, develop PhysVLM-AVR, an MLLM achieving state-of-the-ar art performance on CLEVR-AVR, embodied reasoning (OpenEQA. Robo VQA), and passive visual rea- soning (GeoMath, 300K) Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities.
Introduction
Visual reasoning, fundamental capability in artificial intelligence, has enabled multimodal large language models (MLLMs) [1] 22
5 to excel at tasks such as object counting [6, 7 8 and visual question answering (VQA) 9 10 12] These abilities are crucial for developing intelligent systems that can comprehend and interact with the visual world. As MILMs continue to evolve, their potential to understand and reason about dynamic, real-world environments is increasingly recognized, paving the way for more sophisticated and autonomous agents [13
3
[4 1
Equal contribution Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pastive Visual Reasoning
ACC
GPT- (iPt VLy
G: How many cubes in the desktop and Its appendage have the same color as banana?
0.6
incomplete observation eods incorrect answers
Answer:
X
02
Namber of cooolatee objocts
Average count accuracy for categories
Step0
Step1
Step2
Plak the
Actiono:
Actiont:
Answer:
Tabletop scenanio that infirmation
Active Visual Reasoning
putiill due occlnsion or ara container
Loop Step: Partially observable perception
acquire new
Figure 1: Passive Visual Reasoning (top) fails with partial views. Active Visual Reasoning (AVR, bottom) actively interacts to gather information for correct answer.
Despite these advancements, current MLLM approaches to visual reasoning- iincluding prominent tasks like VQA [88 [2 19 20 and spatial reasoning [21] 23 -ree--ommmi rely on static, passive visual inputs. This inherently limits their applicability in real-world physical environments where information often partially observable due to occlusions, containment, or limited field of view 24). As illustrated in Figure [ passive model observing only the initial image lo answers incorrectly, whereas an active agent interacts to uncover occluded objects across I and 12, arriving at the correct answer. And in the task of counting all object categories in the scene, the more occluded objects there are, the worse the model performs in passive visual reasoning mode. This highlights that effective reasoning in partially observable settings necessitates interaction actively acquire missing information.
Humans, in stark contrast, naturally overcome such limitations through active perception. We instinctively move, change viewpoints, and manipulate objects to gather information needed for specific goal. This active engagement exemplifies core aspect of human cognition: closing the perception-reed reasoning-aae -actiinn loop to incrementally resolve uncertainties and establish the causal links essential for complex reasoning.
However, existing computational frameworks often suffer from paradigm fragmentation and thus fail to holistically model the complete perception- cettriioo-rraonnnn---ee -atiion loop. Conventional visual reasoning tasks (25) 10 (e.g.. CLEVR (6) assume complete visual inputs, requiring only single "perception
27 28). While embodied question answering (EQA) tasks (e.g.. OpenBQA (2),, RoboVQA (30] involve interaction, they often focus
+ reasoning" step without active information gathering [26]
on reasoning from single, albeit potentially long, video sequences. Similarly, embodied exploration
methods [31 32 33 also interactive, tend to prioritize task completion metrics (e.g., navigation success) over the explicit information gain needed for nuanced reasoning. The fundamental limitation across these approaches is their failure to effectively link reasoning with the strategic, sequential actions essential for information gathering. Specifically, they often overlook how an agent's actions dynamically alter available information and, crucially, how these changes should guide and refine the ongoing reasoning process, especially when multiple evidence- evidence-gathering steps are required.
Inspired by human active perception and to address these limitations, we introduce the Active Visual Reasoning (AVR) task. AVR extends visual reasoning to partially observable, interactive environments, bridging the gap between passive observation and active, embodied understanding. Specifically, AVR combines embodied interaction with temporal visual reasoning, requiring agents to: (1) actively gather information through sequential physical actions; (2) integrate multi-step observations for coherent reasoning; and (3) dynamically adjust decisions based on incremental visual feedback. This paradigm requires models to interpret visual data and strategically decide how to interact with the environment, thereby resolving uncertainties crucial for complex reasoning.
To establish a rigorous foundation for the AVR task and facilitate research in this domain, this paper makes the following primary contributions:
We formally introduce and define the Active Visual Reasoning (AVR) task, mandating ac- tive information gathering, multi-step integration, and dynamic decision-making in partially observable settings.

---

actions and We develop CLEVR-AVR, a simulation benchmark featuring ments. This benchmark specifically designed to assess both the reasoning correctness and the information-gatherin efficiency of agents performing AVR tasks.
interactive environ-
We present AVR-152K, a large-scale dataset with multi-level annotations designed to train agents for AVR. Its core component, AVR-Core models the task as a higher-order Markov Decision Process and features rich Chain-of-Thought (CoT) annotations. These CoT's articulate the structured, iterative reasoning humans employ for active information seeking, including critical steps such as: (i) identifying information uncertainty, (ii) predicting action-conditions information gain, and (iii) selecting
maw.rttiiiimmimiiinng.... actions, thereby providing explicit
supervision for these decision-making processes.
We develop and evaluate PhysVLM-AVR, an MLLM that achieves state-of-the-art performance CLEVR-AVR while maintaining strong results on standard embodied visual reasoning tasks, and reveals insights into current MLLM limitations regarding active interaction.
Collectively, our work provides a foundation for developing MLLMS capable of actively reasoning about and intelligently interacting with their physical environments, thereby narrowing the gap between static visual reasoning and embodied intelligence.
2 Related Work
Visual Reasoning. Traditional visual reasoning tasks, such as Insight-V [34. CLEVR O and its variants (e.g., CLEVR-Math 77, Math Vision (355), have significantly advanced models' abilities in attribute recognition, counting, and spatial reasoning [36]. However, these tasks operate on static, fully observable images, lacking mechanisms for active information seeking in partially observable scenarios. Consequently, they do not model the sequential, interactive process essential for real-world understanding. AVR addresses this by requiring agents to perform sequential actions to actively acquire information necessary for reasoning.
Embodied Question Answering, Embodied Question Answering (EQA) tasks (e.g.. OpenEQA (29]. Robo [30]. CityEQA (37) extend question answering to simulated embodied environments. However, by often relying on pre-captured data, these approaches limit agents to passive observation rather than active, dynamic exploration to resolve current uncertainties. As a result, the critical closed-loop interaction where reasoning about the question drives information-seeking actions, and new information from these actions refines understanding is frequently underemphasized. AVR, in contrast, centers on this loop, compelling agents to integrate multi-step observations acquired through their own goal-directed actions.
Embodied Exploration. Embodied exploration and task planning methods (e.g., Knowledge-based EQA 33 Embodied Reasoner 32] Plan-BBench (38), EXPRESS -encch [9) enable agents to interact with their environments for broader goals like navigation or multi-step task completion. However, these approaches typically optimize for overall task success (e.g.. navigation efficiency) rather than the targeted information acquisition crucial for active reasoning. Their evaluation often prioritizes general task metrics over the specific information gain pertinent to reasoning objective. AVR, conversely, prioritizes dynamic, uncertainty-drit action selection to maximize information gain directly relevant to the goal, highlighting the link between information-seele reasoning refinement.
In summary, while existing research touches upon interaction and reasoning. AVR uniquely integrates these by emphasizing active perception driven by reasoning needs. This directly addresses robust understanding partially observable environments, mirroring a key aspect of human cognition.
3 Active Visual Reasoning
3.1
Problem Formulation
We define Active Visual Reasoning (AVR) as a closed-loop paradigm where an agent must actively interact with partially observable environment E using finite set of atomic actions A to answer question Q. Unlike conventional visual reasoning, AVR models reasoning as an iterative perception- reasoning action cycle. At each timestep {0, 1,.,,,., the agent receives a partial observation
3

---

and maintains an observation history h, = (o...... Based on Q and h, the reasoning module freason generates an internediate reasoning trace:
Think, freason ( Q A,.
The agent then assesses if h, is sufficient answer Q. If so, it generates an answer y; fanswer (Q, 14). Otherwise, it selects an optimal information-gaatering action to maximize expected information gain about the true answer Y:
a = argmax dy EA E 1011
where h = (hr, 0t+1). Executing a, yields 0++l, and h44 is updated. This cycle iterates until an answer is generated or max is reached.
AVR thus integrates three key components: (1) Active Information Acquisition: Strategically inter- acting to gain relevant information. (2) Temporal Visual Integration: Reasoning over accumulated sequential observations. (3) Dynamic Decision-Making: Continuously adapting beliefs and actions based on new evidence.
3.2 Key Challenges
AVR presents distinct challenges beyond static visual reasoning: (1) Efficient Exploration: Strategi- cally exploring partially observable environments to acquire task-relevant information with minimal actions. (2) Temporal Visual Reasoning: Integrating and reasoning extended observation sequences from multi-step interactions. (3) Baasoninn-Drrveen Actions: Ensuring actions are guided by current understanding purposefully reduce uncertainty, avoiding random exploration.
Addressing these multifaceted challenges necessitates dedicated benchmarks and datasets. Accord- ingly, the CLEVR-AVE benchmark and the AVR-152k dataset, detailed in the next section, designed to encapsulate these difficulties. Our dataset construction, in particular, captures the sequen- tial decision- decision-making processes inherent in AVR, providing structured foundation for developing and evaluating agents for these complex tasks.
4
CLEVR-AVR Benchmark and AVR-152K Dataset
4.1
CLEVR-AVR: Benchmark for Evaluating AVR Capabilities
The CLEVR-AVR benchmark is a simulation based evaluation suite designed to rigorously assess an agent's proficiency in the AVR paradigm. Leveraging the Genesis (40) physical simulation platform, extends the classic CLEVR 66 setup into an interactive, embodied domain. This provides a controlled yet rich environment for studying the closed loop of perception, reasoning. and action fundamental to AVR. Further details on the scene types. question template structures, and the distribution of occlusion and stacking challenges within CLEVR-AVR are provided in Appendix Figures A-1 A-2
and A-3
respectively.
Challenge of Efficient Exploration in Partial Observability. A core tenet of AVR is overcoming incomplete information. As shown in Figure 2 CLEVR-AVE confronts agents with diverse scenarios featuring 10 occlusion types, 10 stacking types, and 10 composite scenarios combining these elements. The action space includes object manipulation and viewpoint changes, with agents required to interactively gather information answer diverse question types. For detailed action formats and candidate generation, see Appendix Sec. A.
Challenge of Temporal Visual Reasoning. CLEVR-AVR demands multi-step integration of obser- vations for coherent reasoning. On average, each question requires at least two reasoning steps to resolve, with some scenarios designed to necessitate 4-6 interaction steps.
Challenge of Reasoning-Driver Actions. Each question in CLEVR-AVR is paired with candidate responses that include both final answer options and intermediate [Act ion] options, exemplified Figure This structure allows agents to either provide a conclusive answer or explicitly choose to interact further with the environment. Agents must assess whether current information is sufficient or action is necessary to reduce uncertainty, promoting purposeful, unceriainty-reducing interactions over random exploration. (Further details on action candidate generation in Appendix Sec. A.I)

---

CLEVR-AVR Benchmark
of question types
space
Example
Movelijjot
adz
types CLEVR-AVR
comport
COLFT
2,3%
2-Spic
S-Ceppate
taee rary caee the
55
67
Choic
Compare
Exiot
55 2..2%
A: Bi
Picktbrry bite cstadden.
Peclinnal pepple cxack
Pickikage stlard
onter
11%
632
26
skape
Key Actione IL It, GT Anewer A
Active Visual Reasoning
Updata Cicarvation (s-1)
(Uligee-OOddr Markey Model)
Action Exeoution dion
Environment
Stap
Occition Branch
Insefficeet Intomnndian
Action
Now Cisaeration
Wittl Observatio
Poosoning Stotd Modellng
Optimization Oojective
Information
Aroouee
max(ICV:yylQ.h...)
Figure 2: Top: CLEVR-AVR Simulator Benchmark, showing distributions of question types, action space, scenes, and examples. Bottom: Higher-order Markov Decision Process (MDP) paradigm for Active Visual Reasoning (AVR).
Evaluation Metries.
CLEYR-AVR primarily evaluates: (1) Information Sufficiency Judgment Accuracy (ACCisi): Accuracy in judging whether the initial observation provides sufficient information to answer the question. For example, crucial objects for counting task are occluded in the initial view, a correct judgment would be acknowledge insufficiency and select action option rather than attempting a premature final answer. (2) Information Gain Rate (IGR): This metric is derived by dividing the count information- gainnng decisions by the total number decision steps. (3) Final Answer Accuracy (ACCFA): Correctness of the final answer, with ground truth from the simulator.
An action achieves Information Gain if it reveals new, question-relevant visual information (e.g.. uncovering hidden objects, changing viewpoints for obscured areas). This is determined by simulator ground truth: action gains information if it unhides relevant objects/surfaces previously occluded or stacked upon. This objectively measures if the action yielded more complete observation, crucial for tasks like counting or attribute querying needing comprehensive exploration. These metrics together comprehensively assess an agent's efficient and accurate Active Visual Reasoning (AVR).
4.2 AVR-152k Dataset: Modeling Active Visual Reasoning Higher-Order MDP
To facilitate the development of agents capable of Active Visual Reasoning (AVR) particularly in addressing the challenges Efficient Exploration, Temporal Visual Reasoning, and Reasoning- Driven Actions we introduce the AVR-152k dataset. A key component of this dataset, AVR-Core, specifically designed to model AVR tasks as a higher-order Markov Decision Process (MDP), as illustrated in Figure 2 This MDP formulation provides a principled framework for agents to learn how to sequentially gather information and reason in interactive environments.
Within this MDP, As shown in Figure 3, AVR-Core provides rich Chain-of- Thoughh (CoT) annotations for the reasoning state (Think,) ). These CoTs meticulously structured reflect a natural, human- like information-seekle -sekking process: (1) Assessing Current Understanding: evaluating the information
available from observation history (h₂) relation to the question (Q) and identifying key uncertainties or missing details. (2) Evaluating Potential Actions: considering available actions (A) and forecasting their potential to resolve identified uncertainties and yield valuable new information. (3) Strategic Decision-Marn Makingg: based on the assessment and evaluation, deciding whether to take an information- gathering action (a, or, if uncertainty is sufficiently resolved, provide a final answer. Training with these explicit CoT annotations enables models to internalize this multi-step reasoning.
5

---

AVR-Caption
AVR-Core
How cubes the desktup apoendage have color pursimmun?? 2: B: D: Open F: Close drawer. persimmon, H......
Caption 1oox
Genini
Eabclingant vertynng quality
AVR-Eunbodied Reasoning
D435i
CrT-Nose
Data Collection
CoTo
CoTl
Cof2
UMI
Col-ecfiee
blox +
OUT-Verfiid
Actiond Grasp
Actiont
Arswerr
Mun (oonecer)
E: Open crower
Questions
Datal
Data3
<hinks
CoTo
de Information Cion Production
Pc Open drower This sction wosld roaal the oontents of
Vihinic
Uncettistt Hhpothzsi
dewar This hos ligh poicetial to find mor orange sqparos if the are arrol mode:
There siditiona
behiad.
Exft
OsT
2800.5 0.208, 0.75, inside
Grapp periimen
There nigltt additional ohe 20000 0.122.
Hi Move puppl enall-stzed block the right
Refine
V3
8,630) Theroforo occiesions neuelved
To
lower
The pessinmes occludes pat the view, and the drawer
diseć... Guasping the persimmnon is ligh-ptiocty action
Figure 3: AVR-152L Dataset Construction. Left: Workflows for AVR-Caption and AVR-Embodied Reasoning (perception, temporal reasoning). Right: AVR-Core details including its sequential annotation, CoT supervision, and quality verification.
The full AVR-152K dataset, including its foundational subsets, supports training models to master this complex, interactive reasoning loop. As shown in Figure AVR-152K comprises three subsets with progressively increasing complexity:
AVR-Caption (100k samples): Focuses foundational visual perception and spatial understanding, providing dense captions with bounding boxes for indoor scenes from diverse embodied datasets (e.g., SeanNet [41] RT1
42). Captions were generated by Gemini-2.0-flasst 43] using the prompt
detailed in Appendix Figure A-4 and an example data instance shown in Appendix Figure A-3
WYR-Embodied Reasoning (50k samples): Advances to temporal visual reasoning. It consists of multi-image sequences paired with questions requiring spatiotemporal understanding. Dense captions were generated by Gemini-2.00 2.0flash,, while DeepSeek-R11677 1B [4 produced reasoning chains (see Appendix Figure 6 for the prompt), which were subsequently optimized by DeepSeek- V3 [45) (see Appendix Figure A-7 for the refinement prompt). An illustrative example from this subset is provided in Appendix Figure A-8
AVR-Core (2k samples): Directly instantiates the higher-order MDP for AVR. Data was collected using UMI [46] devices interacting with 640 real-world tabletop settings. As shown in Figure each sample in AVR-Core contains expert-structured CoT annotations for the Think, state, initially authored by human experts through multi-step process (details in Appendix Sec. B.I), and subsequently refined by Gemini (see Appendix Figure A-9 for the refinement prompt). These annotations explicitly demonstrate the human-likk reasoning process. They detail how to assess uncertainty, predict information gain from potential actions, and articulate the rationale for selecting an action or providing final answer. This iterative process is exemplified shown in Appendix Figures A- A-11 and A-12 Questions typically involve several reasoning- and-action steps (avg. 3.2), requiring models repeatedly apply this iterative decision-making process. VR-Core underwent rigorous validation: expert annotation, logical consistency verification pnnceptioo--casoningg action validity.
the multi-step interactive reasoning sequences
by Gemini [43] and human expert review
The higher-order MDP embodied by AVR-Core (illustrated in Figure A-10 A-11 and A-12) for- malizes this iterative, closed-loop process. Each sample in AVR-Core captures a single step this interaction. It provides the current context, including the question Q, visual observation history h, past action history 00.-11 and the set of available actions Crucially, it contains the expert-annotater reasoning trace Think,, which details the assessment of current understanding, evaluation of potential actions, and strategic decision-making. As an approximation of the goal Equation ? This reasoning culminates in a recorded outcome: either chosen information gathorringgttiiggggg action E A or final answer yr- an action a, is selected, the resulting new visual observation 0++1 from the environment E is also included, forming the basis for the next step. This structured data allows models to learn the

---

mapping from the current state and available actions to the generation of reasoning trace Think; and the subsequent decision.
AVR-152K, with its structured CoT annotations within an MDP framework, offers a principled ap- proach for developing models that actively seek information for reasoning in interactive environments.
PhysVLM-AVR Model
To equip a MLLM with active visual reasoning capabilities, we develop PhysVLM-AVR. The architecture of PhysVLM-AVR is similar to LLaVA (2). employing Qwen2.5-3B [47] as the LLM decoder and SigLIP-400M 48 the visual encoder. A key modification for efficient multi-image reasoning is the introduction of a max pooling layer after the visual encoder's s output, reducing the number of visual tokens by factor of three. This allows for more efficient processing of multiple visual inputs. More details of the model architecture see Appendix A-114
We employ multi-stage, mixed-data training strategy progressively build the generalizable active visual reasoning capabilities of PhysVLM-AVi
Stage 1: Aligument. In this initial stage, we focus on aligning visual features with the language model. We train only the connector layers (2xMLP) using the VL-Verrrtraann 2 dataset.
Stage 2.1: Single-Image Understanding, develop foundational image comprehension, all parameters are fine-tuned using the LLaVA- Vision-data [3]
Stage 2.2: Comprehensive Visual Understanding, We then enhance the model's broader visual understanding capabilities by training on the d4-nnstruct-data
[ and our Cap-Copt
Stage 3: General Reasoning and Active Visual Reasoning. The final stage aims to instill general reasoning abilities and specialized active visual reasoning skills. For this, we fine-tune the model on diverse mixture of datasets: Reason-RFT-1220 (25). Mi-DeepSeek-R1--Dsiiee 100k 49 our AVR- Embbddedd Reasoning and AVR- Core datasets.
This multi-stage training culminates in the PhysVLM-AVR-33 model. The detailed training configu- ration for PLysVLM-AVR.. including hyperparameters and software environment, is presented in Appendix Section C and Figure
Experiment
6.1
Experimental Setup
Tasks. We demonstrate the effectiveness of our dataset and model through experiments across the following three categories of tasks:
Active Visual Reasoning, To demonstrate that our approach enables models to acquire active visual reasoning capabilities, we first conduct comparative experiments on the CLEVR-AVR (Sec. benchmark. The CLEVR-AVR benchmark is built on the Genesis simulation framework, ensuring that no similar images appear in the training data.
Embodied Reasoning and Planning. To demonstrate that the reasoning capabilities developed with our dataset and exemplified by PhysVLM-AV are also effective in embodied reasoning and task planning scenarios, we include comparative experiments on two embodied benchmarks: OpenEQA 29] and Robo VQA [30].
Visual Reasoning. To demonstrate the generalization reasoning capability of Phys in static visual reasoning tasks, we further evaluate its performance on two visual structure perception benchmarks: GeoMath (50) and Geometry3K (5T)
Baselines. In addition to our Phys PhysVLM-AVR-3B (Sec we also fine-tune Qwen2.5-VL V--BB on the AVR-152K dataset to obtain AVR-Qwen2.5-VL... (fine-tune details see A-13). We compare them against four categories of models: (1) Open-source MLLMs: Qwen2.5-VL-77 00 and
ILLVVA-V--BB [3], representing standard multimodal foundation models. 22Visua Reasoning
MLLMs: R1 RI-Onevision-7B [52] and RFF-77 (25), specialized for visual reasoning tasks. (3)

---

Embodied MLLMs: Embodied Reaiiier-eessnee-1-1 [32 and RoboBrain-27 7B [13] designed for embodied reasoning tasks. (4) API-based MLLMs: GPT-4o [53] and Gemini-2.0-flast [3].
Evaluation Metrics, For CLEVR-AVR, we report ACCisı, IGR, and ACCFa (Sec measuring
the correctness of judging whether the initial observation is sufficient, information gain rate of
action decision and accuracy of final answers. For OpenEQA, we follow the LLM-score [29] (GPT-
40) evaluation protocol of the original paper. For RoboVQ.. we report BLEU1-4 scores established in the original benchmark.
30 as
Table CLEVR-AVR Benchmark: Our PhysVL.M-VRR-3BB and AVR-Qwen2.5-V1.-77 vs. various MLLM categories. Best bold, second underlined.
Occlusion
Stack
Compusite IGR
AVG.
Method
ACC
IGR
ACCps
ACC1st
ACCFA
ACC
ACC₂s
ACC|s
ACC₁x
Open-SaurcoMLL.NG
LLaVA-OV-78 Qwen2.5 VL
0
0
0
0
0
0
0
0
7.7
5.8
7.7
7.1
5.4
0
4,9
3.7
2.6
Reasoning KLIMs
R1-Omevisian.7B
2.0
Reason-RFT RFT-7B
0
0
0
1.5
1.5
0.0
0.5
0.5
0.0
Embodi ed MLLTMs
RoboBrain-7B -mboodiel-Reaonee-78 Reasoner-78
4.5
3.8
0.0
4.6
3.1
3.1
3.6
2.3
Embodied-
21.2
13.6
1.5
16.4
8.2
3.3
23.1
10.8
0.0
20.2
10.9
1.6
API-based XLLMs
Genimi-2.0-laah
50.8
24.4
33.6
52.6
27.4
31.0
56.3
42.0
25.9
53.2
31.3
30.2
GPT-4o
85.2
39.4
53.1
90.5
46.8
41.4
89.6
06.3
42.5
88.4
50.8
45.7
AVR--Wwe225.VL-7B
95.5
43.9
40.9
82.9
37.4
89.3
34.7
PhysVEMM-VRR-38
90.6
27.4
42.2
90.5
22.4
37.9
90.2
40.0
39.1
90.5
29.9
39.7
6.2 Results on Active Visual Reasoning Tasks
Tresultss from CLEVR-AVR offer compelling insights into active visual reasoning and highlight our AVR framework's 's significance. First, the results validate our premise: existing MLLMs, trained on static data, struggle with active reasoning in interactive, partially observable environments. Standard open-source MILMs (LLaVA-OV-7B. Qwen2.5-VL-7B) and passive visual reasoning models (R IRI-Onevision-7B.
RFF-7) show near-zero performance. This illustrates passive capabilities don't 't translate to AVR's dynamic demands, necessitating new paradigms.
Critically, existing embodied MILLMs reveal fundamental limitation. While models like Embodied- Reasoner- 7B can detect information incompleteness (20.2% ACCss),, they largely fail to act effectively for correct reasoning (only 1.6% ACCFA). This highlights: current embodied agents may recognize missing information but struggle to strategically acquire and integrate it.
In contrast, our VLY- 3B and the fine-tuned Sufficiency Judgment Accuracy (ACC1s)) (90.5% and 89.3%), surpassing GPT-4o (88.4%). This validates the CoT of our AVR-Core dataset teach the identification of uncertainty and the need for interaction. Beyond this, WVR-Qwen2.5-VL. V-LBB achieves robust 39.7% average final reasoning accuracy (ACCrA). making it the best open source model, second to GPT-4o and well ahead of other baselines. The PhysVLMMAV M-AVR-3B also shows considerable improvement (39.7% ACCFA), further showing the impact of our data set.
VL--BB excel in Information
However, the gap between our models' high ACC1sı and their final ACCFA (c.g., PhysVLM-AVR-33 90.5% ACC1sı vs. 39.7% ACCFA) highlights AVR's ss central challenge: mastering optimal action selection and multi-step information integration for coherent reasoning. While identifying the need to act is learned, consistently choosing the best action and synthesizing information over time needs more development.
In summary, CLEVR-AVR results show existing MILMS' difficulty with AVR. They also affirm our AVR framework (AVR-152K dataset and PhysVLMMAVV model) a significant step towards MLLMs that can intelligently explore, gather information, and reason in physical environments.

---

6.3 Results on Embodied Reasoning and Planning Tasks
As shown in Figure 4 aa) and (b), our proposed models, PhysVLM-AVR-31 and WVR-Qwen2.5- VL-7B, achieve strong performance on both the OpenEQA and RoboVQA embodied reasoning benchmarks. On OpenEQA, our models consistently outperform standard multimodal models and dedicated embodied reasoning models across all sub-tasks. Notably, even when trained with only 1/20 of the RoboVQA training set (indicated by *), our models deliver BLEU scores close to or surpassing those of fully supervised baselines. These results demonstrate that active visual reasoning and the AVR- 152k dataset significantly enhance embodied reasoning and planning capabilities, even under limited data conditions.
10 RLEU
Ohicoisbiee
bealuation
Tr
60
M
-0
23
World lannl
rcosting
Pinctice
20
23
Atcl socsgrinee
Speiidl
BLEUI
BLEUZ
BLEUB
BLEU4
GmAtah
undentanding
Robebrain-To
HAVEMAAVE
AVR.Qwaa25NLL. 5ML-7B
1..VA OWV-TT
(c) GeeMath and Geometry3K
(a) OpenEQA Benchmark
(b) RoboYQA QA Benchmurk
Benchmarks
Figure 4: Results for Embodied and Visual Reasoning Tasks.
6.4 Results Visual Reasoning Tasks
We further evaluate our approach on static visual reasoning benchmarks, GeoMath and Geometry3K. As shown in Figure c), PhysVLM VLY-VRMM achieves the highest Accuracy among all compared models, outperforming both large-scale API models (GPT-4o) and specialized visual reasoning models (Reason-RFT-7B. This indicates that active reasoning paradigm not only benefits embodied scenarios but also generalizes well to traditional visual reasoning tasks, confirming the broad applicability and robustness of our approach.
6.5 Ablation Study
To assess the contributions of our AVR-Core dataset and its Chain-of- (CoT) annotations, we conducted ablations on CLEVR-AVR (see Table 2.. Removing the entire AVR-Core dataset ("w'o AVR-Core") caused a catastrophic performance collapse: ACC₁sı dropped from 90.5% to 16.4%.ACC, from 39.1% % to 2.3%, and IGR from 29.9% to 11.2%. This highlights AVR-Core's fundamental role teaching the model actively gather information and reason iteratively within the oigeer-order MDP framework.
Excluding only the Chain-of- Thoough (CoT) annotations from AVR-Core ("w/o CoT"") also led to a significant de- cline: ACC1sı fell 47.6%, IGR 18.0%, and ACCFA to 16.9%. This underscores the CoTs' crucial function in providing explicit supervision for the nuanced reason- ing steps of uncertainty identification, action ceriion-ondiiioned information gain prediction, and strategic action selection.
Method
ACC1sı
IGR
ACCPA
Full Model
90.5
29.9
39.7
w/o CoT w/o AVR-Core
47.6
18.0
16.9
16.4
11.2
2.3
Table Ablation study results.
These ablations clearly demonstrate that both the specialized AVR-Core dataset and its detailed CoT annotations are indispensable for developing effective Active Visual Reasoning capabilities.
7 Conclusion
In this work, we introduced Active Visual Reasoning (AVR), novel paradigm that extends visual reasoning to interactive, partially observable environments. We developed the CLEVR-AVR bench-

---

mark to rigorously evaluate AVR capabilitiess and the AVR-152K dataset, with its core AVR-Core component providing detailed Chain-of- -f-ouhhhthh annotations within higher-order MDP framework, to train agents for this task. Our PhysVLM-AV model demonstrates significant progress, achieving state-of-the-ar performance on CLEVR-AVR and showing strong generalization to other embodied and static reasoning tasks. Our findings highlight that while current models can identify information incompleteness, a critical challenge remains enabling them to strategically act to acquire and integrate new information effectively. Future work will focus enhancing the model's ability to predict action-conee conii--oodiiioned information gain and select optimal also plan to explore the application of AVR more complex real-world scenarios and investigate methods for improving sample efficiency in learning these active reasoning skills.
pntermation-gatheee actions. We
Acknowledgments
This work was supported by the National Key Research and Development Program of China (No. 2022 FFB4340330,,0 the National Key R&D Program of China under Grant Nos. 2024YFE0210600 and 2022ZD0160601, and the National Natural Science Foundation of China under Grant Nos. 62 62176254, 62276260.
10

---

References
[1] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-languse model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[2] Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[3] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Tackling multi silaie video, and 3d in large multimodal models. arXiv preprint arXiv:2407.0788% 2024.
Li.
[4] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185 1818-244198 2024.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[6] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[7] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024.
[8] Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L. Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14963-14973, June 2023.
[9] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Yc, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.061:
2025.
[10] Yufei Zhan, Yousong Zhu, Shurong Zheng. Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning, 2025.
[11] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953-14962. 2023.
[12] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[13] Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. arXiv preprint arXiv:2502.2122 2025.
[14] Huajie Tan, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Yaoxu Lyu, Mingyu Cao, Zhongyuan Wang, and Shanghang Zhang. Roboos: A hierarchical embodied framework for cross- embodiment and multi-agent collaboration, 2025.
[15] Weijie Zhou, Manli Tao, Chaoyang Zhao, Haiyun Guo, Honghui Dong, Ming Tang, and Jinqiao Wang. Physvim: Enabling visual language models understand robotic physical reachability, 2025.
11

---

[16] Yunsheng Tian, Karl DD Willis, Bassel Al Omari, Jieliang Luo, Pingchuan Ma, Yichen Li, Farhad Javid, Edward Gu, Joshua Jacob, Shinjiro Sueda, et Asap: Automated sequence planning for complex robotic assembly with physical feasibility. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4380 4386. IEEE, 2024.
[17] Weijie Zhou, Yi Peng, Manli Tao, Chaoyang Zhao, Honghui Dong, Ming Tang, and Jinqiao Wang. Lightplanner: Unleashing the reasoning capabilities of lightweight large language models task planning. 2025.
[18] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recogmition, pages 3195- 3204, 2019.
[19] Yu Wu, Lu Jiang, and Yi Yang. Revisiting embodicdqa: simple bascline and beyond. JEEE Transactions on Image Processing, 29:3984-3992, 2020.
[20] Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, and Chuang Gan. Multiply: multisensory object-centric embodied large language model in 3d world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26406-26416, 2024.
[21] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[22] Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 136442 2024.
[23] Gemini Robotics Team, Saminda Abeyruwan, et al. Gemini robotics: Bringing ai into the physical world, 2025.
[24] Giovanni Sutanto, Austin Wang. Yixin Lin, Mustafa Mukadam, Gaurav Sukhatme, Akshara Rai, and Franziska Meier. Encoding physical constraints in differentiable newton-euler algorithm. In Alexandre M. Bayen, Ali Jadbabaie. George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger, editors, Proceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pages 603. PMLR, 10-11 Jun 2020.
[25] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503. 20752,, 2025.
[26] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216-233. Springer, 2025.
[27] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02 024900 2023.
[28] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[29] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IERR/VVF Conference on Computer Vision and Pattern Recognition, pages 16488-16488 2024.
12

---

[30] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrish- nan. Christine Chan, Gabriel Dulac-Arold. Sharath Maddineni, Nikhil Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 645-652, IEEE, 2024.
[31] Ram Ramrakhya, Matthew Chang, Xavier Puig, Ruta Desai, Zsolt Kira, and Roozbeh Mottaghi. Grounding multimodal IIms embodied agents that ask for help with reinforcement learning. arXiv preprint axXiv2554.00007,, 2025.
[32] Wenqi Zhang, Mengna Wang, Gangao Liu, Xu Huixin, Yiwei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, et al. Embodied-reasoner: Synergizing visual search, reasoning, and action for embodied interactive tasks. arXiv preprint arXiv:2503.216665 2025.
[33] Sinan Tan, Mengmeng Ge, Di Guo, Huaping Liu, and Fuchun Sun. Knowledge- em- bodied question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):1 1948-1181111 1960, 2023.
[34] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models, 2025.
[35] Muhammad Awais, Tauqir Ahmed, Muhammad Aslam, Amjad Rehman, Faten S Alamri, Saeed Ali Bahaj, and Tanzila Saba. Mathvision: An accessible intelligent agent for visually impaired people to understand mathematical equations. IEEE Access, 2024.
[36] Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. arXiv preprint arXiv:2411.166 165377, 2024.
[37] Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, and Jincai Huang. Cityeqa: A hierarchical Ilm agent on embodied question answering benchmark in city space. arXiv preprint arXiv:2502.12532, 2025.
[38] Lingfeng Zhang, Yuening Wang, Hongjian Gu, Atia Hamidizadeh, Zhanguang Zhang, Yuecheng Liu, Yutong Wang, David Gamaliel Arcos Bravo, Junyi Dong, Shunbo Zhou, et al. Et-plan- bench: Embodied task-level planning benchmark towards ppatial-temports cognition with foundation models. arXiv preprint arXiv:2410. 14682, 2024.
[39] Kaixuan Jiang, Yang Liu, Weixing Chen, Jingzhou Luo, Ziliang Chen, Ling Pan, Guanbin Li, and Liang Lin. Beyond the destination: A novel benchmark for exploration-awan aware embodied question answering. arXiv preprint arXiv:2503. 11117,, 2025.
[40] Genesis Authors. Genesis: A universal and generative physics engine for robotics and beyond, December 2024.
[41] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828 9928-5839, 2017.
[42] Anthony Brohan, Noah Brown, et al. Rt-1: Robotics transformer for real-world control at scale, 2023.
[43] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint #xXiv2243.005330 2024.
[44] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[45] Aixin Liu, Bei Feng Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 11X1v22412.19447, 2024.
13

---

[46] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402. 10329, 2024.
[47] Qwen Team. qwen2.5, September 2024.
[48] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023.
[49] Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset empower large language model training, 2025.
[50] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- Sce-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024.
[51] Pan Lu. Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IICNLP 2021), 2021.
[52] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615. 2025.
[53] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 4rX1v:2303.08774, 2023.
14

---

A More details of CLEVR-AVR benchmark
Figure A-1 shows examples of the three scene types in the CLEVR-AVR benchmark, Figure A-2 displays the question template settings for different question types, and Figure A-3 illustrates the distribution occlusion and stacking quantities in each of the three scene types.
A.1 Action Candidate Generation in CLEVR-AVR
The available action space includes Pick, Move Viewer, Rotate Viewer, and Move Object. Beyond the diverse scenarios, the benchmark incorporates a rich variety of question types, including Query, Exist, Counting, Compare, Math Counting, and Math Compare. Agents must utilize the provided Franka robotic manipulator and camera controls actively uncover essential details, thereby tackling the challenge of efficient exploration under conditions of partial observability. The agent communicates its decision by generating text that includes its reasoning (CoT) and the selected action, formatted for example as <action>E</action>> where E would map to specific action like Pick(yellow cube) from the candidate list.
At each interaction step in CLEVR-AVR, the agent is presented with 5-8 candidate actions. The types of actions (e.g., Pick, Move Viewer, Rotate Viewer, Move Object) are predefined. However, the target objects for actions like Pick or Move Object are dynamically selected based on the current visual scene, often including objects relevant to resolving potential occlusions or ambiguities. To rigorously test the agent's 's reasoning-driven action selection, approximately 3-5 of these candidate actions are intentionally designed distractors sub-optimal choices that would yield less information gain towards answering the question. This forces the model to not just pick any valid action, but to strategically select the one most likely resolve its current uncertainty.
B More details of AVR-152K dataset
Figure A-4 shows the system prompt used for generating AVR-Caption data. An example of an AVR-Caption data instance, featuring an image and its corresponding dense caption, is presented in Figure A-5
For the AVR-Embodied Reasoning subset, Figure A-6 displays the prompt used for DeepSeck-RI to generate initial reasoning, and Figure A-7 illustrates the prompt for DeepSeek-V? V3 to refine this Chain-of- Thought (CoT) reasoning. representative data instance from AVR-Embodied Reasoning, which includes multi-image sequence, question, and the associated reasoning chain, can be seen Figure A-8
The prompt provided to Gemini for refining human expert Shaiin-ff-Thoughe annotations within the AVR-Core dataset is detailed in Figure A-9 This prompt utilizes placeholders such (question), {options), J. (answer), Viisual_rassoninn are filled with content from expert human annotations. To concretely illustrate the rich, multi-step interactive reasoning process captured in AVR-Core, Figures A-100 A-11 and A-12 collectively showcase complete, sequential example from an AVR-Core instance.
(hypothesis). again_prediction) and (planning). which
B.1 Initial Generation of Expert CoT Annotations for AVR-Core
The Mhain-of-Thoughe (CoT) annotations in the AVR-Core dataset, prior to their refinement by large language models like Gemini, were meticulously crafted by human experts. The process involved two main stages:
Live Task Execution and Initial Logging: Human experts actively performed the interactive tasks using the UMI devices in real-world tabletop settings. During this live interaction, they made contemporaneous notes capturing their key reasoning steps, uncertainties identified, hypotheses about hidden information, and the rationale behind their intended actions at each decision point.
Post-Inted Dost-Interaction Refinement and Structuring: After completing each task, the experts revisited their initial logs alongside the complete record of the interaction (including all visual observations and executed actions). They then elaborated on their initial notes, structuring them into the detailed, step-by-step CoT format that reflects the iterative process of assessing information, predicting gains from potential actions, and making a decision.
15

---

This human cenaric initial annotation phase was crucial for ensuring that the CoTSs genuinely reflected plausible and effective human reasoning strategies for active information gathering, forming high-quality foundation for subsequent automated refinement and model training.
Specifically, Figure AA-10 depicts the initial state (Step 0) of an active visual reasoning scenario. Here, based on the initial visual observation and the posed question, the agent's CoT reflects its assessment of information insufficiency and its decision to take an action ("Move the yellow object to the left") to acquire more details.
Figure A-11 shows the subsequent state (Step 1) after the execution of the first action. The agent re-evaluates the situation with the new visual input (IMAGEI), and its CoT again indicates the need further information to fully resolve the question, leading to the planning of another action ("Move the green object to the right").
Finally, Figure A-12 illustrates Step 2 of the process. After the second action and observing IMAGE2, the agent has gathered sufficient visual evidence, and its CoT culminates in providing the final answer ("Yes") to the guestion.
Together, these three figures (Figures A-10 A-12)
highlight the core principles of AVR-Core: the iterative cycle of identifying uncertainty, predicting information gain conditioned on potential actions, and making strategic decisions, all articulated through detailed CoT annotations.
C More details of Model and Training
Figure A-13 shows the detailed training parameters for PhysVLM-AVR-3B and AVR-Qwen. VL-7B, including input image resolution, trainable parameters, batch size, maximum token length, learning rate, and number of epochs. We conducted the training
Ubuntu server equipped with p NVIDIA A800 GPUs. The main software used was Py Tyrch=2.6600 transformers=3.72..0 3.72.00 flash attention2, and DeepSpeed. Figure 1-114shown the architecture of the PhysVLM-AVR.
D Baseline Model Input Prompt for CLEVR-AVR Experiments
In experiments the CLEVR-AVR benchmark, the inputs for the compared baseline models prompts that include AVR task instructions and cues for potential actions. The input template is as follows:
You are required to perform active visual reasoning: when the information obtained from image observations is insufficient to answer the question, you need to make action decisions to interact with the environment in order to acquire additional visual information relevant to the question. You should continue gathering new observations until you can infer and summarize a reliable answer based on the accumulated visual history. Choose either answer to the question or an action decision option from the options above. Final option choice follow this format: (your maa.yyia.....Kssseee//////CC/ http:////....///eerr </answer>
analysis) www.s//ea/////////////////
E
Code Availability
The code for our project is available anonymously at the following link: Copennssceence////nonnnnnoo je99tt
Ethical Considerations and Usage Restrictions
Large Language Models (LLMs) and related AI technologies possess the potential for significant societal impacts, both beneficial and detrimental. While they offer capabilities to enhance productivity, creativity, and access to information, crucial acknowledge the inherent risks. These risks include, but are not limited to, the generation and propagation of misinformation, the amplification of
16

---

existing societal biases, potential for job displacement in certain sectors, and the possibility of misuse for malicious purposes.
The code, data, and models provided this work are intended strictly for research and development purposes. They must not be employed in any high-risk applications for activities that could result in harm, discrimination, infringement of rights, or any other negative societal consequences. Users of these resources bear full responsibility for ensuring their applications comply with all applicable laws, ethical guidelines, and responsible Al practices. We explicitly disclaim liability for any misuse of our code, data, or models. We encourage a cautious and ethical approach the development and deployment of AI technologies.
Occlusion
Stacking
Composite
Figure A-1: Examples of the three scene types in the CLEVR-AVE benchmark.
17

---

Question Type
Tomplatss
eWhat is the object with the smallest conterllo-ecnic distance from the ttop, onyy color) stop, stine) Deep, onyy slapp) allppe
top??
Query
FWhht (ant type) the object elosest the (froer _obj.color) (fiott objj.izee [fions, shape)?"
("hhtt type]
object
(dinection]?"
Exist
Ils there small Egrkambbcck ohi, air typeli) that is igeattbbbckk obj, sitr typo2))?
Hooww many objects
value)?"
Count
I'n the stack with (top_od color) cube on top, how many cubes are (aur value)?"
(Howw many objects fatr value) in the (anea] area?"
Compune
the number (atr value] objoccts
area right area equi!?"
('Which azea has [computison) (attr value) objects?"
type]
deseripiinn object the
description) objoct?
Ils the mumber of (atr volue) objects the left aren and the right aren equil"
('Subtract (randomn, shape) (rundom, objects,
value) objects remain!"
('Subtact frandoon, shape) (random, size) objects, how many (astr value) objects remain the [area] area?"
I'A (random, number] [str_ value] objerss, how [atr value) déjects would there be?"
MathCount
(random number) (ette value)
stack (top, onler on top, many fott valise) cubes
the stack?"
('Subinct (randocn, shape) [random size] and (mandom shape2) (random size2) objects, how many (attr value] objects remain?"
d'Add (random number) (ater value) objecss, how many (att sulue) objects would there be?"
(Subttnct all (randoms shape: [andom, size] objocts, the (atr [tpo) the center eabe the stack with ttop m___oby color) cube top the the (nndom description) object?"
MathCompure
("Subtrcc frandomn, shape) [random_ size] and (madom [manoo_shapp2] (random size2) objects, which area would have [comparison] (atte value) objocts!
Figure A-2: Question template settings for different question types in the CLEVR-AVR benchmark.
Number of occlusion stack
Scone
Somee
Scene ID
(a) Compusite
(b) Occlusion
(c) Stack
Figure A-3: Distribution of occlusion and stacking quantities across the three scene types in the CLEVR-AVR benchmark.
18

---

PROMPT CAPTION COMPT_CAPTIONN ...
Your tasks are: I. Provide a Detailed, Systematic, and Well-Organized Description Entire Seene:
Describe the secne from macro to micro perspective, moving from the overall view to specific regions then to individual objects in step-by-step manner.
When describing objects, be thorough, including not limited
Color
Shape and Structure
Material and Texture
State (e-g,, on/off, neat/messy, use/idle)
Estimation (relative approximate size)
Position and Spatial Relationships (relative other objects)
Determine the relative hierarchical relationships between objects, their distribution of distance, and layering.
** IIteeacction and Usage Context** (if applicable); If the combination objects context (e.g., bowls, chopsticks, and cups dining table), reasonably infer and describe the possible purpose the activity taking place.
suggests particular usage
### Analyze Current Observation Perspective (Camera Position) and the Relative Positional Relationships and Spatial Distance Estimates with Other Objects, and Deseribe Orientation the Current Viewpoint:
### III. Generate 2D Object Detection Results:
Perform 2D detection all recogmizable objects the image.
For each object, output its bounding box in the format min, x_ max, x_ j_mux]: Add 'deseribe' field for detailed description of the object distinguish between similur objects.
**Seene Description** **
(Fill in the detailed seene description here)
*+2D Detect **
"json
[t
"label": "*
"bbox": [].
"describe":
Figure A-4: System prompt for generating AVR-Caption data.
19

---

"image": "aw.lOXE/brdgge/brigg..ol.. 0.jpp".
"conversations".
"from": "human",
"value": "RGB MMAGE:nrimagee>nnPPro a thorough and organized depiction of the scene utilizing RGB, starting with the overall layout and then detailing specific objects, their attributes, and spatial relationships. Incorporate any suggested interactions or contexts use. Analyze the camera's viewpoint and the relative positions Generate 2D object detection results with normalized bounding boxes descriptive labels JSON format."
"from": "gpt",
"value": "Here's a breakdown of the scene, object detection results, and viewpoint analysis based the provided images. "nnees.nnnnnnnn Scene Description nn'nn The scene depicts play kitchen setup. The backdrop consists a wall the left. The main focus is on the countertop of the play kitchen. "Cunterror The countertop surface has light brown, wood-grain pattern, resembling butcher block. appears constructed from horizontal planks. On the countertop are several play kitchen items:inin* **Metal Pou** A small, shiny silver pot sits the left side of the countertop. is round and has small handles either side. appears to made stainless in camera appears to be positioned at medium height and angle, looking slightly down at the play kitchen countertop. The viewpoint is slightly to left center, providing good view all objects on the counter. The perspective is relatively close, allowing for clear details of the objects and their arrangement.uuivn II' 2D Detect Results* In''n"" "ison['n I'n v'labell": V"pot"", In \"bbox!" ['m
faux wood paneling design top and blue-gray textured surface on
**II. Analysis Observation Perspective**
0.313, 0.451,An
0.611,
0.71'm 1,In \"describe\": \"Small, silver, stainless \"labeli": Vecoth!",, \"bboxi": [In 0.435,In
steel play with two handles.\"'n ),1m (in
0.302, 0.714,nn
0.548'n 1An \"describe)": ("Flded yellow cloth. J,in fin 0,83,'n "describel": B"Baack plastic play faucet hanging above the countertop.!"nn Yn]"""
V'labell": ("corl"..nn In
\'bboxl: [in
0.742,
0.391,In
0.532 I.... L.m
Figure A-5: Data instance of AVR- AVR-Caption.
20

---

PROMPT REASONING fuee
Multiple images in the ulong with task resktrelated descriptions and object bounding boxes provided your visual observations, You need combine the observations from all imuges questions ubout the entire (question]
Image descriptions:
(caption)
Note that all images are taken in the same scene, the same object may appear in different images. The final answer should be phrase, word, other concise and clear response (if the answer cannot be found, respond with "Unable answer"), following the format similar to <answer>a sofa, coffee table, s//nswrr>>
Figure A-6: Prompt for DeepSeek- RI reasoning in AVR-Embodied Reasoning data.
PROMPT CoT REFINE pm
**Enhance and Modify the "Chain-of-Thoughee (CoT) by Integrating Scene Captions, Adhering to the Following Guidelines::*
STYLE TRANSFORMATION (Most Important):** Replace phrases such as "based image description. "based on scene caption," "based on detection results" "explicitly described as" with "the image shows," detected." Rewrite CoT the FIST PERSON, focusing on image observation, analysis, and reasoning. Don't let readers feel that you have obtained the caption detection results of the scene with IMAGE, rather that you have analyzed them image.
from the visual information
Sequential Image Summarization:* For each image (IMAGEL to IMAGEn), provide a sequential summary of main content within CoT, even specific information related to the question isn't visible. Avoid statements "xxx docs not appear in IMAGEn-t.
**Retain Object Bounding Boxes:** Maintain object bounding box information each image within the CoT to preserve spatial and contextual details. peae--bbbeettiiiinnn ymin, xmax, ymax]</boxx" <<box>>>
4. Enhunce Logicul Coherence:** Ensure CoT flows logically, with clear connections between observations inferences, improving the overull coherence reasoning process.
**Manttain Tag Integrity:** Preserve integrity tags like '<think>", '</think>", "annwerr" <lanswers' that the "answwer"" tag remains concise, retaining its original brevity.
Scene Captions:** {seene cantienss
#Chain-of/Thouuhh (CoT))*** _ons_arssrrr
Figure A-7: Prompt for DeepSeek- V3 to refine Chain-of-TTe Thoughh (CoT) reasoning in AVR- Emn-mmiddied Reasoning data.
21

---

"scene id": "scene0000 01",
"images"::
"scenc0000 01//oolor/002400.jjgg
"scene0000 01/color/002500.jpgs
"scene0000 01/colorr0022000jjpp"
"scene0000 01/color:002700.jip""
"scene0000 01/color:002800.jpg""
"scene0000 01//oolor0002000.jpp""
"scene0000 01 0f/color/000000..ig""
"scene0000 01/coolo/0031000.jps
"question": "Where the shoes located?",
"answer": "<<think>\nn 'nl.ct me analyze images to determine where the shoes are located the scene. Since all images are part of the same environment, I'll examine each one sequentially to piece together the answer. 'n'nstti # **IMAGE0:** 'n- **What I observe:** In The image shows a room with light green curtain on the left side, hanging from the top to the floor. \n -I detect two pairs of dark-colored shoes placed near the base of the curtain (<box>shoes:(0.. [0.04, 0.62, 0.12, 0.77j</box> and <box>shoes:(0 10, 0.6, 0.24, 0.7)</box>).
of
floor has light-colored tiles, and bicycle partially visible in the upper right. In'nnffilftt +IMAGEE::* 'n- **What observe:** \n The image shows a living space with bicycle leaning against the wall, black air pump nearby, and backpack the floor. in -I detect cabinets, a couch, and curtains but no shoes. These images focus different angles of the room, primarily showing sofas, ottomans, remote controls, and backpack.
And.te **MAGG4-IMAAA
'n-
**What observe:**
do not detect shoes in any of these frames. \nin#uu 'LLgiiaal Reasoning:** 'n- Since *MMAG0 is the only image where clearly see shoes, and all images belong the scene, the shoes must remain in their original position
the curtain. 'n- The other images either do not capture that area or focus on different parts of the room. \n- Therefore, the shoes are consistently placed at the base of the left-side curtain, as observed in **MAGE0**. in'n' aatiink=/n/ansswerreeee the base the curtain side</answer>""
Figure A-8: Data instance of EVR-Embodied Reasoning.
22

---

PROMPT_ CoT REFINE
Active visual reasoning: When the information obtained from image observations enough question, action decisions need be made interact with the environment oblain visual information related to the question. New observations continuously collected until reliuble cun inferred and summarized based the accumulated visual history.
Question: (question)
Choice: (options)
GT answer: (answer)
CoT #4l Goal Analysis
Wil Visual Reasoning
(Detailed description of iiems on the table, especially task-relevant items)
(visual reusoning)
## Uncertainty Hypothesis
[hypothesis)
## Information Gain Prediction (Analyze the information gain that would result from each action option) igain |
##l Planning Pseudocode
(Based on the reasoning in this step. plan the subsequent logic after this action) {planning]
#Nl Conclusion
(Summarize the information from visual reasoning, uncertainty assessment, and information gain predictions, including key items' point and bbox 2d spatial position information.)
You need to earich the CoT reasoning process. The CoT should include GT prompts, should not change the first (# CoT) second level structures, and should optimize reasoning logic each section (more efficient, rigorous). Replace the planning section with concise pscudocode. further action needed, simply write END. Use key points pointly, x)' bounding box 'bbox 2d(ymin, xmin, ymax, xmax)' in the assist visual reasoning.
mmm
Figure A-9: Prompt for Gemini to refine Chain-of-Thouue Thought annotations in AVR-Core data, where (question), (options). ), (answer), { (visual, reasoning). (hypothesis), (gain prcdin__rcctioo), and {plan- ning) all derived from expert human annotations.
23

---

StepO
"question type": "Existenee",
"image": ["avriavt-core/rruupl I/group11 10000-1rr0ppp11111 rgb_0-nnit.jpg"1
"conversations":
"from": "human".
is the initial observation. 'n'nYou required perform active visual reasoning: when the information obtained from image observations insufficient to answer question, you need to make action decisions to interact with environment order to acquire additional visual information relevant to the question. You should continue gathering new observations until you can infer and summarize a reliable answer based on the accumulated visual history.In lnuucstinnt Is there an object of the same size as the white small-sized block and further back in the desktop and its appendage/'atChoiinn not exist, C: Open drawer, D: Close drawer, E: Grasp purple small-sized block, F: Move lime the left, G: Grasp blue large-sized block'nChoose either an answer question action decision option from the options above." "). "from": "gpt",
"value": "IMAAGG0:-iaaggeNIMMAAM..
A: Exists, B: Does
"valuc": "<think<"mttt Goal Analysis'n in The user wants to determine there is any object on the table or inside the drawer that has the same size as the white small-sized block "bbox_ s_g_d_00066 (2, 0,45, 0.71, 0,544) and is located further back (greater y-coordinate, considering perspective) than the white block inimii Visual Rcasoning'nIn the initial image (IMAGE0), we observe:'n* A white small-sized block 'bbox 2d(0.62, 0.45, 0.71, 0,544) An* A purple small-sized block at bbbex 2d(0.531, 0.496, 0.614, 0.603) Ann orange small-sized block at 'bbox 2d(0.681, 0.528, 0,782, 0.637) !n..... The drawer "bhox_ 24(0.0, 0.141, 0.447, 0.528) is also further back in parts, but it is closed, so its contents are unknown.'ni 'nin H* Uncertainty Hypothesisinl. The lime "bbox 2d(0.533, 0.243, 0.698, 0.406) might be occluding an object of the same size the white block "bhox 2d(0.62, 0.45, 0.71, 0.544) located behind it./22. The closed drawer 'bbox 2d(0.0, 0.141, 0.447, 0.528) might contain object of the size as the white block 'n'n Information Gain Prediction' C: Open drawer Would resolve uncertainty hypothesis High potential information gain regarding objects inside the d D: Close drawer - The drawer is already closed; no information gain.'r \n E: Grasp purple small--szzed block - Does not help check areas further back or inside the drawer. Low information gain. "e..n.i"............ .......nCommmi Co.oppriigg C and F, moving the lime (F) directly addresses the occlusion in the area behind the blocks, which is primary location check based on the question (ffurther backi").
Opening the drawer (C) checks another potential location. Moving the lime seems slightly more direct for checking the immediate area behind the visible blocks. inint Planning Pseudocodein' pseeddoodeenMOVE limein""'ninnii"" "nniiii Conclusion'''Tee white block is located at 'bbox 2d(0.62, 0.45, 0.71, 0.544). The lime 'bbox 2d(0.533, 0,243, 0.698, 0.406)' is further back but larger. The lime occludes the area behind it. The drawer 'bbox 2d(0.0, 0. 141, 0.447, 0.528) is closed and could contain relevant objects. To resolve the uncertainty about potential objects behind the lime, the best action is F: Move lime to the left. /ns/tinkk>>> <answer>>nn wwteerrnnna////nnnaaeaa /answerrr>
Figure A-10: Data instance of AVR-Embodied Reasoning. Step of the active visual reasoning process, which make action decision to get more information.
24

---

Step1
"question type": "Existence",
"image": "vvraavr-coreggouup wtwpooneeggguul//lggggggg //group 0/0-groupl! Lar_oll_ititt__ppppe O-init.jpg".
avvraavr-coreggooul l/group 0/ropll l_rbb 0-F Move lime.jpg"
"conversations":
"from": "human".
"value": MMM.:00-mmaeeMMMMMMMMMM----0 MIMAAGO is the initial observation. ...... .....nQuuetton:: Is there an object of the same size as the white small-sized block and further back the desktop and its A: Exists, B: Does not exist, C: Open drawer, D: Close drawer, E: Grasp purple small-sized block, F: Move Time the left.....InChoose either answer to the question or action decision option from the options above."
"from": "gpt".
"valuc": "sthink=-nntt Goal Analysisin objective is determine if there any object on the table or its appendage (the drawer) that has the size the white small-sized block and.....## Visual Reasoning'nhn the current observation (IMAGEI))n* The white block located 'bbox 2d(0.62, 0.45, 0.71, 0.544). ..... 'n Uncertainty Hypothesis'n only remaining possibility for an object meeting the criteria inside closed drawer. The drawer itself positioned further back than the white block. It is uncertain whether the drawer contains an object of the same size as the white block.In
## Information Gain P C: Open drawer This action will reveal the contents of the drawer, directly addressing the uncertainty and allowing definitive answer to the question. High information gain. An D: Close drawer- The drawer alrcady closed. No information gain,'n E: Grasp purple small-sized block Irrelevant finding an object further back. No information gain.'nt In Move
the left This action was already performed. No further gain. 'n* G: Grasp blue large-sized block Irrelevant to finding an object further back. No information Daiin.nnnnOOennng the drawer (C) is the only action that can resolve the current uncertainty.. An'nff An' Planning Pseudocode'n" pseudocode nOPEN drawer'nn "nintt Conclusion'a. Affter moving the lime, visible object the table surface 0.544) located further back. The only remaining unexplored location where such object might exist is inside the closed drawer 'bbox 2d(0.0, 0.141, 0.447, 0.528y, which positioned further back. Therefore, next necessary action is to open the drawer to check pontents.ns/thinkk Sanswer>inC'n<//nswe
the criteria of being the size the white block 'bbox 2d(0.62, 0.45, 0.71,
Figure A-11: Data instance of Embodidd Reasoning. Step of the active visual reasoning process, which make action decision get more information.
25

---

Step2
"question type": "Existence",
"image":
"avriavr-core/group)) //group |_0/--groupl | |
avvrian--cor//goupl! /groupl1 0/1-group1] rgb 0-F Move lime.jpg".
"avraar--coreggouupll 1 0/2gggooppplll __rgb___ 0-C Open drawer.jjg"
"conversations":
"from": "human".
wn......nunuuttoon: Is there an object of the same size as the white small-sized block and further back in the desktop and its appendage? Exists, B: Does not exist, C: Open drawer, D: Close drawer, E: Grasp purple small-sized block, F: Move lime to the left....... "nChoose either answer the question or an action decision option from options above."
"value": MAGGEO-iiag--inIMAAGG
3.
"from": "gpt".
"value": "stthink=inétt Goal AnalysisinThe objective is determine if there exists object on the table its appendage (the drawer) that is the same size the white small-sized block An!!! Visual Reasoning'nll current observation (IMAGE2))n* The white small-sized block 'bbox 2d(0.62, 0.45, 0.71, 0.544). Its center is approximately point(00666, 0.497) .......
## Uncertainty sypothesis'nIhee primary uncertainty was whether the closed drawer contained an object meeting the criteria. Opening the drawer (Action C) resolved this uncertainty by revealing its empty interior. There are no remaining uncertaintics unexplored areas relevant to the question. in ## Information Gain Prediction'nNo further actions are needed. All relevant like closing drawer, grasping blocks, moving the again will not provide new information relevant finding an object *further back* and the same size* as the white block. The answer can determined from current visual information. nnin Planning Pscudocade observations, including the initial state, moving the lime, and opening the drawer, all potential locations have been examined. The lime 'bbox 2d(0.435, 0.259, 0.597, 0.387) is further back than the white block 'bbox 2d(0.62 26(0.62, 0.45, 0.71, 0.544) but back, was opened and found to empty. No other objects are located further back the white block.
have been observed. Actions
Aimclusion'nBaseee on the sequence of
the size. The drawer 'bbox 2d(0.0, 0.142, 0.542, 0.679),, which is also further
Therefore, there is no object of the same size as the white small-sized block located further back. The answer is B: Does not exist./n=/think>> <answer=>nBBi Ban//sseerr"/
Data instance of AVR-Embodied Reasoning. Step of the active visual reasoning process, which getting the final answer.
Figure A-12:
26

---

PhysVLMM-VR--1 training details
Stagel: Feature Alignment
Stuge2: Visual Understanding
Temporal Visual Understanding
Stage3: Basic Reasoning
Active Visual Reasoning Abilities
Stagel
Stage2.1
Stage2.2
Stage3
Image Resolution
384
384
384
384
Trainable
Projector
Full Model
Full Model
Full Model
Batch Size
32
16
Max length
2048
2048
4096
8192
LR (all parameters)
le-3
le-5
1e-5
2e-6
Epoch
AVR-Qwen2.5-4 5-VL-77 fine-tuning details
Val dataset
Datasets
Trainable
Batch Size
Max length
Epoch
CLEVR-AVR
AVR-Core
LLM
8192
5.0e-6
OpenEQA and Robo VQA
All AVR-152K
LLM
16
8192
5.0c-6
Figure A-13: Training configuration details for PhysVLM-AVR-3B and AVR-Qwen2.5-VL-7B.
<hhnnk.... wWhinkeiinsnnswerr </answer>
Large Language Model (Qwen2.5)
Projection Layer
(2xMLP)
Dawn-Sample (MaxPooling)
Vision Encoder (SigLip-400M)
Tokenizer
T
Figure A- 14: Model architecture of Phys VLM-AVR.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We reflect the contribution and scope of the paper at the end of the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

This article does not involve theoretical results that require proof of assump- tions and reasoning.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in Experiment, Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We use anonymous code links that follow submission guidelines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.or cc/
public/ http:/guudess/cceeeeee
ss Policy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
httpscccccccccccccc i ies CCodeeoodSuaaee bm ss i oonool icy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide information on data splitting in Experiment and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experiments in the article were averaged multiple times, and the random seeds in the experiments were set to 42.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information in Experiment, Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper complied with the NeurIPS Code Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss this in Appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We reference the relevant code and models respecting the license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, baapeettspppersssee paperswws thoode. com datasett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
32

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper don't 't release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This article does not involve crowdsourcing experiments and studies with humans as subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Not applicable this study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
33

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non- noa-standard components.
Please refer to our LLM policy btttps://neuripe. wwtConfereecee///eeeccccooaaea LLM) for what should or should not described.
34


Reviews:
[{"review_id": "BDUsHUFqjm", "reviewer": "Reviewer_CyMs", "summary": "The paper proposes a new task, Active Visual Reasoning (AVR), which requires agents to actively interact with the environment to obtain enough information to answer the question. To evaluate, the paper proposes CLEVR-AVR benchmark that extends CLEVR using Gensis. Also, the paper introduced AVR-152k with CoT annotations and used it to train a PhysVLM-AVR in the higher-order Markov Decision Process. The introduced PhysVLM-AVR is well validated on multiple benchmarks to show the powerful performance.", "questions": "See above", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The proposed AVR task is relatively novel. The novelty is sound. \n\n2. The paper proposes a comprehensive pipeline, including a dataset, a benchmark, and open-source model checkpoints. I believe the community will benefit from this solution. \n\n3. The experiments are extensive, and the workload is sufficient. Also, the paper writing and the figures are quite clear.\n\nOverall, this paper does not have major weaknesses. The overall experiments and claims are quite sufficient.\n\n## Weaknesses\n\n1. Although Figure 1 shows that passive reasoning fails in the occlusion scenarios, it still raises concerns about the necessity for active reasoning. The paper seems to mention only two active reasoning tasks: occlusion and stack (and the composition of these two). Also for the stack, it is still possible to get the correct reasoning based on static images alone. So, are there more applicable scenarios for active reasoning that passive reasoning fails? Is active reasoning also necessary for tasks where static images can already show enough information?\n\n2. CLEVR-AVR is a synthetic benchmark, and is hard to transfer to real-world scenarios. And this issue is more important for the proposed active reasoning task. Also, can the introduced PhysVLM-AVR-3B operate in real-world scenarios?\n\n3. The AVR-Core dataset seems to be very useful in the ablation. However, this subset only contains the tabletop scenarios. The diversity is a concern.\n\n4. How is the IGR calculated? What is the definition of the information-gaining decision? The article lacks sufficient details for this part.\n\n5. The paper may contain a detailed failure case study. The authors may analyze the issue in detail, that the ISJ of the model is really high, but the Final Answer Accuracy is still low. Could this be because the model does not understand the visual scene well?", "was_revised": true, "final_justification": "I have carefully read the rebuttal and convinced that this paper should be accepted.", "rebuttal": "### **Response to Reviewer CyMs**\n\n**Dear Reviewer CyMs,**\n\nThank you for your meticulous review and insightful comments on our work. Your questions regarding the necessity of active reasoning, the sim-to-real gap, and dataset diversity directly address the core challenges of our research and have helped us to further refine our understanding and presentation.\n\nIn response to your feedback, we have provided additional analysis and clarification on several key points, including the expanded scope of active reasoning, the connection of our work to real-world scenarios, a precise definition of our evaluation metrics (IGR), and a detailed failure case analysis. These are detailed in the point-by-point responses below.\n\nThank you again for your valuable feedback.\n\n---\n\n### **Q1: On the Necessity and Scope of Active Reasoning**\n\n> **Reviewer's Comment:** [...] Are there more applicable scenarios for active reasoning that passive reasoning fails? Is active reasoning also necessary for tasks where static images can already show enough information?\n\nThank you for this insightful question, which is crucial for clarifying the value of our work. We argue that the necessity of Active Visual Reasoning (AVR) extends far beyond resolving simple occlusion and stacking, and we frame its importance on three levels:\n\n1.  **Expanded Application Scenarios:** Active reasoning is indispensable in numerous scenarios where passive observation is inherently insufficient:\n    *   **Limited Field of View:** Tasks that require information from all sides of an object, such as checking for labels on the back of a container or inspecting its contents.\n    *   **Dynamic Environments:** Verifying the state of an object after it has been moved or interacted with by another agent.\n    *   **Functional Reasoning:** Determining an object's affordances, such as checking if a drawer can be opened or if a container is empty, requires physical interaction.\n\n2.  **Enhancing Robustness in Static Scenes:** Even when a static image appears complete, active reasoning serves as a critical verification mechanism:\n    *   **Disambiguation:** A slight nudge can differentiate whether two objects are merely adjacent or are partially overlapping.\n    *   **Hypothesis Verification:** Actively changing the viewpoint can rule out visual illusions caused by lighting, shadows, or specific perspectives, thereby increasing the robustness of the final conclusion.\n\n3.  **Simulating Higher-Order Cognition:** Human cognition often involves a **hypothesis-verification loop**, even in seemingly complete scenes (e.g., leaning in to read fine print). AVR aims to model this fundamental cognitive process, moving beyond a simple tool for \"filling in missing information\" to become a general strategy for robust and grounded reasoning.\n\n---\n\n### **Q2: On the Sim-to-Real Gap and Real-World Applicability**\n\n> **Reviewer's Comment:** CLEVR-AVR is a synthetic benchmark, and is hard to transfer to real-world scenarios. [...] Also, can the introduced PhysVLM-AVR-3B operate in real-world scenarios?\n\nThank you for raising this critical point about real-world transferability. We address this concern from two perspectives and provide **new experimental evidence** to demonstrate our model's real-world capabilities.\n\n1.  **The Scientific Value of a Synthetic Benchmark:** The primary purpose of CLEVR-AVR is to serve as a controlled environment to **precisely isolate and evaluate core agent capabilities**, such as information-gathering efficiency, multi-step reasoning, and uncertainty estimation. This allows for rigorous scientific analysis, free from confounding variables like complex textures and lighting found in the real world.\n\n2.  **Bridging the Gap to the Real World:**\n    *   **Real-to-Sim Generalization:** A key aspect of our work is that the `AVR-Core` dataset is collected entirely from real-world physical interactions. Our model, trained on this real-world data, demonstrates effective active reasoning capabilities in the unseen simulation environment (CLEVR-AVR). This itself is evidence of promising `real-to-sim` generalization.\n    *   **New Real-World Experiments:** To directly address your concern, we conducted 20 new sets of experiments in real-world tabletop scenarios with task setups not seen during training. In these experiments, `PhysVLM-AVR-3B` achieved a Final Answer Accuracy (ACC_FA) of 40.2% and an Information Gain Rate (IGR) of 30.9%. This performance is consistent with its behavior on the CLEVR-AVR benchmark and provides strong evidence that our proposed method can operate and solve tasks in real-world settings.\n\nWe fully agree that scaling up the diversity of real-world training data is crucial for broader generalization, and this is a key direction for our future work.\n\n---\n\n### **Q3: On the Diversity of the AVR-Core Dataset**\n\n> **Reviewer's Comment:** The AVR-Core dataset seems to be very useful in the ablation. However, this subset only contains the tabletop scenarios. The diversity is a concern.\n\nWe appreciate your concern regarding dataset diversity. Our design choice was deliberate, and we explain our rationale from two perspectives:\n\n1.  **Tabletop Scenarios as a \"Minimal Viable Unit\":** We position the tabletop environment as a **\"minimal viable unit\"** for studying active information gathering. This approach is analogous to how the original CLEVR dataset used simple geometric shapes to isolate and study core visual reasoning skills. While simple, tabletop scenes contain the fundamental physical interactions central to our research—occlusion, stacking, and spatial relationships—allowing us to focus on and rigorously test an agent's ability to close the perception-reasoning-action loop.\n\n2.  **Future Work and Scalability:** Our work on tabletop scenes establishes a strong methodological foundation. Having demonstrated the effectiveness of our approach in this controlled setting, our roadmap includes extending this paradigm to more complex and diverse environments, such as kitchen manipulation, warehouse sorting, and lab assistance. We believe this work provides a solid starting point for the community to build upon and apply active visual reasoning to a wider array of scenarios.\n\n---\n\n### **Q4: On the Calculation Details of IGR**\n\n> **Reviewer's Comment:** How is the IGR calculated? What is the definition of the information-gaining decision?\n\nThank you for pointing out the need for clarification. A precise definition of our metrics is essential.\n\nThe **Information Gain Rate (IGR)** measures the efficiency of an agent's exploratory actions.\n\n*   **Formula:**\n    ```\n    IGR = (Total Number of Information-Gaining Decisions) / (Total Number of Decision Steps)\n    ```\n\n*   **Core Concept: What constitutes an \"Information-Gaining Decision\"?**\n    > A decision (i.e., selecting an action) is defined as **information-gaining** if and only if its execution **reveals new, task-relevant visual information** that was previously unavailable to the agent.\n\n    This new information must directly help in reducing the agent's uncertainty about the correct answer. In our benchmark, this is determined objectively using the simulator's ground truth.\n\n*   **Examples:**\n    *   **Effective Decisions (Gain Information):**\n        *   Moving a cup to reveal a key hidden underneath (to answer \"Where is the key?\").\n        *   Changing the viewpoint to see the color on the back of an object.\n        *   Lifting a block from a stack to count the total number of blocks beneath it.\n    *   **Ineffective Decisions (No Information Gain):**\n        *   Moving the camera to a viewpoint that reveals no new relevant objects.\n        *   Interacting with an object that is already fully visible and irrelevant to the question.\n        *   Repeating a previous action that yields no new information.\n\n---\n\n### **Q5: On the Failure Case Analysis (High ISJ but Low ACC_FA)**\n\n> **Reviewer's Comment:** [...] The authors may analyze the issue in detail, that the ISJ of the model is really high, but the Final Answer Accuracy is still low. Could this be because the model does not understand the visual scene well?\n\nThank you for this excellent suggestion, which prompted us to conduct a deeper analysis of our model's limitations. Your intuition is correct.\n\nOur analysis of failure cases where the model correctly identifies the need for more information (high `ACC_ISJ`) but still provides an incorrect final answer (low `ACC_FA`) reveals two primary causes:\n\n1.  **Perception Errors:**\n    *   **Phenomenon:** The model correctly decides to interact with the environment to get a clearer view (e.g., moving an occluder), but upon observing the new, unobstructed scene, it misidentifies a key attribute of a newly visible object (e.g., its shape, color).\n    *   **Example:** To count all cylinders, the model successfully moves a cube occluding a target object. However, it then misclassifies the newly revealed short, wide cylinder as a sphere, leading to an incorrect final count.\n\n2.  **Temporal Integration Failures:**\n    *   **Phenomenon:** After a sequence of interactions, the model accumulates a history of multiple images and reasoning steps. However, during the final reasoning stage, it fails to effectively integrate the entire sequence of historical observations, sometimes forgetting or confusing details from earlier steps.\n    *   **Example:** In a task, the model correctly identifies a red cube on the left in step 1 and a blue sphere on the right in step 2. When asked, \"What are the shapes of the red and blue objects?\", it might only focus on the most recent observation (the blue sphere) and fail to recall the information from the first step, leading to an incomplete or incorrect answer.\n\nYour assessment is spot-on: these failures highlight remaining challenges in both **fundamental visual perception** and **long-horizon temporal reasoning**. Addressing these weaknesses is a key priority for our future research."}, {"review_id": "JUjMVDV5ZV", "reviewer": "Reviewer_F7gX", "summary": "This paper introduces the Active Visual Reasoning (AVR) task, in which an agent must proactively interact with its environment and perform visual reasoning. The authors present a simulation benchmark and accompanying dataset for training and evaluating agents on this task. They also develop a hierarchical learning model, PhysVLM-AVR, which is demonstrated to successfully tackle the defined tasks.", "questions": "1. In practical execution, how is the granularity of information gathering determined? How much information does the agent need to acquire before it can make a decision (i.e., is the agent more conservative or more aggressive)?\n\n2. In a scene with little occlusion, if the agent is conservative then both ACCISJ and IGR will be relatively low; if the agent is aggressive, then both ACCISJ and IGR will be relatively high, while their ACCFA may remain similar. In a heavily occluded scene, the outcome is completely different. Is this evaluation metric weighted in any way, or is there a better metric?\n\n3. In Fig. 1, what does “GPT-4o + AVR” refer to? It is not explained later in the text. Does it mean using GPT-4o to carry out the AVR paradigm defined in Section 3.1? In Table 1, the experiment lists “GPT-4o” rather than “GPT-4o + AVR”—why wasn’t “GPT-4o + AVR” used? Can the plain GPT-4o already outperform PhysVLM-AVR-3B and AVR-Qwen2.5-VL-7B?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes a novel and valuable task, with a clear and rigorous definition.\n\nIt provides a dedicated benchmark and dataset, enabling systematic evaluation of AVR agents.\n\nThe authors conduct experiments that validate the usefulness of the dataset and show that PhysVLM-AVR achieves strong performance.\n\n\nWeaknesses:\n\nDespite a relatively high ACCISJ score, the final answer accuracy (ACCFA) remains inadequate. The model still struggles to integrate information over multiple steps and to perform coherent multi-stage reasoning.\n\nThe description of baseline GPT-4o is unclear, which hinders the proper evaluation of the experiment results.", "was_revised": true, "final_justification": "I maintain my original assessment and score.", "rebuttal": "**Dear Reviewer F7gX,**\n\nThank you for dedicating your time to review submissions for NeurIPS 2025. We are the authors of Paper #23215, titled \"**AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments**\".\n\nAfter carefully reading your insightful questions, we believe there may have been a significant mix-up with another paper. Our work introduces **AVR**, a framework for Multimodal Large Language Models (MLLMs) to perform **active** information gathering in physical environments.\n\nYour questions, however, seem to be directed at a paper focused on **Visual Reasoning Reinforcement Learning (VR-RL)**, referencing specific components and results that are not part of our submission. For example, your questions mention:\n\n*   A **\"Table 1\"** comparing **\"Grounding-WorldModeling\"** with **\"WorldModeling\"**.\n*   An **\"RL algorithm\"** whose performance is compared against \"off-the-shelf VLMs\".\n*   A **\"visual reasoning reward\"** and a **\"Bi-Level GAE\"** architecture designed to ensure stable training.\n\nThese specific technical terms, experiments, and architectural components are entirely absent from our paper. To be clear:\n*   Our paper does not contain a \"Table 1\" with \"Grounding-WorldModeling\".\n*   Our proposed AVR framework is not an RL algorithm in the sense described, and we do not use components like a \"visual reasoning reward\".\n*   The term \"Bi-Level GAE\" is not mentioned or implemented in our work.\n\nWe understand that reviewing numerous papers is a challenging task, and we suspect your detailed review was mistakenly assigned to our submission.\n\nWe would be deeply grateful if you could take a moment to re-examine our paper. We believe you will quickly see that your current review does not apply to our work.\n\nThank you for your understanding and your invaluable contribution to the community.\n\nBest regards,\n\nThe Authors of Paper #23215"}, {"review_id": "Nyu5UgofnA", "reviewer": "Reviewer_dTPo", "summary": "The paper studies the active visual reasoning (AVR) problem with multimodal large language models (MLLMs). AVR goes beyond static visual understanding and requires actively acquiring environment information and feedback for task solving. To assess the capability of MLLMs for AVR, the authors formulate an AVR task, construct CLEVR-AVR, an AVR benchmark, introduce a dataset, AVR-152k, for developing the AVR ability of MLLMs, and train an MLLM, PhysVLM-AVR, on AVR-152k. The experimental results show that the model achieves the best performance on CLEVR-AVR while maintaining good performance on normal embodied reasoning and planning tasks, as well as static visual reasoning tasks.", "questions": "1. The multi-choice setup mixes candidate answers and actions; I wonder if this would introduce a bias towards choosing an answer or an action. Can the authors provide an analysis on this?\n2. The comparison between AVR-QWen2.5-VL-7B and PhysVLM-AVR-3B is unfair: 1) AVR-QWen2.5-VL-7B does not rely on the more complex multi-stage training that is applied to PhysVLM-AVR-3B; 2) the model sizes are not on the same scale, i.e., 7B is more than twice as large as 3B. Can the authors at least keep the same VLM backbone?\n3. GPT-4o, in general, outperforms all other models, including those trained on CLEVR-AVR, presumably because of its larger model size. Is it possible for the authors to train larger AVR-QWen2.5-VL and PhysVLM-AVR and check if AVR-pretraining remains helpful as the model size increases?\n4. The current experiments focus only on MLLMs, but in the context of embodied reasoning, models that can predict and execute actions (e.g., vision-language-action models) seem to be better candidate models to be tested, can the authors explain if it is possible to apply CLEVR-AVR to VLA evaluation and in which ways if so?", "limitations": "See the Questions section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## strengths\n\n1. The paper is well motivated and addresses an important limitation (i.e., active visual reasoning) of MMLMs.\n2. Apart from formulating an AVR task and constructing an AVR benchmark, the paper provides AVR training data, a pre-trained MLLM on it, and interesting experimental analysis.\n\n## weaknesses\n\nI do not see major issues with the paper, but I do have a few questions (see the Questions section).", "was_revised": true, "final_justification": "accept. a solid work, and rebuttals cleared my concerns.", "rebuttal": "### **Response to Reviewer dTPo**\n\n**Dear Reviewer dTPo,**\n\nThank you for your constructive feedback and for raising important questions about our experimental design and model comparisons. Your comments on potential biases in the multi-choice setup, the fairness of our model comparisons, and the scalability of our approach are highly valuable and have prompted us to conduct additional experiments and analyses to strengthen our work.\n\nWe have addressed each of your points below, including an analysis of the multi-choice setup, new experiments with a same-scale baseline, and an investigation into the effects of model scaling.\n\nThank you once again for your thoughtful review.\n\n---\n\n### **Q1: On Potential Bias in the Multi-Choice Setup**\n\n> **Reviewer's Comment:** The multi-choice setup mixes candidate answers and actions; I wonder if this would introduce a bias towards choosing an answer or an action. Can the authors provide an analysis on this?\n\n**Our Response:**\n\nThank you for raising this important concern about potential bias in our experimental setup. We analyzed the fairness of the multi-choice format from two perspectives:\n\n1.  **Balanced Option Design:** In the CLEVR-AVR benchmark, the candidate sets for most questions are designed to be balanced. A typical question includes 3-5 answer options and 3-6 action options. The number of options in each category is comparable, which avoids introducing a bias stemming from a significant quantitative difference between the two types of choices.\n\n2.  **Analysis of Model Behavior:** The prompt provided to the models includes a detailed description of the task, explicitly distinguishing between answer options and action options and explaining their respective meanings. This ensures that the model understands how to correctly select either an answer or an action. Furthermore, we have qualitatively inspected the models' generated reasoning traces and their final selections. We found no instances where the semantic intent of the reasoning process was inconsistent with the type of choice made (i.e., the model did not reason about taking an action and then select a final answer, or vice-versa). This indicates that the mixed-option setup does not negatively impact the model's decision-making process.\n\n---\n\n### **Q2: On the Fairness of Model Comparison**\n\n> **Reviewer's Comment:** The comparison between AVR-QWen2.5-VL-7B and PhysVLM-AVR-3B is unfair: 1) AVR-QWen2.5-VL-7B does not rely on the more complex multi-stage training that is applied to PhysVLM-AVR-3B; 2) the model sizes are not on the same scale... Can the authors at least keep the same VLM backbone?\n\n**Our Response:**\n\nThank you for this critical comment on the fairness of our model comparisons. We acknowledge your concerns and have conducted **a new experiment with a same-scale baseline** to provide a more direct comparison.\n\nFirst, we wish to clarify our original intention. The Qwen2.5-VL model is a pre-trained foundation model, so we directly fine-tuned it rather than applying our full pre-training pipeline. The comparison between the fine-tuned AVR-Qwen2.5-VL-7B and our PhysVLM-AVR-3B was intended to highlight the significant impact of our multi-stage, progressive training strategy.\n\nHowever, we fully agree that a comparison with a model of the same scale is essential. We have now trained and evaluated **AVR-Qwen2.5-VL-3B**, which has the same scale as our model and was fine-tuned using the same protocol as the 7B version. The results are as follows:\n\n| Model | Final Answer Acc. (`ACC_FA`) | Information Gain Rate (`IGR`) |\n| :--- | :---: | :---: |\n| **PhysVLM-AVR-3B (Ours)** | **39.7%** | **29.9%** |\n| AVR-Qwen2.5-VL-3B (New Baseline) | 29.6% | 28.7% |\n\nThese results demonstrate that PhysVLM-AVR-3B significantly outperforms the same-scale baseline. This suggests that its superior performance stems from our architectural modifications (e.g., max pooling for efficient multi-image processing) and our dedicated multi-stage training strategy, rather than being an artifact of model size.\n\n---\n\n### **Q3: On Scaling Laws and Comparison with GPT-4o**\n\n> **Reviewer's Comment:** GPT-4o, in general, outperforms all other models, including those trained on CLEVR-AVR, presumably because of its larger model size. Is it possible for the authors to train larger models and check if AVR-pretraining remains helpful as the model size increases?\n\n**Our Response:**\n\nThank you for this excellent suggestion regarding model scaling. To investigate this, we fine-tuned a larger model, **AVR-Qwen2.5-VL-32B**, to analyze the impact of model scale on active visual reasoning performance.\n\nDue to time and resource constraints, we were unable to train a larger version of our PhysVLM-AVR from scratch, as this requires a full pre-training pipeline unlike the direct fine-tuning of the Qwen2.5-VL models. However, we hope the analysis of the 32B model provides valuable insights.\n\n*   **Performance of Fine-tuned AVR-Qwen2.5-VL-32B:**\n    *   The `ACC_FA` improved to **41.2%** (compared to 29.6% for the 3B version and 45.7% for GPT-4o).\n    *   The `IGR` improved to **39.5%** (compared to 34.7% for the 3B version and 50.8% for GPT-4o).\n\nThese results confirm that increasing the model's scale leads to improved performance on active visual reasoning tasks. While our fine-tuned 32B model closes the gap, it does not yet surpass GPT-4o, suggesting that both scale and potentially architectural differences play a role. A more in-depth analysis would require training our PhysVLM-AVR at a larger scale, which we plan to explore in future work to investigate its potential to match or exceed proprietary models.\n\n---\n\n### **Q4: On the Applicability to Vision-Language-Action (VLA) Models**\n\n> **Reviewer's Comment:** The current experiments focus only on MLLMs, but in the context of embodied reasoning, models that can predict and execute actions (e.g., vision-language-action models) seem to be better candidate models to be tested, can the authors explain if it is possible to apply CLEVR-AVR to VLA evaluation and in which ways if so?\n\n**Our Response:**\n\nThank you for suggesting this exciting application of our benchmark. We agree that Vision-Language-Action (VLA) models are a natural fit for this task, and we designed CLEVR-AVR with this possibility in mind. CLEVR-AVR can be readily adapted for VLA model evaluation in the following ways:\n\n1.  **Direct Simulation Platform Support:** CLEVR-AVR is built on the Genesis simulation platform, which natively supports direct control of the robot arm via action sequences. This means the environment is already equipped to handle the low-level action outputs from VLA models.\n\n2.  **Action Space Mapping:** The discrete, high-level action space of CLEVR-AVR (e.g., \"move object X,\" \"rotate view by Y degrees\") can be straightforwardly mapped to the action vocabulary of prominent VLA models (e.g., the action tokens used by RT-1 or RT-2). The VLA model's task would be to generate the correct low-level action sequence corresponding to the high-level reasoning step.\n\n3.  **Expanded Evaluation Metrics:** In addition to our existing metrics (`ACC_ISJ`, `IGR`, `ACC_FA`), we can introduce VLA-specific metrics such as:\n    *   **Action Execution Success Rate:** Measures whether the generated action sequence successfully accomplishes the intended high-level goal (e.g., was the correct object moved to the correct location?).\n    *   **Reasoning-Action Consistency:** Evaluates whether the executed action aligns with the model's stated reasoning for information gathering.\n\nWe plan to extend our work to evaluate state-of-the-art VLA models like RT-2 and RoboCat on CLEVR-AVR. This will allow for a comprehensive assessment of their ability to perform targeted, reasoning-driven actions to acquire information in service of a broader goal."}, {"review_id": "zpHono5g3X", "reviewer": "Reviewer_wcc4", "summary": "This paper introduces the Active Visual Reasoning (AVR) task, which evaluates visual reasoning in partially observable and interactive environments. To support this task, the authors develop the CLEVER-AVR benchmark, which provides interactive simulation environments, along with a large-scale chain-of-thought dataset, AVR-152k. Additionally, the paper presents PhysVLM-AVR, a model trained via a multi-stage paradigm that achieves SOTA performance on the proposed AVR task.", "questions": "- How does the training time of PhysVLM-AVR compare to that of standard fine-tuning?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Active visual reasoning is an interesting and practical task with real-world use. Developing a benchmark contributes valuable resources to the research community.\n- The experimental results and corresponding analysis are comprehensive.\n- The paper provides detailed prompts and examples, offering clear illustrations and ensuring reproducibility.\n### Weaknesses\n- In Table 1, the comparison is between Qwen2.5-VL-7B (trained on AVR data) and the proposed PhysVLM-AVR-3B. It would be more informative to also include results for Qwen2.5-VL-3B trained on the same data, allowing for a fair comparison at the same model scale.\n- According to Table 1, API-based models significantly outperform open-source 7B models, which score below 10% or even 0% accuracy on many tasks. It would be helpful to evaluate larger open-source models to better understand their potential on AVR.", "was_revised": true, "final_justification": "This work contributes a benchmark beneficial to the research community. Also, the provided implementation details ensure reproducibility. Including evaluation on larger models during rebuttal further improves the quality.", "rebuttal": "### **Response to Reviewer wcc4**\n\n**Dear Reviewer wcc4,**\n\nThank you for your valuable feedback and your suggestions for improving our experimental evaluation. Your points regarding fair model comparisons at the same scale, evaluating larger open-source models, and clarifying training time are all well-taken.\n\nWe have conducted additional experiments to address your first two points and provide a detailed breakdown of the training time as requested. The results and analysis are included in our detailed responses below.\n\nThank you for helping us strengthen our paper.\n\n---\n\n### **Q1: On Fair Comparison at the Same Model Scale**\n\n> **Reviewer's Comment:** In Table 1, the comparison is between Qwen2.5-VL-7B (trained on AVR data) and the proposed PhysVLM-AVR-3B. It would be more informative to also include results for Qwen2.5-VL-3B trained on the same data, allowing for a fair comparison at the same model scale.\n\n**Our Response:**\n\nThank you for this excellent suggestion. To ensure a fair comparison, we have conducted a **new experiment by training Qwen2.5-VL-3B on our AVR data**, using the same fine-tuning protocol as the 7B version. This allows for a direct, same-scale comparison with our PhysVLM-AVR-3B.\n\nThe results are as follows:\n\n| Model | Final Answer Acc. (`ACC_FA`) | Information Gain Rate (`IGR`) |\n| :--- | :---: | :---: |\n| **PhysVLM-AVR-3B (Ours)** | **39.7%** | **29.9%** |\n| AVR-Qwen2.5-VL-3B (New Baseline) | 29.6% | 28.7% |\n\nThe performance gap between the two 3B models highlights two key advantages of our approach:\n\n1.  **Architectural Optimization:** The max pooling layer in PhysVLM-AVR is specifically designed to enhance the efficiency of integrating features from multiple images, a crucial capability for multi-step reasoning that is not a native focus of the standard Qwen2.5-VL architecture.\n2.  **Targeted Training Strategy:** Our multi-stage training curriculum places a strong emphasis on the `AVR-Embodied Reasoning` subset, which is rich in multi-image reasoning tasks. This targeted training better equips our model for the temporal reasoning demands of the AVR paradigm.\n\n---\n\n### **Q2: On Evaluating Larger Open-Source Models**\n\n> **Reviewer's Comment:** According to Table 1, API-based models significantly outperform open-source 7B models, which score below 10% or even 0% accuracy on many tasks. It would be helpful to evaluate larger open-source models to better understand their potential on AVR.\n\n**Our Response:**\n\nThank you for suggesting this. To better understand the potential of larger open-source models on our task, we have conducted **a new experiment with the Qwen2.5-VL-72B model**.\n\nThe results are presented below:\n\n| Model | `ACC_ISJ` | `IGR` | `ACC_FA` |\n| :--- | :---: | :---: | :---: |\n| Qwen2.5-VL-7B | 4.9% | 3.7% | 2.6% |\n| **Qwen2.5-VL-72B** | **36.0%** | **15.6%** | **16.6%** |\n\nThese results clearly show that scaling up the open-source model from 7B to 72B parameters leads to a substantial improvement in performance on the AVR task. This indicates that model scale is indeed a significant factor for acquiring active visual reasoning capabilities.\n\nWhile there is still a gap compared to API-based models like GPT-4o, this experiment suggests that combining larger open-source backbones with our specialized multi-stage AVR training strategy could be a promising direction for future work to further close this gap.\n\n---\n\n### **Q3: On the Training Time Comparison**\n\n> **Reviewer's Comment:** How does the training time of PhysVLM-AVR compare to that of standard fine-tuning?\n\n**Our Response:**\n\nThank you for your question about training efficiency. Here is a comparison of the training times:\n\n*   **PhysVLM-AVR-3B (Full Training):** The complete multi-stage training process for our model takes approximately **76 hours** on a server with 8x A800 GPUs. This includes all stages detailed in Section 4 and Appendix C of our paper, starting from feature alignment.\n\n*   **Standard Fine-tuning (AVR-Qwen2.5-VL-7B):** Fine-tuning the pre-trained Qwen2.5-VL-7B model *only* on our `AVR-Core` dataset takes approximately **2 hours** on the same hardware.\n\nIt is important to note that the Qwen2.5-VL model has already undergone an extensive, multi-stage pre-training process by its developers. Our PhysVLM-AVR training starts from a much earlier stage (aligning visual and language features), which accounts for the significant difference in training time.\n\nWhile our multi-stage approach is more time-intensive, it improves data efficiency by allowing the model to \"focus\" on learning specific capabilities at each stage, leading to superior performance across multiple benchmarks. We will explore methods to improve training efficiency in our future work."}]

Author rebuttal:

Meta-review: (a) Summary: The paper claims that passive visual understanding is insufficient for many real-world scenarios and that AVR is a fundamental necessity. To address this, the authors propose
- CLEVR-AVR benchmark: A synthetic environment built on Genesis, extending CLEVR for evaluating active information gathering, multi-step reasoning, and uncertainty estimation.
- AVR-152k dataset: A large-scale dataset with Chain-of-Thought (CoT) annotations to train MLLMs for AVR.
- PhysVLM-AVR model: A hierarchical learning model trained on AVR-152k via a multi-stage paradigm, designed for active visual reasoning.
The paper demonstrates that PhysVLM-AVR achieves strong performance on the proposed AVR task and other embodied reasoning, planning, and static visual reasoning tasks. The authors argue that AVR's necessity extends beyond simple occlusion and stacking, enhancing robustness in static scenes and simulating higher-order cognition.

(b) Strengths:
- The proposed AVR task is novel, addresses an important limitation of MLLMs, and represents a sound contribution. It is seen as a practical task with real-world utility.
- The paper provides a dedicated benchmark (CLEVR-AVR) and a dataset (AVR-152k), along with open-source model checkpoints (PhysVLM-AVR), which are valuable resources expected to greatly benefit the research community and accelerate future research.
- The experiments are extensive and sufficient, validating the usefulness of the dataset and showing strong performance. The paper also provides interesting experimental analysis.
- The paper writing and figures are quite clear, and detailed prompts and examples ensure reproducibility.

(c) Weaknesses
- Despite a relatively high ACC_ISJ (information-seeking judgment) score, the final answer accuracy (ACC_FA) remains inadequate, indicating the model still struggles to effectively integrate information over multiple steps and perform coherent multi-stage reasoning. This is attributed to perception errors and temporal integration failures.

(d) Reasons to accept: The contribution in defining and tackling the novel Active Visual Reasoning (AVR) task, which is seen as a critical step in bridging the gap between static visual understanding and embodied intelligence for MLLMs. The reviewers have unanimous positive assessment.

(e) Rebuttal: 
- Initial concerns were raised about the necessity of active reasoning beyond simple occlusion and stacking, and the diversity of the AVR-Core dataset (limited to tabletop scenarios). The synthetic nature of CLEVR-AVR also raised questions about its transferability to real-world scenarios. The authors expanded on the necessity of AVR, detailing additional application scenarios like limited field of view, dynamic environments, and functional reasoning. They also highlighted its role in enhancing robustness and simulating higher-order human cognition. This comprehensive response fully resolved the reviewer's concern.
- The initial comparison between PhysVLM-AVR-3B and AVR-Qwen2.5-VL-7B was considered unfair due to differing training strategies and model sizes. There was also a need to evaluate larger open-source models to understand their potential better, as API-based models generally outperformed open-source ones. The authors provided additional results.
- The initial submission lacked sufficient details on how Information Gain Rate (IGR) is calculated and the definition of an "information-gaining decision". The authors provided explanations.
Decision: Accept (poster)