Pursuing Minimal Sufficiency in Spatial Reasoning
Authors: Yejie Guo
Yunzhong Hou
Wufei Ma
Meng Tang
Ming-Hsuan Yang
Venue: ICLR 2026 Poster (2026)
Keywords: spatial reasoning
agent
VLM
Primary area: foundation or frontier models, including LLMs

Abstract: Spatial reasoning, the ability to ground language in 3D understanding, remains a persistent challenge for Vision-Language Models (VLMs). We identify two fundamental bottlenecks: \textit{inadequate} 3D understanding capabilities stemming from 2D-centric pre-training, and reasoning failures induced by \textit{redundant} 3D information.
To address these, we first construct a Minimal Sufficient Set (MSS) of information before answering a given question: a \textit{compact} selection of 3D perception results from \textit{expert models}. We introduce \textbf{MSSR} (Minimal Sufficient Spatial Reasoner), a dual-agent framework that implements this principle.  A \textit{Perception Agent} programmatically queries 3D scenes using a versatile perception toolbox to extract sufficient information, including a novel \textbf{SOG} (Situated Orientation Grounding) module that robustly extracts language-grounded directions. A \textit{Reasoning Agent} then iteratively refines this information to pursue minimality, pruning redundant details and requesting missing ones in a closed loop until the MSS is curated. 
Extensive experiments demonstrate that our method, by explicitly pursuing both sufficiency and minimality, significantly improves accuracy and achieves state-of-the-art performance across two challenging benchmarks. Furthermore, our framework produces interpretable reasoning paths, offering a promising source of high-quality training data for future models. Source code will be made publicly available.

Published as a conference paper at ICLR 2026
PURSUING MINIMAL SUFFICIENCY IN SPATIAL REASONING
Yejie Guol-i, Yunzhong Hou?* Wufei Ma², Meng Tang", Ming-Hsuan Yang*
Shanghai Jiao Tong University BBiijnng Institute of Technology
3 Johns Hopkins University *University of California Merced
ABSTRACT
Spatial reasoning, the ability to ground language in 3D understanding, remains a persistent challenge for Vision- LanguaL Models (VLMs). We identify two fundamental bottlenecks: inadequate 3D understanding capabilities stemming from 2DD-centric pre-training, and reasoning failures induced by redundant 3D information. To address these, we first construct Minimal Sufficient Set (MSS) of information before answering given question: a compact selection of 3D perception results from expert models. We introduce MSSR (Minimal Sufficient Spatial Reasoner), dgual-agent framework that implements this principle. A Per- ception Agent programmatically queries 3D scenes using a versatile perception toolbox extract sufficient information, including novel SOG (Situated Ori- entation Grounding) module that robustly extracts language-grounder directions. A Reasoning Agent then iteratively refines this information to pursue minimality, pruning redundant details and requesting missing ones in closed loop until the MSS is curated. Extensive experiments demonstrate that our method, by explicitly pursuing both sufficiency and minimality, significantly improves accuracy and achieves sfate-of-the-art performance across two challenging benchmarks. Further- more, our framework produces interpretable reasoning paths, offering promising source high-qualit training data for future models. Source code is available https:/ github...... com/ gyi155/msss
INTRODUCTION
Spatial reasoning-tth ability perceive and reason about object relationships in 3D space-is cornerstone of general intelligence and a critical prerequisite for deploying Al in the physical world, from robotics AR/VR (Cheng al., 2024a; Kim et al., 2024). While modern VLMs (OpenAI, 2024; DeepMind, 2025) have achieved remarkable success, they still consistently fail on spatial reasoning tasks (Yang et al., 2025c; Li et al., 2025a). In this work, we diagnose this critical gap by identifying two fundamental bottlenecks:
Inadequate 3D perception. Trained predominantly on 2D data, VLMs lack geometric priors and thus struggle to perceive 3D information like layout, orientation, and depth (Ma et al., 2023; 2024b).
Redundancy degrades reasoning. 3D environments are information-dent Naively aggregating all percepts floods the context with weakly relevant details, potentially diluting attention (I.iu et al., 2023) and encouraging shortcut heuristics (Xiao et al., 2024) ultimately degrading performance (Fig. l).
In the face of these two challenges, cognitive science offers a compelling insight: humans navigate complex scenes not by exhaustively processing all sensory data, but by constructing task-specific, minimal mental models (Tversky, 1991). Based on the mental models, they then selectively attend to the details needed make decision, and incrementally update the mental models required (Byrne& Johnson-Lain Laird, 1989; Johnson-Laird, 2010). This principle is formalized in statistics by
"Equal contribution. Work done while visiting UC Merced.

---

Published as a conference paper at ICLR 2026
Heuristic Shortcuts
Attention Dilution
Q. the chair facing the window?
Q. Is the chair between the desk and the window? Non-minimal Set:
Non-ninimal Set:
yaw(chair)=+49"
position (chair, desk, window) chair near to the door
window position
office desk is in front chair
desk the right
notebook and mug on desk
window on the left Mall
Reasoning:
Reasoning:
Desk is front of chair, as chair typically faces .os....No.
Desk is right, chair near door.
There is no direct Line from Uhi....No.
Figure I: Irrelevant (highlighted in red) information can overwhelm the VLM and hurt its performance. Left: VLM ignores crucial information and hallucinates. Right: VLM cannot attend the important part.
the Minimal Sufficient Statistic (Lehmann & Casella, 1998), which captures all relevant information from a sample in the most compressed form possible.
We show that the key to robust spatial reasoning lies in a similar pursuit: actively discovering Minimal Sufficient Set (MSS) the most compact representation of spatial information required to answer specific query. Motivated by this principle, we introduce MSSR (Minimal Sufficient Spatial Reasoner), a zero-shot framework that operationalizes this pursuit through a dual-agent architecture. MSSR disentangles the challenges of perception and reasoning into two specialized, collaborative agents:
To bridge the 3D perception gap, we equip Perception Agent (PA) with suite of vision modules to programmatically query the scene for spatial primitives (e.g., locations, directions, relations) and return structured, VLM-friendly state. A key limitation of existing tool-augmented VLMs is their inability to ground complex, situational directions specified by language descriptions. We address this with SOG (Situated Orientation Grounding) module, which reformulates the orientation estimation task as a multi-choice question, and to overlay candidate 3D directions on 2D images as visual prompting. Through a procedural coarse-to-fine approach and alternative 3D view rendering to eliminate ambiguities, it robustly extracts object orientations ("Is the chair facing the door?") and behavior-centric directions ("Which way is the person facing while ascending the stairs?").. Thus, the PA provides rich and accurate 3D data without costly end-to-end training.
While the PA gathers extensive perceptual data, it risks worsening the second bottleneck: information redundancy that could degrade reasoning accuracy. To solve this, a Reasoning Agent (RA) strate- gically prunes this stream of information and curates the MSS by explicitly filtering redundant or task- inss-irrelevant 3D information. It first formulates high-level reasoning plan, assesses the collected information, and subtracts non-con contribbiib information. If this set is deemed insufficient to confi- dently answer the question, the RA issues targeted, specific requests back to the PA to acquire only the missing information. This iterative refinement continues until a MSS is formed. The final answer is derived exclusively from this curated set, sharpening the model's focus and mitigating errors from distracting data.
We evaluate MSSR situated multi-view reasoning in complex scenes, and ViewSpatial-Bench (Li et al., 2025a), which focuses on perspective relational understanding. MSSR achieves state-of-the-arr performance against strong monolithic and agentic basclines, while producing compact, interpretable reasoning paths. Extensive ablation studies demonstrate the harmful effect of redundancy and the efficacy of the RA'S pruning in restoring performance. Our contributions are threefold:
challenging benchmarks: MMSI-Bench (Yang et al., 2025c), which tests
We formulate 3D spatial reasoning as Minimal Sufficient Set construction and introduce dual-agent framework that interleaves perception with high-level planning to acquire just enough information.
We design perception agent that has access to a versatile toolbox including a SOG module for robust directional grounding; and reasoning agent that directs the entire process and provides final answers.
Our MSSR effectively improves the 3D spatial reasoning performance on two challenging bench- marks and produces interpretable reasoning traces that can provide supervision for future 3D-aware models.
2

---

Published as a conference paper at ICLR 2026
Query: direction of entering through the door considered the north, what is the correct order the north:
fallowing objects TV, mural, sofa, door
Complete
Minimal Sufficient
Perception Agent
Reasoning Agent
Sn
Situated Orientation Grounding
<Frune>
updated Sais (..),
<Request:: Find projections on the north vector.
Locate
Sx+1 <Request>
<Decide> Rensoring upon the final information set, door, TV, sofa, mural
Calibrate
Going Upstairs Sitting on Safa
Figure 2: The MSSR dual-agent framework. Top:
To avoid overwhelming the VLM, MSSR extracts Minimal Sufficient Set that only contains the necessary information. Bottom: Dual-agent loop: the Perception Agent provides a spatial information set (S,), and the Reasoning Agent prunes irrelevant information, requests missing information, makes final decision based the curated set.
2 RELATED WORK
Monolithic VLMs in Spatial Reasoning typically inject 3D knowledge by fine-tuning on synthetic data (Chen al., 2024a; Cheng al., 2024b; Ma al., 2025a) integrate specialized modules to process 3D modalities like point clouds (Hong et al., 2023; Huang et al., 2024; Ma et al., 2025b). While demonstrating progress, these methods are fundamentally limited: they require prohibitively expensive 3D instruction datasets (Yang et al., 2025b) and risk forgetting pre-trained knowledge, which degrades the VLM's crucial general-purpose reasoning abilities (Kirkpatrick al., 2017). In contrast, MSSR is a zero-shot, training-frre -free framework that bypasses these issues. By preserving the VLM's full capabilities and instead explicitly structuring the perception-reasomee process, our approach maximizes spatial reasoning performance without compromising the model's versatility requiring costly data and retraining.
Agentic Frameworks are prominent line of work where complex problems decomposed into steps solved via tool-use. Pioneering methods like ReAct ( (Yao et al., 2023) showed that LLMs can interleave reasoning and action, paradígm extended to 3D where agents gather spatial information for tasks like embodied exploration (Yang et al., 2025d) and 3D VQA (Ma et al., 2024a). These approaches focus primarily on information gathering, often adopting a purely accumulative strategy. However, for the tasks we address, the dense nature of 3D scenes introduces significant redundancy, where an excess of irrelevant spatial details degrades performance. MSSR is therefore designed to not only gather information, but also prune irrelevance, which is a key departure from prior agentic designs.
Visual Programming is a paradigm often used for 3D attribute querying (Marsili et al., 2025; Yuan al., 2024). enhances VI.Ms by decomposing complex visual tasks into executable programs that leverage specialized modules (Gupta & Kembhavi, 2023; Surís et al., 2023). We adopt the visual programming as the execution backbone for our Perception Agent, leveraging its modularity to integrate specialized tools. Instead of the typical one-shot execution, paradigm by integrating visual programming into a closed loop. By preserving the full execution state across iterations, subsequent perception steps build upon prior computations, enabling dynamic information refinement while avoiding redundant work.
framework advances this
3

---

Published as a conference paper at ICLR 2026
When I open the door and come outside, where is the clock located relative to me? Directly in front, left, C: Pront right,
Save
Load
<Roquest>: Establish coordinate and calculate angles.
Seel
<Deissio??
According to the calculated angles,
Front left
W Perception Agent
Sx
Reasoning
Sufficient?
the direction coming
Agent
north, from north and 52" from the west.
5 Loc(daon)
Sartf( (ocor)
(ecck)))))
Loc(clock)
That moans the clock is frunt left of you.
(rine (conee__n.. iin)
Orient(cone, in)}
Camena_ Canena_Posess Loc (Dutsiddd
Longth:3
......) (ength:18
Figure 3: A detailed example. The Perception Agent provides an 18-item set Sg, and the Reasoning Agent prunes essential items in S2+1. Deeming this insufficient, the RA issues a <Request> for missing calculations. With sufficient information, it makes a final <Decisime sio?>.
METHOD
We address language-conditione spatial reasoning: given M views I = (/1,-.., MM} from the same scene and natural-language query q, the goal is produce the answer a. Before answering the 3D reasoning question, our approach first builds Minimal Sufficient Set (MSS)-a compact representation that both sufficient answer the query and minimal prevent failures from redundant and distracting information.
3.1 OVERVIEW
As illustrated in Figure 2, our method actively curates the MSS. The process iterative, interleaving information acquisition and strategic pruning converge representation that is both sufficient and minimal. Formally, we define the target of this process, the MSS, as follows: Let W represent of all spatial and semantic information derivable from the full 3D scene. We consider a set of spatial information S C W. which is subset of W. Our goal is to find the MSS
the comprehensive
S* C W, which satisfies two properties:
1. Sufficiency: Ideal MSS S* must contain enough information for an oracle reasoning agent, R", to correctly answer the query q. Formally, this means:
R* (S* .q) a²
where at is the ground-truth answer. This ensures that no essential information for reasoning is omitted.
2. Minimality: MSS S* should be free of redundant or irrelevant information that would burden the reasoning model. Ideally, is the smallest such set that maintains sufficiency. This can be expressed
AS CS' R'(S', F
Through updating the spatial information set S, we aim to approximate the MSS S*, In this attempt, MSSR conducts closed-loop collaboration between the Perception Agent (PA) and Reasoning Agent (RA), as shown in Figure 3. The process begins with an empty S and proceeds iteratively. Initially, the PA executes a broad perception directive to populate S with a comprehensive, potentially non-minimal set of spatial primitives. This set is then passed to the RA for curation, where formulates a reasoning plan and prunes any information not causally linked to the plan, thereby enforcing minimality. If the pruned set is deemed insufficient, the RA performs an assessment and issues a targeted request back to the PA for precisely the missing information, which then augments S. This cycle of curation and targeted augmentation repeats until the RA judges S to be sufficient for answering the query. At this final stage, the RA discards all prior context and reasons exclusively over the curated MSS produce the answer, ensuring both focus and interpretability.
4

---

Published as a conference paper at ICLR 2026
3.2 PERCEPTION AGENT
The Perception Agent (PA) serves as the perception engine in our framework, responsible for bridging the gap between high-level reasoning directives and raw elements from the 3D scene. To equip the PA with robust and precise 3D perception capabilities, we adopt the Visual Programming (Gupta & Kembhavi, 2023) paradigm. The PA is provided with suite of pre-designed modules, which act as specialized tools. These modules leverage vision expert models for tasks like geometric reconstruction and object localization, significantly augmenting the LLM's capabilities for complex spatial computations. Furthermore, the structured nature of code generation within the visual programming framework ensures that the information extraction process is logical, transparent, and reproducible.
At each turn, the PA receives the current S, the original query, scene images I and a natural language request r from the Reasoning Agent representing the current information information-gathering goal. Its objective to generate a Python script that invokes the appropriate foundation modules fulfill the directive. The process begins with an empty S and an initial, broad instruction, such as:
"Extract all potentially relevant information to solve the problem. Your goal is to find as much information as possible."
The generated script populates a designated dictionary with newly extracted information such as object coordinates and spatial relationships. This dictionary is then merged into S. Crucially, after each execution, the entire state the Python environment, including all intermediate variables and data structures, is preserved as a snapshot. When the PA is invoked in subsequent turn, this snapshot reloaded. This mechanism allows the PA to contextually build upon its previous computations, avoiding redundant processing and enabling more complex, stateful exploration of the scene. The newly updated S is then passed to the Reasoning Agent.
Spatial Reasoning Modules. The Perception Agent implements directives by invoking curated suite of spatial modules, which bridge high-level goals and raw perceptual data to construct the MSS. Following established designs (Surís al., 2023), our toolkit includes basic modules: locate module that utilizes vision expert models (Liu et al., 2024; Ravi et al., 2024) to pinpoint object coordinates in 3D, and a computation module that offloads complex numerical tasks, such as coordinate frame transformations. More critically, to address challenges often overlooked by prior work. we propose novel modules designed for more robust spatial reasoning. These advanced modules cover the full pipeline of spatial understanding, from establishing a coherent 3D representation to grounding gegguag--conditonne attributes. The specific implementation of each module is detailed in Appendix C.
Foundational 3D Scene Reconstruction. To bridge the critical gap between sparse 2D images and a coherent 3D scene representation, this module leverages recent breakthroughs in rapid based-based reconstruction models (Wang et al., 2024b;a; 2025) to estimate the camera parameters, depth maps, and unified 3D point cloud of the scene. In our experiment, VGGT (Wang al., 2025) shows robust performance and high speed. This output serves as the foundational canvas upon which subsequent spatial information is extracted. Similar to Chen et al. (2024a), additionally segment the ground plane during reconstruction.
Global Coordinate System Calibration. To resolve the inherent ambiguity of view-dependent spatial terms (e.g.. "left," "behind"), this module establishes a unified global coordinate system. It aligns the scene's 's axes based reference vector, which is derived either from explicit instructions in the query ("assume the window faces east") or from prominent landmarks. This calibration ensures all directional and relational information stored in the MSS is consistent and unambiguous, a prerequisite for reliable multi-step reasoning.
Situated Orientation Grounding (SOG). Beyond simple localization, reasoning often requires understanding orientation. We introduce the SOG module, which utilizes Visual Prompting (Qi et al., 2025) to ground complex, language-conditing conditioned directional concepts into 3D vectors. Critically, SOG handles not only intrinsic object orientation ("the front of the chair")) but also orientations ("the direction to exit the room") capability largely overlooked in prior work. This module dramatically expands the range of addressable queries from static localization to dynamic, perspectival reasoning.
5

---

Published as a conference paper at ICLR 2026
Target: Direction of facing away from the wall in the clock area.
Renderer
Input: <ing_ coarse><rrompt>> Output: <Reasoning> Vector3.
Input: <img finexxprompt>
Output: <Reasoning> Vectorl
<Render around vector3>
Return: (x,
Carera
Est.Depth
Target
Figure 4: Situated Orientation Grounding as a multi-choice question. We prompt a VLM to select a direction from multiple candidates using the original image and rendered image, both overlaid with the candidate's 3D orientation vectors. Starting from coarse directions, then refines this direction to fine-grained result.
The power of SOG lies in bridging the gap between VLM's rich semantic scene understanding and its inability to directly regress 3D geometric outputs. Instead of attempting this intractable regression, more reliable visual selection task, implemented through
reframe orientation grounding coarse-to-fine strategy as illustrated in Figure
For query anchored at object position Po. first randomly generate sparse set of four coplanar, orthogonal vectors {d ) parallel to the ground plane, resembling compass directions. To provide the VLM with sufficient context and resolve perspective ambiguity, we render these vectors onto two distinct views: Situated View using the original image to preserve natural context, and synthetic Canonical View from an elevated perspective to reduce foreshortening. The VLM ís prompted to select the candidate that best aligns with the language query. This selection is subsequently refined by generating denser set of candidates around the chosen vector and repeating the selection process, allowing the system to converge on a precise direction. Empirically, this coarse-to- coarse-to-fine strategy robustly and efficiently identifies the target 3D orientation. While not designed for sub-degree accuracy, our experiments confirm that this level of precision is sufficient for most challenging spatial tasks. The vector is then added to S, empowering the RA tackle wide range of orientation- dependent problems that were previously intractable.
3.3 REASONING AGENT
The Reasoning Agent (RA) acts as the cognitive core in MSSR, responsible for ensuring the informa- tion set S is both sufficient and minimal. At each step, the RA receives the current information set S,, and the original query q. It operates in a two-stage process information curation and strategic decision-making.
In the first stage, Plan-Guided Information Curation, the RA formulates a high-level reasoning plan outlining the necessary steps answer the query. With this implicit plan, it initializes empty, updated information set, S2+1- The RA is then prompted to systematically scrutinize each item within the current set Sn, critically evaluating its relevance to the reasoning plan. Only those deemed necessary for the plan are preserved and added to Sn+1. This subtractive filtering step is crucial for maintaining the conciseness of S, aggressively pruning any information that is irrelevant to the specific query. After completing the curation, the RA enters the second stage, Strategic Decision-Muking. Here, it makes one of two decisions:
<Request>: If the RA determines that S.+1 is insufficient to complete the reasoning plan, it then formulates a targeted, natural-language directive that precisely articulates the missing infor- mation (e.g., *<Request> The facing di rectic on of someone
on the chair. ") This request, along with the pruned Sn+1. is passed back to the Perception Agent. The PA uses this focused guidance and updated set to initiate new round of programming, generating Su+2 that populates the requested information.
<Decide>: Conversely, if the RA concludes that S,+1 contains all necessary information derive a final answer, it triggers the <Deci de> action. It then discards all prior context and reasons exclusively over this final, minimal set using Chain-of-Thhogh (CoTT) (Wei et al., 2022) to produce

---

Published as a conference paper at ICLR 2026
Table I: Comparison with baselines on MMSI-Bench and ViewSpatial-Bench. Our method performs favorably against previous methods on these challenging spatial reasoning tasks. Blue is to highlight the improvement of our method over GPT-4o backbone.
MMSI-Bencch
ViewSpatial- Beenhhttll-encchh
positional relationship
multi-stop reasoning
attribute & motion Proprictary LLM
overall
camera based
person based
overall
36.4
41,0
51.1
Gemini 2.5 Pro
38.5
34.3
36,1
37.0
44.3
41.6
43.0
Gemini 2.5 Flash GPT-4a (backbone)
37.4
33.9
35.0
41.6
36.9
38.4
28.0
30.8
34.3
30.3
33.6
36.3
35.0
Open-source LLM
Llama-3
28.8
LLaVA LLaVVVOne Visiision 7B
28.0
11.6
27.1
24.5
28.5
26.6
27.5
Qwen2.5-VI.3
26.5
Owen2.5-V1.77
25.9
25.8
26.1
25.9
40.6
33.4
36.9
Qwen2.5-V1.-22R
31.2
27.3
32.1
M0.7
47.6
38.9
43.1
Owen3-V1L-4B
32.8
27.8
30.3
31.1
42.3
35.9
39.0
Qwen3-VL-8B
32.6
30.3
28.9
31.1
45.1
37.5
41.2
VB
28.7
IntemV1.3.14P
25.5
29.3
27.5
26.8
47.1
33,9
40.3
3D-VLM
Vidco=3D-LLM
25.5
25.8
25,3
24.4
35.8
30.3
LLaVA-3D
29.5
19.7
31.0
28.0
39.6
28.3
33.8
Specialist
LEO
42.3
32.3
38.6
39,3
41.5
45.8
43.7
Agentic
ViLaSR
30,2
42.4
34.4
38.2
VADAR
32.8
26.1
28.9
34.2
33.2
33.7
MSSR (Owrx)
50.6
50.0
47.1
49.5 (9.222
51.0
54.4
51.8 (6.68))
the answer. This disciplined use of only the pruned ensures that the final reasoning is efficient and shielded from the distracting influence of irrelevant data.
Notably, unlike many current 3D agentic systems (Li al., 2025b; Marsili et al., 2025), both our agents operate in zero-shot fashion. They are guided by high-level principles rather than in-context learning (ICL) examples. This design endows our method with strong generalization capabilities and mitigates the risk of overfitting to dataset-specific exemplars.
4
EXPERIMENTS
4.1 BENCHMARKS AND BASELINES
We evaluate our method and various baseline models on two challenging spatial reasoning benchmarks. MMSI-Bench (Yang et al., 2025c) a hand-crafted multi-image dataset focusing on spatial reasoning. The tasks are designed probe model's understanding of the positions, attributes, and motions of three elements cameras, objects, and regions-within within real-world environments. Furthermore, it incorporates multi-step reasoning split, which composes elementary tasks into long-horizon questions to test for deeper spatial reasoning. ViewSpatial-Bemn this by addressing critical limitation of the gap between egocentric and allocentric spatial reasoning. Designed to evaluate multi viewpot spatial localization and recognition capability, this benchmark is structured into five distinct task categories to rigorously assess model's ability generalize across different spatial viewpoints. These two benchmarks provide a robust, multifaceted platform for testing spatial reasoning capabilities that we aim advance.
(Li et al., 2025a) complements
To provide a comprehensive evaluation, we establish four distinct categories of baseline models for comparison: Proprietary LLMs: We include powerful, closed- source models with strong reasoning capabilities, such as GPT-4o (OpenAI, 2024), Gemini-2.55ffaat (DeepMind, 2025), Gemini-2.5- pro (DeepMind, 2025), and (OpenAI, 2025). Open-Source LLMs: We compare with state- of-the-art models such as LLaMA-3.2. 2isiio (Meta, 2024), LLaVA-One Qwen2.5-Vi VL (Bai et al., 2025), Qwen3- (Yang al., 2025a), IntemVL3 (Zhu et al., 2025), Intern VL2.5 (Chen et al., 2024b) and DeepSeek-Vil VI.2 (Wu et al., 2024). Specialists: We compare with LEO (Huang al., 2024), a multi-modal generalist model recognized for its proficiency in various 3D tasks. 3D-VLMs: We evaluate recent 3D-VLMs including VLM-3R (Fan et al., 2025),
(Li et al., 2024).
7

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study of our framework's 's key components. We report accuracy (%) MSR sub-task, MMSI-Bench, ViewSpatiil-Benchh
Accurssy
48.1%
MSR (MMSI-Bench)
MMSI
ViewSpatial
54-
Method
Bench
Bench
45.8%
6.5
5.9
Ours (Full)
GPT-4o
50.0
30.8
49.5
30.3
51.8
35.0
Only PA
33.8
37.1
32.5
Only RA
31.8
31.1
35.3
w/o SOG Iteration
47.0
46.9
43.2
Figure 5: Effects conciseness on accuracy.
44.4
47.2
48.8
Hideo-3D-LLM, (Zheng et al., 2025), and LLaVA-3D (Zhu et al., 2024), which trained and infer with augmented 3D feature representations, making them directly relevant to spatial tasks. Agents: We consider VADAR (Marsili al., 2025), visual programming framework targeted spatial inference, and ViL (Wu et al., 2025), a model notable for its ability of visual manipulation.
4.2
MAIN RESULTS
the MMSI-Bench, our method achieves an overall accuracy of 49.5%. We outperform even the strongest LLM evaluated, o3 (41.0%), demonstrating an absolute gain of 8.5 percentage points.
Compared to the best-performing open-source LLM, Qwen3-VL-8P (31. (311%)), our approach shows pronounced relative improvement of over 60%. When contrasted with state-of-the-art 3D-VLM models such as VLM-3R (32.0%), generalist model such as LEO (39.3%), and agentic frameworks such ViLaSR (30.2%), our method maintains clear lead.
Transitioning to the ViewSpatial-Bent Bench, which focuses on multi-viewpoint spatial localization benchmark with an overall accuracy of 51.8% The consistent strength observed in both Camera Based (51.0%) and Person Based (54.4%) tasks highlights our method's robust generalization across varied spatial viewpoints. Excelling in both categories indicates MSSR's 's proficiency in bridging the gap between egocentric and allocentric spatial understanding, limitation for current methods.
and recognition, our method again
4.3 ABLATION STUDY
Effects of Minimality, A central idea of our framework is that pursuing minimality is crucial for robust reasoning. To rigorously test this, we conducted a controlled ablation study on representative subset of MMSI-Bench problems MSSR solved in three iterations. For these problems, aufficieney-normaaized information sets for each step by retrospectively adding critical information (discovered in iterations) to earlier, larger sets. This isolates the effect of set size on the RA's performance, ensuring each set contains the same level of sufficiency. The RA was then tasked to solve the problem independently using these three sets, corresponding to the state after each iteration.
created
As illustrated in Figure 5, the results reveal a clear inverse correlation between information set size and accuracy. As our iterative pruning strategy reduces the average set element count from initial 17.3 to a concise 5.9, the RA's inference accuracy concurrentyy rises from 45.8% to 48.3%. This finding provides empirical evidence that excess information is significant distractor for LLM-based agents. This affirms that pursuing minimality is fundamental for high-fidelity spatial reasoning, not merely an efficiency optimization.
Component Analysis. We conduct ablations to quantify the contributions of each component. In Table 2, we report accuracy on the representative Multi-step Reasoning sub-task and the overall score on MMSI-Bench and ViewSpatial-Bena
Only PA: Removing the RA and tasking the PA to programmatically deduce the answer after information extraction leads to significant performance drop. In fact, its sequential, top-down
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance across different LLM backbones.
MSR (MMSI-Rench)
MMSI Bench
ViewSpatial Bench
RA Backhone
PA Backbone
Uniform
LLaVA-One Vision---
LLaVA-OneWisionn77
33.6
35.0
34,1
Qwen2.5-VL-78
Qwen2.5-VI.-7B
38.6
39.8
40.2
Qwen3-VL-8B
Qwen3-VL-8B
44.2
43.1
46.5
GPT-4o
GPT-4o
50.0
49.5
51.8
Cross-Model
GPT-4o
Qwen2.5-VL-7B
41.3
40.1
43.0
Qwen2.5-VL-7B
GPT-4o
44.4
44.2
46.6
execution flow, while effective for information gathering, is less effective in reasoning and question answering, as often returns sub-optimal results.
Conversely, removing the PA and forcing the RA to rely solely on the initial context yields negligible improvement over the bascline. This result affirms that prompting alone cannot substitute for the precise, targeted 3D scene perception provided by the PA, even when using powerful reasoning model. This highlights the crucial synergy between the two agents.
Only RA:
We replaced SOG with baseline directly prompting the VLM to infer directional vectors, causing an overall performance drop to 46.9%. Our observations suggest this degradation stems from the VLM's inability to regress 3D coordinates, rather than failure in semantic understanding. This result validates our design of SOG, which circumvents this limitation by reframing regression as visual selection task.
w/o SOG:
Setting the maximum iteration count to forces the RA to make an immediate decision without the ability to request additional information. This leads noticeable drop in performance. The degradation is more pronounced on the MSR sub-task, underscoring that for intricate multi-step tasks, the iterative feedback loop especially vital for achieving information sufficiency.
w/o Iteration:
4.4 GENERALIZABIILTY ACROSS LLM BACKBONES
To validate the generalizability and scalability of MSSR beyond proprietary models, we evaluated our framework using wfate-of-hhe-at open-source backbones, including 2024), Qwen2.5-VL-7B (Bai et al., 2025), and Qwen3-VL-SB (Yang et al., 2025a). We explored both uniform configurations (same backbone for PA and RA) and cross-model configurations to analyze agent-specific sensitivity in table 3.
(Li et al.,
Consistent Framework Gains, MSSR consistently boosts performance regardless of the backbone. For instance, while the Qwen2.5-VL-7B baseline achieves 25.9% on MMSI-Bench (see Table 1), integrating it into MSSR improves accuracy to 39.8% (+13.9 % The performance scales predictably Qwen3-VL-S8 GPT-4o),
with model capability (LLaVA-One 7B < Qwen2.5-VL- demonstrating that MSSR's performance scales consistently with the underlying backbone's strength. This validates the framework's scalability and suggests that future improvements in language models can directly translate to stronger spatial reasoning performance within MSSR.
Agent Sensitivity. Cross-model ablations indicate that the Perception Agent is more sensitive to backbone capability than the Reasoning Agent. Downgrading the PA (GPT-4o - Qwen2.5) causes larger drop (-9.4%) than downgrading the RA (-5.3%). This suggests that the PA's requirement for precise code generation and API utilization demands stronger model capabilities, whereas the RA's natural language planning is relatively more robust.
Deployment. This asymmetry enables strategic trade-off: using strong model for the PA and a lighter model for the RA (e.g., GPT-40 + Qwen2.5) retains 90% of the performance (44.2% vs 49.5%) while significantly reducing inference costs.

---

Published as a conference paper at ICLR 2026
Table 4: Accuracy on MMSI-Bench after fine-tuning with data annotated by our framework.
Model
Accuracy (%)
Owen2.5-VI-77 Qwen2.5-VL-7B-SST (annotated by MSSR)
25.9
30.1 (+4.2)
4.5 APPLICATION: ANNOTATING SPATIAL REASONING DATA
Beyond zero-shot inference, MSSR serves as a powerful data annotation engine. In addition to numerical results (Hu et al., 2024), its final output-a Minimal Sufficient Set (MSS) and an explicit reasoning trace-provides rich foundation for creating CoT (Wei et al., 2022)- data.
To demonstrate this potential, we curated a dataset by sampling 300 (30%) correctly solved questions from our MMSI-Bench runs. We then employed GPT-4o for automated quality filtering, which scrutinized the RA's reasoning for logical consistency and removed cases of incidental correctness, yielding 258 high-fidelity traces. GPT-4o then synthesized these traces into CoT-style annotations. This process grounds the RA's abstract reasoning in gathered spatial information by systematically interleaving high-level strategic steps (e-g.. "I need to determine the chair's position"") with the specific perceptual evidence from the MSS that substantiates each step (e.g.. "The Locate module confirms the chair at [x.y.z]"). We conducted a preliminary fine-tuning experiment using this synthesized dataset Qwen2.5-Ve V-- As shown in Table 4, despite the modest scale of our experiment, the fine-tuned model's Is accuracy on MMSI- of 4.2%. This result is particularly noteworthy it elevates the 7B model to performance level competitive with its much larger 72B counterpart. This validates MSSR as an effective data engine for distilling complex spatial reasoning capabilities into future models (details in Appendix F).
rose to 30.1%% 16, an absolute improvement
CONCLUSION
We present MSSR, which builds a Minimal Sufficient Set before answering spatial reasoning ques- tions. Specifically, it adopts a dual-agent loop: a programmatic Perception aggme-ad with SOG for directional grounding extracts spatial information, while plan-guided Reasoning Agent ensures sufficiency and minimality through targeted pruning and requesting. This design mitigates redundancy-indueee errors. Our framework boosts the performance of the backbone VLMM significantly, achieving state-of-the-arr results on AMSS--Bench and ViewSpatial-Bar -ench. Beyond inference, the MSS reasoning traces also serve as high-quality supervision signals for training future 3D-aware models.
ceuuninn-extrratts
Acknowledgment
Research was sponsored by the Office of Naval Research and was accomplished under Grant Number W911NF-24-1-0295.
REFERENCES
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In JEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, 2022. 23
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint 250i 13923, 2025. 7, 7,9,25
Ruth MJ Byrne and Philip N Johnson-Laird. Spatial reasoning. Journal of Memory and Language, 2865):544-575, 1989.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-languaee models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 14455, 2024a. 3,
10

---

Published as a conference paper at ICLR 2026
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b. 7
An-Chich Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem BIyIk, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision- aision-laguage-acttn model for navigation. arXiv preprint ar wrxi:24412.044555 2024a.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in -aanunla models. Advances in 2024b.
Neural Information Processing Systems, 37: 37:135062-13500
Google DeepMind. Gemini 2.5: Our most intelligent ai model. https://bloo..ooqq/ technol Logy/ google- 2025/, 2025. 7,7
ndd hed/geminni carmmin-modee--ee
Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Tianlong Chen, Jiachen Li, Zhengzhong Tu, Zhangyang Wang, and Rakesh Ranjan. Vlm-3r: Vision-language models augmented with instrection-aligned 3d reconstruction, 2025. URLhtttps://axxi
20279.
flask. flask. http:://feask.ppllleggpppeesss.s httpl//ssspprooccttcccocc/ee//aa
2025. 25
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning with- out training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 14953-14965 2023. 3,5
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 18:20482-204994 2023. 3
Gan. 3d-llm:
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference
Learning Representations, 1(2):3, 2022. 22
Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition Conference, pp. 9590-9601, 2024. 10
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In International Conference on Machine Learning, 2024. 3,
Philip N Johnson-Laird. Mental models and human reasoning, Proceedings of the National Academy of Sciences, 107(43):18233-1855) 2010.
Yeseung Kim, Dohyun Kim, Jieun Choi, Jisang Park, Nayoung Oh, and Dachyung Park. A survey integration of large language models with intelligent robots. Intelligent Service Robotics, 17(5): 1091-1107, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinskwinn et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114
(13):3521-3526, 2017. 3
E. L.. Lehmann and George Casella. Theory of Point Estimation. Springer, 2nd edition, 1998. 2
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024. 7,9
11

---

Published as a conference paper at ICLR 2026
Dingming Li, Hongxing Li, Zixuan Wang. Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang. Wenqi Zhang, Yongliang Shen, et al. Viewspatial- Evaluating multi- perspective spatial localization in vision-language models. arXiv preprint arXiv:2505.21500 2025a. 2,
Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. Seeground: See and ground for zero-shot veee-vocabulary 3d visual grounding. In IEEE/CVE Conference on Computer Vision and Pattern Recognition Conference, 2025b. 7
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, arXiv preprint arXiv:2307.03172 2023. 1
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference Computer Vision, pp. 38-55, 2024. 5, 18
Chenyang Ma, Kai Lu. Ta- Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. Advances in Neural Information Processing Systems, 37:68803 68832, 2024a. 3
Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Celso M de Melo, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. arXiv preprint arXiv:2412.0782. 2024b.
Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, and Alan Yuille. Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning. arXiv preprint arXiv:2504.2000 2025a.
Wufei Ma, Luoxin Ye, Celso M de Melo, Alan Yuille, and Jieneng Chen. Spatiallim: A compound 3d- informed design towards spatially-intlligennt large multimodal models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 17249-17260, 2025b.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In International Conference Learning Representutions, 2023. URL https: /wwp//ovewwww..eec//e/e neet forum?id-IDDx97BBC 1, 23
Damiano Marsili, Rohun Agrawal, Yisong Yue, and Georgia Gkioxari. Visual agentic ai for spa- tial reasoning with a dynamic api. In /EEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 19446-19455 2025. 3, 3,7,8, 16
Meta. Llama 3.2:
Revolutionizing edge ai and vision with open, customizable mod- els. https: //aii wwwa. wwoom//llg/llammm 3- 2--connect-ee 2024 vision-edge robile-devices// 2024.
OpenAl. http:://ppena..ccm//n/ee///lllop
2024. 1,7
http://wwwaaaaaaaaaaao.o///o/oar and-o4-min:/
OpenAl. Introducing o3 and o4 mini. http:s///pppaaa..cc////e///aaaaa
Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision- language models. arXiv preprint arXiv:2501.01428, 2025. 5, 20
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007744 2024. 5, 18, 19
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python exccution for reasoning, In IEEE/CVF International Conference Computer Vision, pp. 11888-11898, 2023. 3,5
Barbara Tversky. Spatial mental models. Psychology of Learning and Motivation, 27:109-145, 1991.
12

---

Published as a conference paper at ICLR 2026
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In /EEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 21686-21698 2024a. 5
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, 2025, 5, 18, 24
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, 2024b. 5
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022. 6, 10
Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, and Tieniu Tan. Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing, 2025. URL https: arxiv. org/ abs/ 2506. 09965. 8
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao. Kang Guan, Aixin Liu. Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12. Mixture-of-expen vision-language models for advanced multimodal understanding, 2024. URL. https: //axxiv. org/abs/244 2412. 0302. 7
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can trust your answer? visually grounded video question answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 13204-13214, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL httpt:ttp////c.tc //rrxiv//vvvvv/a//aa orgg abs / 2505. 09388. 7, 9
Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Madhavan Iyengar, Shengyi Qian, David F Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 29501-29512. 2025b.
Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang. Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. Mmsi-bench: A benchmark for multi-image spatial intelligence. arXiv preprint arXiv:2505.2555 23764, 2025c. 2,
Yuncong Yang, Han Yang. Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 17294-17303, 2025d. 3
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations, 2023.
Zhihao Yuan, Jinke Ren, Chun-Ma Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for shro open-vocabulary 3d visual grounding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference, pp. 20623-20633 2024. 3
13

---

Published as a conference paper at ICLR 2026
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position- video represen- tation for scene understanding, 2025. URL https: http:///axiivvooooo http://rrgggaa///aa/eeeaeee 00493. 00493.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume System Demonstrationss, Bangkok, Thailand, 2024. Association for Computational Linguistics. URL www.:tttpp/.... /ttte:::/ariiie org/ abs/2403. 13372.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: simple effective pathway to empowering Imms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024. 8
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for source multimodal models. arXiv:2504.10477 2025.
14

---

Published as a conference paper at ICLR 2026
This appendix provides additional details and results that complement the main paper:
Implementation Details: hardware setup, inference time breakdown, efficiency comparison, and iteration analysis.
Perception Agent Execution: API specifications and the execution environment.
Module Implementations: detailed descriptions and mathematical formulations of percep- tion modules.
Additional Quantitative Analysis: component ablation studies, robustness to perception noise, and evaluation ScanQA and SQA3D.
Fine-funing Details: training configuration for supervised fine-tuning.
Failure Analysis: statistical distribution of failure modes and qualitative visualizations of eтror cases.
Qualitative Results: step-by-stey execution traces and reasoning studies.
Limitations and Future Work: discussion of perception dependence and potential verifica- tion mechanisms,
A Visualization Tool: web-based interface for inspecting code, execution, and reasoning.
The Use of LLMss: clarification that LLMs only used for polishing the text.
A IMPLEMENTATION DETAILS
All experiments were conducted on server equipped with 8 NVIDIA 3090 GPUs. For evaluations, we ran eight independent inference processes in parallel, with each process exclusively allocated to single GPU. Consequently, all performance metrics and timing statistics reported hereafter correspond to the exccution of a single such process.
A.1 INFERENCE TIME
The per-question inference time is shown Table 5. The primary computational bottleneck the latency of API calls to the large language models that serve the backbone for both the PA's code generation and the RA's deliberation (GPT-4o our case). These calls account for approximately 81.7% of the total iteration time. The execution of our local perception toolbox constitutes the remaining 18.3%. This breakdown highlights a clear avenue for future optimization through the use of smaller, locally-hosted models more efficient API endpoints.
Table Breakdown average inference time per iteration on MMSI-Bench. The dominant cost is from API calls.
Component
Average Time (s)
Local Vision Modules Execution
8.3
LLM API Calls (PA Code Generation & RA Deliberation)
37.1
Total per Iteration
45.4
A.2
ITERATION ANALYSIS
Table 6 presents the average number of iterations required for MSSR to resolve problems across different benchmarks and task types. The average number of iterations required for task completion demonstrates potential correlation with problem complexity. The more demanding Multi-hop Spatial Reasoning (MSR) sub-task within AMSS-Beench which often needs intricate multi-step information retrieval and reasoning, demands an average 2.41 iterations. Across all tasks MMSI- Bench, the average iteration around 2.15. Problems on the ViewSpatial-Best Bench were generally resolved with greater efficiency, averaging 1.88 iterations overall.
15

---

Published as a conference paper at ICLR 2026
Table 6: Average number of iterations per task. Our framework converges solution in a small number of steps, demonstrating the efficiency of its iterative information curation process.
Benchmark
Task
Average Iterations
MMSI-Bench
MSR
2.41
MMSI-Bene
Overall
2.15
ViewSpatial-Bende
Camera-based
1.84
ViewSpatial-
Person-based
1.92
NewSpatial-Bende
Overall
1.88
A.3 EFFICIENCY COMPARISON
To evaluate the computational efficiency of MSSR, compare it with VADAR (Marsili et al., 2025), a representative baseline that utilizes visual programming for spatial reasoning. We measure the inference latency and memory consumption per query on MMSI-Bench. The detailed breakdown is presented in Table 7.
Table 7: Efficiency comparison with VADAR on MMSI-Bench. The comparison highlights the cost-accuracy trade-off, showing that MSSR's higher latency is driven by the performance-critin iterative reasoning process.
Metric
VADAR
MSSR (Ours)
Programming Time (s)
24.4
20.1
Reasoning Time (s)
49.5
Execution Time (s)
28.1
17.0
Total Time (s)
52.5
86.6
Memory (GB)
Accuracy (%)
28.9
51.8
Inference Latency. As shown in the table, MSSR requires longer total inference time (86.6s) compared VADAR (52.5s). The primary source of this overhead the Reasoning Time (49.5s). Unlike VADAR, which performs one-shot visual programming, MSSR engages the Reasoning Agent to iteratively plan, filter, and curate the Minimal Sufficient Set. Crucially, this additional computation not redundant; is the core mechanism that enables framework achieve significant performance improvement (+22.9% 9% absolute gain). We observe that the programming and execution phases in MSSR actually faster, benefiting from optimized module design.
Memory Efficiency. In terms of resource utilization, MSSR maintains moderate memory usage. This consumption is only slightly higher than the baseline and remains well within the capacity of standard consumer-grade GPUs, ensuring the practical deployability of our framework.
A.4 ITERATION ANALYSIS VIEWSPATIAL--ERT BENCH
To enhance coverage, we extend the iteration-count analysis to the ViewSpatial-Bende Figure illustrates observations consistent with results MMSI-Bent (Figure 5). As the Reasoning Agent refines the information set through iterations, the average element count decreases from 10.5 to 4.6, while the accuracy consistently improves from 45.0% 48.5%.
16

---

Published as a conference paper at ICLR 2026
Accuracy
14
leeration
Figure Effect iterations on ViewSpatial-Bende
B PERCEPTION AGENT PROMPTING AND EXECUTION DETAILS
Prompt Structure. The PA receives prompt that explicitly defines the available toolset via the predef_as fe S_siefaturrseeeee gnatures placeholder. The prompt instructs the model to generate a program using these tools without needing explicit import statements. The core instruction formulated as follows:
Listing 1:
The system prompt structure provided to the Perception Agent.
Now here is an API of methods. You will want collect data logical and sequential manner. You don't need to import these functions in your program; you can use them directly:
API
[predef_signaturren
API Using the provided API, output a program inside the tags <program></ program to collect all necessary data for answering the question.
API Specification. The pwwdef_signatttiieeeeeepeeepeeee signuuuresss contains the function signatures accompanied by detailed docstrings. These docstrings describe arguments, return types, and provide example usage to guide the LLM in correct function invocation. An example for the object localization module is shown below:
Listing 2: Example API signature for the 3D localization module.
Gets the 3D world coordinates of object in specific image using pre- extracted gecmetric information. Args:
image (image): The image containing the object.
extrinsics (np.array): Camera extrinsic matrices. shape: (N, 4, 4)
intrinsics (np.array): Camera intrinsic matrices. shape: (N, 3, 3)
depth_map (np.array): Depth maps for all images. shape: (N, H, W) image_id (int): Index of the image (0-based). dbject_description (string): Description of the object (e.g., "white chair").
Returns:
np.array: The 3D coordinates 1x, y, 2 of the object's center in world coordinates. Returns None if object cannet be lecated.
Example usage:
extrinsics, intrinsics, depth_maps, point get_geo_info( images)
poject_3d_pos get pat__ojjct__d__p__ intrinsics, depth_maps, 0, "red sofa")
(images [0], extrinsics,
**.
def get_obbjct_3a__oiitteee (image, extrinsies, intrinsics, depth_map, md, object_ desject_dsscrittion)
17

---

Published as a conference paper at ICLR 2026
Execution Environment. Each module is encapsulated as Python class containing model weights and inference logic. Before the PA's generated code is executed, instances of these modules are injected into the execution namespace. This allows the PA to invoke them as if they were built-in Python functions.
C MODULE IMPLEMENTATIONS
This section provides detailed explanation of the implementation specifics for each specialized module within the Perception Agent's toolkit. These modules collectively enable the robust extraction of 3D perceptual information, contributing to the construction of the Minimal Sufficient Set (MSS) required for complex spatial reasoning.
C.I RECONSTRUCTION MODULE
F Functionality:
This module transforms sparse 2D images I into coherent 3D scene representation. estimates intrinsic and extrinsic camera parameters, predicts depth maps, and produces a unified 3D point cloud. Additionally, it segments and reconstructs the ground plane, providing a spatial reference for downstream reasoning.
3D Point Cloud Generation: We use rapid, state-of-the-art 3D reconstruction model VGGT (Wang et al., 2025) to estimate intrinsics K, extrinsics (Row, Tcw), and depth maps d(u, v). For each pixel (u, v), back-projection into camera coordinates is defined by:
Z = d(u, v),
Cc
Cy. Sy
Xc Zo
Yo Zo
Jz.
Xw
(Xc)
Yw
= Rew
Yc
+ +Tww,
Zw
Zce
where (Ja, focal lengths and (exs Gyy the principal point. Here (Xc, Ye, Zc)
frame coordinates. mapped to world coordinates via Rew. Tew. Aggregating these across images yields the scene point cloud.
Ground Plane Estimation: We detect floor regions using GroundingDINO (Liu et al., 2024) and segment with SAM2 (Ravi al., 2024). Masked pixels are back projecroe 3D, and PCA is applied to fit the ground plane:
n D p p= (Xw: Yw, Zw).
the resulting n points downward (in our setting, the positive component points downward), we flip its sign to ensure it points the 'up" of the real world.
Fallback: floor detection confidence is low, PCA is applied to the full point cloud; the eigenvector with smallest eigenvalue is used as the approximate ground normal.
Output: The module outputs calibrated camera parameters, depth maps, a dense 3D point cloud, and the estimated ground plane equation. These are used in later modules.
C.2 OBJECT LOCALIZATION MODULE
This module localizes objects in the 3D scene based on natural language queries. Given textual description of object, outputs the estimated 3D coordinates of the object within the reconstructed scene.
Functionality:
Object-Centra View Selection: For a query description "obj", we first tried to identify bounding boxes across all input images using GroundingDINO (Liu al., 2024). However, we observe that the box with the highest confidence score often does not correspond to the most complete or unoccluded view of the object. To address this, employ low-cost vision- languaee model (VLM), Gemini- 2.5 2.5-Flash-No. Thinking, to evaluate all scene images jointly. The VLM prompted to return the image ID that best captures clear and complete appearance of the queried object; if such image
18

---

Published as a conference paper at ICLR 2026
exists, the module returns None. This step ensures robust selection of the most informative viewpoint before localization.
2D Segmentation and 3D Projection: Given the selected image ID. with the object description to obtain the bounding box with the highest confidence. The bounding box is then refined into segmentation mask using SAM2 (Ravi al., 2024). Pixels within this mask are projected into 3D world coordinates using the estimated camera parameters and depth maps. The object's 3D position is computed as the centroid of all projected points:
GroundingDINO again
Pobj Σ Pa.
where p; E R$ denotes the 3D world coordinates of the i-th pixel inside the segmentation mask, and N is the total number of such pixels.
Output: The module outputs the estimated 3D location of the queried object, represented as a single point in the world coordinate system.
C.3 GLOBAL COORDINATE CALIBRATION MODULE
Functionalityy
This module establishes scene-level directional coordinate system (c.g., North, South, East, West or cquivalently Front, Back, Left, Right) based on natural language calibration such "the table is north of the chair." With this calibrated system, relative directional relations of other objects (e.g.. "the cabinet is to the west of the chair") can be consistently computed. An example is shown in Figure 7.
Question:
wardrobe the northeast corner of the roon, what is the southwest corner the room? Options: A: Desk, B: Bed, C: Nightstand, D: Window
Table to the room center
Target
wardrobe
0.1
00
SOUTHEAST
-01
-12
-1.0
-0.8
-0.5
-0.
-322
0.0
0.4
table
Figure 7: Coordinate Calibration Example. Here, the coordinate is calibrated by "wardrobe is in the northeast of the room"". Then the relation of the table to the room center can be calculated as "Southeast".
Calibration via Reference Relation: Given calibration statement "A is north of B," along with the 3D world coordinates of objects A (target) and B (anchor), both objects first projected onto the ground plane estimated previously:
PA (Pa)),
PB M(pp).
G
where II(-) denotes orthogonal projection onto the ground plane. The direction from anchor to target then defined the north vector:
D4
P
nN
P
19

---

Published as a conference paper at ICLR 2026
Constructing the Local Coordinate Frame: Let ng denote the ground plane normal. The west vector is obtained as
nw nG ny
By symmetry, the east and south vectors are defined
ng nw,
Thus, the four orthogonal directions (nN. ny ny, nw) form the calibrated ground plann coordinate system.
Directional Querying: To answer queries such as "the cabinet is which direction relative to the chair," set the chair anchor and the cabinet target. After projecting both the ground plane, the relative vector
P Peahinet
Pohair
The directional relation is determined by measuring the angle between and each of the basis vectors {nN, ns. ng. nw). assigning the label corresponding to the vector with the smallest angular deviation.
The module outputs calibrated ground- ground-plane coordinate system anchored to reference relation, together with directional labels for object pairs under this system.
Output:
C.4 SITUATED ORIENTATION GROUNDING (SOG) MODULE
Functionality: The Situated Orientation Grounding (SOG) module leverages Visual Prompting (Qi et al., 2025) to align complex, language-conditionee directional concepts with explicit 3D vectors.
Grounded Vector Generation: Given an input image, the camera pose, and the 3D location of target object Po (obtained from the Object Localization Module), we first select representative point near the ground plane within the object's ' point cloud. This ensures that rendered arrows appear closer to the image hottom, mitigating perspective distortion (e.g., avoiding the degenerate case where arrows collapse into single line at eye level). A sparse set of four orthogonal candidate vectors
Decare = {d, 1
1==1
parallel to the ground plane and centered at Po. is then generated to approximate compass-like directions.
Multi-View Visual Prompting: Each candidate arrow is projected into the current camera frame using the estimated camera parameters, preserving true 3D spatial relations. To further mitigate foreshortening, we synthesize a Canonical View by rotating the camera 45 upwards around the axis defined by the cross product of the camera-to-objeet vector and the ground plane normal. Formally, the virtual camera pose is
Rvin = Rot Rot(Rong: 45', (Veo X nG)),
where Vco is the camera-to-object vector and nG the ground normal. Both the Situated View (original camera) and Canonical View (synthetic top-down-like camera) are rendered and jointly input to the VLM (Gemini-2.5-Pro in our experiments). The VLM prompted to select the arrow that best matches the query description.
Iterative Refinement: To improve precision, a denser set five candidate vectors generated around the chosen coarse direction:
Drne [Rottd, 0) 0 -2255,-22.5
-22.5 00..5".0.,00 , 22.5 22.5",45"]
where d, is the vector selected in the coarse stage. The same multi-view prompting and VLM selection process is repeated, yielding the final grounded orientation vector. As illustrated in Figure 8, overlaying candidate orientation arrows on the original camera view alone can suffer from severe per- spective distortion, making disambiguation difficult. By incorporating the synthetic Canonical View, SOG enables the VLM to reason correctly about the intended direction, demonstrating robustness to challenging viewpoint effects.
20

---

Published as a conference paper at ICLR 2026
Step 4a: Selecting Closest Single Arrow
laggt drozton chiazion petig the main door
and through the The secand confirms thI, showing aroow poenting "upp"
which repronnts the diretion into the ware from the slacated perspection. "ngenw ("allow)=** both images this arrew points horizntally
VLM resocnse: L Andlyze the comeet's position and ortentation The first image shows a wel-
left, oppostta anow O "nArron (Grous):** the imaps, lses arow langer than arron and
with
large foor and amohar The camera
poatiuedd insiie scom, looking lomart lage doat aposa
the camen''s postion. Ihis represents
disccion "lawards the cameraa entaring the soom, accond image shows pointing "downn" conferning the asposil
be on euit Analyye Target Diectoo The desorption "diection eaiting the man door"
rosm, pensdig through
arrow 1. Cumpare and Select: The targe: directon is "loling the main deoe" which maats anay the currett position (wherre the camoa and through doonway,
teacing towards the space viible beyoed the door.. This is dirediin that moves away from
& frow Orieniations: The
directly comespondss the dirction "away from the cameraa" As established throug the
are bocted thrishid the lirge glass stom. Through the door, another rom outroor area with pool table vibble syeroww (Bodjon in both images. thi anow points hortooetslly right, parale door
socond, Arrow the vestor that paints away from the canera and through the door. Therefore, Arrow the cholce.
analyss foresh ortaring the frrst imge and the oecihood
"trrow (Blue)
imaae, thes aroww in netiorably shortar then armee o and das to fonsstrttringg Thi carseati
sucofictly "sway"
Solocted cloost arron:
Figure 8: Situated Orientation Grounding with multi-view overlay. The target object is first localized and grounded in the scene. Candidate orientation arrows are overlaid on the original camera view (Situated View), which suffers from perspective distortion (e.g., arrows collapsing into near-collinear directions). To mitigate this effect, we additionally render a Canonical View from a virtual elevated camera pose. Combining both views, the VLM robustly selects the correct orientation vector despite visual foreshortening in the original view.
Output: The resulting 3D vector d' is added to the information set. This equips the reasoning agent with explicit, Ainuage-conditionee orientation vectors that support downstream tasks.
C.5 NUMERICAL COMPUTATION MODULES
Functionality:
In addition to high-level reasoning modules, design lightweight numerical utilities that provide interpretable relative spatial relationships. These modules support both camera motion analysis and object positioning, serving building blocks for downstream reasoning.
Relative Camera Movement. Given two camera extrinsic matrices Eg, E1 E R4x4 the module computes the relative transformation T E, From T, we decompose translation (t.r ty. tx) and rotation angles (0z, Oy, 0z), which correspond to intuitive notions such as forward/backward, right/left, up/down, and yaw/pitch/roll. The output is dictionary of movement descriptors, e.g..
{forward: 1.0, right: 0.0, up 0.0, rotate_right 30", rotate_up 10%).
Relative Object Position. Given a camera extrinsic matrix E' e 4xx4 and an object position p E R3 in world coordinates, the module transforms p into the camera coordinate frame:
--=Ba P [p]
pC=
The resulting (æ, y, z) is then expressed right, up, and forward distances relative the camera, yielding an interpretable relation such as {forward: 1.0, right : 0.0, up : 0.0}.
Output: These numerical descriptors provide low-level geometric interface that complements higher-level grounding modules, enabling fine-grained spatial reasoning.
D COMPONENT ANALYSIS OF PERCEPTION AGENT
To explicitly quantify the contribution of individual components within the Perception Agent, we conducted systematic ablation study on MMSI-Bench By selectively removing key modules, analyze their specific impact on spatial reasoning performance. The results are presented in Table 8.
The analysis reveals the distinct role of each component:
21

---

Published as a conference paper at ICLR 2026
Table 8: Ablation study of Perception Agent components MMSI-Bench. We report the accuracy (%) on the Multi-Step Reasoning (MSR) split and the overall dataset, along with the performance drop relative to the full model.
Component Removed
MSR
Overall
Overall A
Full Model
50.0
49.5
w/o Reconstruction
28.3
30.1
-19.4
w/o Locate Module
33.3
35.7
-13.8
w/o Numerical Module
38.9
39.5
-10.0
w/o Global Calibration
43.4
40.8
-8.7
w/o SOG
40.9
44.9
-4.6
3D Reconstruction proves to be the most critical foundation, with its removal causing a sub- stantial - 19.4%% drop. This confirms that coherent 3D scene representation is prerequisite for establishing spatial relationships. It is worth noting that while this module is critical, state-of-the- art reconstruction models generally robust in practice.
Locate Module and Numerical Module show strong contributions (-13.8% and - 10.0%, respectively). This underscores that precise object localization and explicit geometric calculation (e.g., distance, relative position) are essential for reasoning, VLMs struggle to infer these metric properties directly from visual features.
Global Calibration provides consistent gains (-8.7%) by establishing unified coordinate system, which is particularly valuable for resolving riew-dependent ambiguities in multi-view scenarios.
SOG (Situated Orientation Grounding) contributes -4.6% overall. While aggregate impact appears moderate, we observe strong task-dependency: it is indispensable for orientation-specifee queries (c.g., "facing direction") where removing leads to significant failures, whereas it has minimal effect pure localization tasks.
E ROBUSTNESS TO PERCEPTION NOISE
Our framework relies on the Perception Agent to extract spatial primitives. To quantify the system's 's sensitivity perception errors and verify robustness, we systematically injected controlled noise into the outputs of key perception modules during inference on the MMSI-Bench, We simulated realistic failure modes as follows:
Reconstruction (Depth Scale): We applied multiplicative Gaussian noise to the depth maps D simulate scale and consistency errors: Dnoisy D (1 c), where ~ (N, o.)-
Object Localization (Jitter): We added additive Gaussian noise to the final 3D object centroids to mimic bounding box jitter. Given that the scene is normalized into a unit radius ball, the noise is defined as PRoisy P where 8 ~ (O00, of.)
SOG (Orientation Deviation): We introduced angular Gaussian noise to the predicted orientation vectors to simulate imprecise heading estimation.
The results are summarized Table 9. MSSR demonstrates strong resilience to moderate perception noise. For instance, substantial depth error of 10% results in only marginal performance drop of 2.3%. Similarly, the system maintains reasonable accuracy even under significant orientation noise. We observe that the framework relatively more sensitive to localization errors compared to depth or orientation noise, highlighting the importance of precise object grounding in spatial reasoning tasks.
F FINE-TUNING DETAILS
We fine-tuned Qwen2.5 Doven2.5-VA. 5-VL-7B (Bai et al., 2025) using LLaMAFactory (Zheng et al., 2024) on NVIDIA A6000 GPUs. We adopted LoRA (Hu et al., 2022) with global batch size of 32 and trained for epochs, resulting in approximately 45 optimization steps in total. A peak learning rate of 2x 10 with cosine decay and 3% warmup used.
22

---

Published as a conference paper at ICLR 2026
Table 9: Sensitivity analysis under simulated perception errors MMSI-Bench. The model shows robustness against moderate noise levels, particularly in depth reconstruction and orientation grounding.
Module
Error Type
Magnitude
Accuracy (%)
A (%)
Baseline
49.5
10%
45.2
-2.3
3D Reconstruction
Depth Scale Noise
20%
39.1
-9.4
0.2
44.6
-5.9
Object Localization
Position Jitter
0.5
37.8
-11.7
15"
45.3
-4,2
SOG (Orientation)
Angular Noise
30
40.1
-4.4
G EVALUATION ON SCANQA AND SQA3D
While our primary focus lies in multi-view, multi-step spatial reasoning, also assess our frame- work on established benchmarks to demonstrate broader applicability. To this end, we conducted evaluations ScanQA (Azuma al., 2022) and SQA3D (Ma al., 2023), which primarily target object-centric QA within pre-scanned indoor environments.
Since MSSR designed for multi-view image inputs, we adapted the imput by uniformly sampling 16 frames from each scene video. We compared our method against fully supervised 3D-V DL-VLMs (trained on in-domain data) and state-of-the-at general-purpost VLMMs in a 2-shot setting.
The results are presented in Table 10. We observe two key trends:
Comparison with Supervised Specialists: As expected, fully supervised methods like LLaVA- 3D achieve higher scores. This gap is also attributable the evaluation metries (CIDEr and Exact Match), which rely heavily text matching and favor models trained to mimic the specific lexical style of the dataset ground truth.
Despite the metric disadvantage inherent in the few-shot setting, MSSR consistently outperforms powerful general-purpos baselines, including GPT-4o, GPT-o3, and Cimini--.55-Prs For instance, ScanQA. MSSR achieves CIDEr score of 32.49, surpassing the strongest baseline (GPT-o3) by a competitive margin.
Comparison with General VLMs:
These results suggest that while MSSR is specialized for reasoning over minimal sufficient sets in dynamic multi-vies view scenarios, its core dual-agent mechanism generalizes effectively to standard 31) scene understanding tasks.
Table 10: Performance on ScanQA and SQA3D. We compare MSSR with fully supervised special- ists and gencral-purpose purpose VLMs (2-shot setting). Despite not being trained on these datasets, MSSR outperforms general VLMs, demonstrating strong generalization capabilities.
Category
Method
ScanQA (CIDEr)
SQA3D (EM@ 1)
Fully Supervised
Video-3D-LLM
102.1
58.5
(In-domain training)
LLaVA
103.1
60.1
GPT-4o
26.52
24.21
GPT-o3
33.30
21.93
Gemini-1.5-Flash
21.87
32.28
2-Shot (Generalist VLMs)
Gemini-1.55Prr
31.69
22.35
Qwen2.5
18.32
9.45
Qwen2.5-VL-
20.81
13.11
MSSR (Ours)
32.49
27.40
23

---

Published as a conference paper at ICLR 2026
H FAILURE ANALYSIS
To systematically identify the bottlenecks of our framework, we conducted manual inspection of failure cases on MMSI-Bench. Due to the labor-intensity, analyze half of all failure cases, which we believe provides a representative distribution of the system's 's failure modes, The detailed distribution of failure modes is presented in Table 11.
Table 11: Distribution of Failure Modes on MMSI--ennch We categorize the primary cause of emror for failure cases. The Perception Agent (PA) accounts for the majority of errors, primarily due to grounding issues.
Category
Subcategory
%
Description
Logic Errors
21.4% RA creates correct plans but makes errors in intermediate steps (e.g.. hallucination).
Reasoning Agent (31.7%)
Plan Errors
6.7%
RA formulates an incorrect or inefficient reason- ing plan.
Premature Decision
3.6%
RA makes decision before gathering all neces- sary information.
Object Grounding
23.0% Fails to detect or localize the target object cor- rectly.
Perception Agent (55.9%)
3D Reconstruction
15.1%
Errors due to extreme angles, lack of overlap, or scene complexity.
Segmentation
9.9%
Incorrect segmentation masks leading to wrong 3D centroids.
SOG (Orientation)
7.9%
VLM selects the wrong direction vector in the SOG modulc.
Others (12.3%)
12.3%
Question ambiguity, oulput parsing errors, etc.
Perception Failures. Perception errors constitute the majority (55.9%) failures. Object Local- ization (combining Grounding and Segmentation) is the dominant factor (32.9%). These errors have cascading effects: if an object is mislocalized early in the pipeline, subsequent spatial computations (e.g., relative distances orientations) become unreliable. These issues primarily arise with partially occluded objects or ambiguous referring expressions. 3D Reconstruction errors (15.1%) occur mostly in challenging scenarios with extreme viewing angles reflective surfaces (e.g., mirrors). However, we observe that modern reconstruction models like VGGT(Wang et al., 2025) are generally robust for most indoor scenes. SOG Errors (7.9%) typically stem from the VLM making incorrect selections among candidate direction arrows, particularly in scenes with sparse visual cues.
Reasoning Failures. The Reasoning Agent accounts for 31.7% of failures. Logic Errors (21.4%) are the most common, where the RA successfully curates the necessary information but hallucinates facts not present in the MSS or makes logical leaps. We specifically investigated the potential for "Looping" behavior where the RA repeatedly requests information the PA cannot provide. This occurs in approximately 6% of failure cases. Our framework handles this by forcing decision at the maximum iteration limit, ensuring the system always produces an answer, though often incorrect in these deadlock scenarios.
I QUALITATIVE FAILURE CASES
We provide qualitative visualizations of representative failure cases corresponding to the categories discussed above. Figures 9, 10. 11 illustrate common perception failures and orientation grounding errors, while Figures 12, 13 demonstrate reasoning errors.
J QUALITATIVE RESULTS
To further illustrate the effectiveness of our system, we provide qualitative case studies. Figures 14 and 15 visualize the step-by-step execution trace of the Perception Agent, including its intermediate
24

---

Published as a conference paper at ICLR 2026
variables and the process of incrementally populating the information set (analysis data). Figure 16 then demonstrates how the Reasoning Agent consumes this structured information and performs detailed reasoning over it, ultimately producing the correct answer.
K LIMITATIONS AND FUTURE WORK
Dependence on Perception Quality. Our framework relies geometric information provided by 3D reconstruction models. Although modern reconstruction models are highly advanced, they may still produce noisy or unstable results under challenging conditions. Such errors inevitably propagate to downstream modules.
Perception Verilication. An explicit mechanism to verify the correctness of the PA's outputs would be beneficial. We observe that PA errors typically fall into two categories: logic errors (e.g., incorrect coding logic) and numerical errors (e-g., noisy bounding boxes depth estimates). While MSSR currently mitigates logic errors through the Reasoning Agent's inspection of the generated code, numerical errors are implicit and difficult to detect without external validation.
To address this, promising direction for future research is the integration of dedicated Verification Agent. This agent would serve to validate perception outputs by visualizing them for instance, by overlaying detected bounding boxes 3D vectors onto the original 2D images- and cross-checking them against the textual query (e.g.. "Does this box correctly capture the black table?"). By closing the loop with verification step that trigger refinement requests, the system's 's robustness in complex and ambiguous scenes could be enhanced.
L A VISUALIZATION TOOL
The dual-agent pipeline naturally produces multiple forms of intermediate artifacts, including pro- grammatic code snippets, execution traces, and reasoning trajectories across iterative steps. While these outputs are valuable for analysis, inspecting them separately can be cumbersome. To address this, developed lightweight web-based visualization tool built on Flask (flask, 2025) (Figure 17). This interface allows users to conveniently observe the entire problem-solving process of given query, including code generation, execution results, reasoning iterations, and the evolution of the MSS. Such visualization serves as useful resource for debugging, qualitative evaluation, and future research. We will release the complete source code of both the framework and this visualization tool to the community.
M THE USE OF LARGE LANGUAGE MODELS (LLMs)
In this work, LLMs were employed solely for language polishing and editing purposes. They were not involved in the design of methods, the development of algorithms, or the analysis of experimental results. All technical contributions, experiments, and conclusions are the product of the authors.
25

---

Published as a conference paper at ICLR 2026
When you are facing the sink, where the window localed?
Options: Back Front Directly Directly right
Ground Trath:
Soymoiutioo completr with sore GNLL
Feeting mypocte tentter's lenge wiing reparrinne
Halk cown Hird pivkt out b0lela cana
Lead'e socies
Eone 110 2113 1787 55
Figure 9: Failure Case I: Object Grounding Error. The PA fails to correctly identify the window due to occlusion and ambiguity, returning the bounding box of the shower glass. This led to incorrect segmentation and coordinate calculation later.
images taken continucusly perspective. At the moment of the last image, in which direction
a first-person
bathroom relative you? Options: A: In front, B: To the right, C: To the left, D: Behind Ground Truth: B
shower area
Figure 10: Failure Case 2: 3D Reconstruction Error. In this complex scene with large mirror, the reconstruction module fails to predict the depths correctly. Consequently, the projected point cloud is distorted, causing metric calculations be inaccurate.
In the divection w where the wardrobe faces into the room considered the front; in which direction relative to the wardrobe is the office area?
Options Right, Fromt, C: Grond Truth:
shation
nill expere: inder camort yoderr and staetooe lnner thin
Vinscon neco sass the dedear pasicaal be butice
utrlibi paitii eduard
spta wakes aripe coracc
tieppe repie prrque
Propeding
suriners Jane and apone longe eondyy
Dartel 2000 ""oooe inaci (ierta Prosccsee Ih's vow dil our
know yoirs my
poilit layal denbul sard
deedyy
bmaei the
lrtet vith teer netrrtie nes fo The cofterly natros
Figure 11: Failure Case 3: SOG Error. Although arrows overlaid views help selection, the VLM still confused Arrow I with Arrow 3 with regard to the direction of "wardrobe facing into the room".
26

---

Published as a conference paper at ICLR 2026
The room with black chairs and a brown n table is the meeting room, and the table with blue and yellow bottle is the dining table. The lown is to the north of the outdoer sofa, Which the following stalements
GEADRING
slyen mas segest Las foom velioating cndtur redetindii
Optices:
entors nod orter
A: The cefrigenator is to the west of the lawn
Option spmiotooo nast rest "Tosl lioandd was
When person is stiting on the sola faciing the TV and TV, dining table
CI waine proride cnlaalere exgae
left
indoor sofa The mseting room to north
cuideer
wite the cntoatatioo hereors sortt. no clese coandretee tefrreenter is sipeifleanll to this putitiiv i sice
Ground Trulk C
Thurefore deartt the anp devietion the offect contime Moorract
Optinn inker sate
wnder cota
The calalltedd directiol "Guaceast" each aigles South (16-47 and Last
wiie Soutwast endles serty selte South tat. strict masuicyy,
dustout diaped (enn inglat
attanis clanty ond Sauthrat miom aimtly Taatsauth
that almawd fowe luttes to simmaly ted Jouiss
into: is Thansfore, cenidr striemer "Mparmee""
FINAL
The RA commits a logic error during the verification of Option A. Despite the PA providing clear data that the object is to the "South" (15" angular deviation), the RA attempts to perform its own manual coordinate reasoning, It hallucinates geometric rule by incorrectly inferring that shared positive X-component implies alignment and overriding the correct pre-computed data. Consequently, it falsely accepts Option A as correct and fabricates reason reject the valid Option
Figure 12: Failure Case 4: Reasoning Logic Error.
The picture is taken continuously from first- person perspective, The small sofa the bed to the northwest of the bed. In which direction is the fizeplace relative to the bed?
QUESTION INEERSTREIDNG:
Options: Northwest, B: Southwest, Sou theast, D. Northeast
The use vanta to find the direstion of the firspiase reletio to the bad. teput coset ets contonunucly tabm firet-pereen taaget, lecr reforerse is providal: "The small sofu by the
Ground Truth: D
PLANOIN:
Ti the lmage conterning the "tedd" and the "esall safa". Looriiy the Lmage consaliting the "Hknoplace"
The images are contirucas Fiint- parson perszective,, Whet
ortattation the
ocross
relartvv
undentiinn whot "sartiwact"
comnoperds
image plane.
5. Then, wilt. tose at the postitics of the fireptace relative to the bed the teage pixets to tniier les direccion.
FINAL ANSWER:
Figure 13: Failure Case 5: Plan Error. The RA commits Plan Error. Instead of requesting 3D coordinate calibration to handle the "continuous first-person perspective". it formulates a naive plan based on 2D image heuristics. It incorrectly assumes a fixed camera orientation across frames and attempts to map 3D cardinal directions directly to 2D pixel positions (e.g., equating "Left" to "Northwest"). This flawed strategy leads an erroneous derivation (concluding "Southeast" instead of "Northeast") despite correctly identifying the objects in the images.
27

---

Published as a conference paper at ICLR 2026
Question ID: 27
Question Type: Positional Relationship (Obj-Reg.)
Question:
In the living reon, the main door faces north; in which direction is the coffee table located in the dining roce? Opt Options: A: Nest, Bi fast, Et Soath, D: North
Answer:
Thought:
direction in which the wain door Living roon faces nutward is north, and coffeee table is to the west the dining roon.
the ind Mupeitte tie oet
Perception Agent
deth
hoother talees
Seccol atiiial porch th betevooo 3. beferior
Geting 20 poitin tafee so linge sitiig Higararton
plaw (ncegr)))))
Local collas talle
Ground Plane Detection
Segrnented coffee table:
Stop Finding image ground floor' using VQA
Finding objects: ground floor
Stop Bent ground ostection Image with confidencs 0.342
Image with
Stop Russing VOGT 30 reconstruction
suacriactionn "aristoo Actlen the sur asuntag "Hose morte Untttel wy
strentt
Soving Polet and
"alenpping on lagg
Situnted Orientetion Grounding: direction exiting main door
Finding objects: door, coffee table
Step Otject
Bot
Awpyang images enettaaue disctian extting
sucisuss cleta cucire sedoraa schel
gialal umat caae aedirs acoe
Succea: U2ged parton Lakkee (-0.9)) 4.894 1.127]
Figure 14: Execution trace of the Perception Agent (part I). This figure shows the first part of the Perception Agent's execution trace. The step-by-step code exccution and intermediate variables are visualized, highlighting how raw observations processed before being integrated into the shared analysis data.
28

---

Published as a conference paper at ICLR 2026
Greating
Deoate lew owil trge wih dindional prouc
Reuratanns west visions Burly Joat, tadl, baud
pettenstion)
"Choaaeeing sentar stroccio "toftte intettv
catarat lifo cffie parttics
Selected dosest
acee ceter contact toletto DentrrMaorr
Camera waks in wordd [1.000, 0.000, -0.000]
1211 cof wwu.oo mertte wtt.://wwwwtttttttotoooooo www.toocsscceee....... A.t.mttttteeettt
40.000)
Cac. aing wall diedin...
Perpendicular vector (VP): (0,0c0, -00002, 1.000)
Criited fro gminedd
(22.5* spacing):
Thrget is the WEST of Center
Eleveted fineggriined overler
Teponamte" 55 vedte rorcoentin the atrection cetting the note dor, entih to onteront
aelp arsuuearrt www.ooaaaarrttttiiiioooo grount ond www.sooootcctto......o cecaant I-1
22. Nosu laster" war connlaccts the and rite recc orienctiion.
cocnt restiren nemerent fremerk
tesoation stecs wener/eeeeiiieeeeeeerr
resuttar desssrea
The fiems seris
150 grand smatt
of Deenttouiee pelo citaroxs.ree.weeet
Celouiin redorna fore dieciin secoi..
Grouugg aadooa..
puresi and punn deramaad
twil peneremei sittt 1,89, 6.60
"sonth': catiratton... infell wotth J.tslsee..
"suth": setoaaation.nn trtef worhh J.stise..
Figure 15: Execution trace of the Perception Agent (part II). A continuation of Figure 14, showing how the Perception Agent progressively fills the analysis data structure. This illustrates the transition from level perception results to a coherent, structured representation of the scene.
29

---

Published as a conference paper at ICLR 2026
Reasoning Agent
STAGE PLAN-GUIDED INFORMATION CURATION
Reasont ng
The task determine coffee table's cardinal direcion the dining room, given that the door faces north.
Steps needed:
main door establish north.
Calibrate the cordinal directions (north, south, east, west).
Identify coffee table's direction relative to calibrated system.
Only keep information directly needed confirm steps.
Information Curation Process:
defjectddtecction.art lod_localizaaion context General description, not needed. Discard. object_detection. ait_and__ooccl local iiaiiiininiooo object_ jejec__iage__iiiii image_ids IDs not needed. Discard.
object detection and local pecalization..e main. door position Row coordinates, superseded by orientation groundiig.. ng. Diacard.
oeneteddeeecciioo_____t__ localization.
coordinates, superseded by
final calculation. Discard.
situated_oom ceuuate__orcnttooe centtation_gronddiii context General description, not needed. Discard.
ddor_door_eeeeeee exiting_directtion vector Establishes north. Keep. situated orientation www.innngooougnnnneeeeee main door poctrrposstiooo_e For_orrentooion Reference point for calibration. Keep.
situteed_cc orientation.oo www.riiinngooonngg...o
rection calibration, context General description, needed. Discard. calibrated rectionn + Defines cardinal directions. Keep. cable_direction caaicctiooncccllaattiii calccultionnn context General description, not needed. Discard. coffee_table www.ddeetionncco cal.__alclllaiaaa www..ulattionddreettioonnncc.mn. direecion__elltiveee.. sellatve__t__sceee Provides final enswer for coffee table direction. Keep.
coffee_
Refined mss Keys:
situutee_.oreenaaiion
situated_ orientation grw.iiiinnngungng....gmm. wwwuundig..aiinn..mmm exitinngdireccton dwwecciion.veeeeee vector
situated_ ceantted_orieetttie wwwiortationnoooeeeeee grw.nniingg.ggggn..om main coiinddoo__poo pwtppppssttooofooott _or_ooioettttiioooo
cardinal_directionn ialitration cal illibbated_dee dilibrae_diieectiioo rections
coffee_ Lable_d:rection_ca culatioin www.taii...eeeeeionnnnoooomo leosceee_lo_scene
STAGE 2: STRATEGIC MECISIGN-MAKING
Decision Analysis:
The refinad set contains:
North orientation from the main door. Calibrated cardinal directions.
Coffee table's corputed relative direction.
This is sufficient to solve the problem.
{
1 stands for <Decide>
"action": 1,
"refined_keys":
"si www.eeettiiongrroooooooooooo "si tsituatdd_doom www.:ootiiinnnnoooeeeeeee grounding main_ potoor_posttiooofoooo wor.ooocntattion
main_door_exit wwwtiinnndeeetio..eeeeeee
"cardinal direction calibration. calibrated directions",
ani.o.uu.aiioonoo....oooo..o http://c........oooiiieeoooooooa sirrection.rllatiieeto.ccco
].
"reasoning": "The main door orientation grounds north. Calibration establishes the scene's caldinal directions. The coffee table's computed relative direction 'west", which natches option A.",
"Final_answer": "A"
Figure 16: Reasoning over analysis data. The Reasoning Agent receives the populated analysis data from the Perception Agent and performs detailed multi-step reasoning. By grounding its inference in the structured information, the agent arrives at the correct final answer, demonstrating the synergy between perception and reasoning.
30

---

Published as a conference paper at ICLR 2026
f Dual-Age Agent Results Viewer
Velles dech
Recoriaben -A LEREH RCAOR StSrits
porne's Tammuyyn seght unee Waa erropontt atrmary de parsie
riatio
siem modes mitie "rwee" Hart mesmooess
mat Jestrt deatuio nxtoor restte comernatio miten
Iees wratti masiii
Collected Data
toal cator Jestwet
wise DUADO acdeta (SHAA))
wecyinng swen pcricion anvenetiie for slours (hrage analian your
Mizakiing csecion Coea omegpeet relocie oner aicuee, Aractly atwer dowt
areepaert
mise ("arme"" Ce0LERIHHNN may "aggtt Caattdatioe eit,, sP" cnnooouaaee
"egget
prrpocttir
Prolding Crendie rilaii polisios comprren na sace masies antecia
fomon DocOme1: Lww.EMOOHOA.
Reasoning Agent's Reasoning
OocotAA poparn
izeatioarier
"ideel
"Flowe annoggeeet" detzenlntig
weld."
Fied ladge shere flower crrangrantt apprese
magee,, pospct)
analyete, Assee" Floeer wor anggnnee Leaer MA.N 1 Rowe pwworpeirt__teee 16.
flowr sott
if "itoer onc spenens Ua itown, wuw.erpreeeee phorgpeett. Lnage, 14.g..14.0.. aop
trag anvergment floeer orrangment laage wap flower areagemntt)
pestston(
tingeeElnegg floer
estrinalca,
txtriratic.
flowr
"Flowe onr peresseans"
"sucripstents "The scrld canne satue
"olue" rlawe. arrergeert ttet
Connno Ieformation
analyeta, comerss pees, dnformation. contect =>% "ealyyting cawrr's postition and ontantation the test pieture (hrage sctablto "yorn pertoactive."
director
steum
Figure 17: Web-based visualization tool for MSSR. The interface integrates code, execution traces, and reasoning trajectories unified view, enabling convenient inspection of the full problem-solving process.
31


Reviews:
[{"review_id": "3eJzp4bg2E", "reviewer": "Reviewer_9uKo", "summary": "This paper proposes MSSR, a 3D spatial reasoning framework that constructs a minimal sufficient set of spatial facts before answering a question. The model comprises two agents: A perception agent that uses visual programming with tool calls to extract 3D information, and a reasoning agent that plans and iterative requests missing information until the set if sufficient. The model is evaluated on MMSI-Bench and ViewSpatial-Bench, claiming SOTA against prior works.", "questions": "Please see the above section for my questions regarding the ablations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model gives good motivation with Figure 1, introducing attention delusion. \n\nThe baselines compared are quite strong, including both closed and open-source models, as well as specialist and other visual programming frameworks like VADAR.\n\nThe design of the model allows plug-and-play of powerful models like VGGT, GroundingDINO, SAM2, etc, which can be easily swapped out with better models.\n\nThe ablations are quite thorough with the key components (Only PA and Only RA, without iterations, etc.).", "weaknesses": "This paper is generally quite strong in my opinion. My only issues are perhaps more datapoints for the ablation.\n\nFor Figure 5, I think the paper would be better benefitted by showing 1) more iterations to see whether more iterations would cause models to have errors in eliminating the number of sets needed for a correct answer and therefore leading to worse performance and 2) on more than just a subset of the MMSI-Bench.\n\nFor Table 2, it would be great if the authors can also provide information on ViewSpatial-Bench for a more comprehensive ablation.\n\nHowever, these are just minor points in general, and I would think the paper’s idea is in fact quite solid. I am therefore leaning towards accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGib5c1JRl", "reviewer": "Reviewer_gXuZ", "summary": "The paper introduces MSSR (Minimal Sufficient Spatial Reasoner), a dual-agent framework aimed at improving 3D spatial reasoning in vision-language models through explicit pursuit of minimal sufficiency. The approach separates perception and reasoning into two collaborating agents: a Perception Agent and a Reasoning Agent. The framework achieves competitive results on MMSI-Bench and ViewSpatial-Bench.", "questions": "1. Does the “only PA” setting in Table 2 refer to using PA to collect all information without any filtering, or does it mean collecting subset information for the first iteration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method is well-motivated, as MSSR presents a sufficient yet non-redundant agent framework that enables effective spatial reasoning.\n\n2. MSSR achieves strong performance on two challenging, vision-centric spatial reasoning benchmarks.\n\n3. The paper is well-organized and easy to follow.", "weaknesses": "1. The state-of-the-art claim is unconvincing since Table 1 omits recent strong 3D-LLM baselines (e.g., Video-3D-LLM, VLM-3R) that are directly relevant to spatial reasoning. Including these comparisons is necessary for a fair evaluation.\n\n2. Although the paper includes ablations on the PA and RA, it would be beneficial to further analyze how individual components within the Perception Agent, such as the spatial reasoning modules, foundational scene reconstruction, and global calibration, contribute to overall performance.\n\n3. One potential concern is that MSSR appears to lack a mechanism to verify the correctness of the information provided by PA. Although the RA agent continuously queries PA and filters out irrelevant information, it remains unclear how the framework handles spurious or inaccurate inputs from PA. For future extensions, incorporating a verification module to ensure the reliability of PA’s outputs would be a valuable improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HLG4ktnMEx", "reviewer": "Reviewer_QPFG", "summary": "The paper proposes MSSR (Minimal Sufficient Spatial Reasoner), a dual-agent, zero-shot framework for improving 3D spatial reasoning in Vision-Language Models. The framework works by constructing a Minimal Sufficient Set (MSS), the smallest subset of 3D perceptual information sufficient to answer a query, by a dual agent iteratively. MSSR consists of a Perception Agent, which programmatically queries a scene using modular visual tools including a new Situated Orientation Grounding (SOG) module for robust direction grounding, and a Reasoning Agent, which prunes irrelevant information and iteratively requests only what is missing until the MSS is achieved. Experiments on MMSI-Bench and ViewSpatial-Bench show MSSR achieving state-of-the-art results, outperforming models like GPT-4o and Gemini 2.5 Pro.", "questions": "Please refer to the weakness section.\n\nMinor typo:\n- line 165, $S$ and $S^\\*$  should be $\\mathcal{S}$ and $\\mathcal{S}^\\*$", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and logically structured, making the methodology and motivation easy to follow.\n- The related work section provides a well-organized and comprehensive discussion covering VLMs for spatial reasoning, agentic framework, and visual programming paradigms.\n- The proposed MSSR framework is conceptually sound and technically well-motivated.\n- Extensive experiments demonstrate strong empirical performance, achieving state-of-the-art results on both MMSI-Bench and ViewSpatial-Bench while maintaining interpretability through reasoning traces.", "weaknesses": "- The paper formatting does not fully comply with ICLR submission requirements.\n- The approach heavily depends on the accuracy of perception modules (e.g., reconstruction, localization, orientation grounding); sensitivity or robustness analysis is lacking.\n- The criterion for minimality of the MSS is conceptually emphasized but lacks clear quantitative verification - how to ensure the generated MSS is minimal?\n- Details on the prompting and coordination between the Reasoning Agent (RA) and Perception Agent (PA) are insufficient—specifically, how the RA is informed of the available tools or APIs.\n- Both benchmarks used (MMSI-Bench and ViewSpatial-Bench) are newly proposed and not yet peer-reviewed; the authors are suggested to conduct evaluations on established benchmarks such as SQA3D[1] or ScanQA[2].\n- The paper would benefit from qualitative examples of failure cases.\n\n\n[1] Ma, Xiaojian, et al. \"SQA3D: Situated Question Answering in 3D Scenes.\" The Eleventh International Conference on Learning Representations.\n\n[2] Azuma, Daichi, et al. \"Scanqa: 3d question answering for spatial scene understanding.\" *proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojJsMRTgfB", "reviewer": "Reviewer_KaHi", "summary": "This paper proposes MSSR (Minimal Sufficient Spatial Reasoner), a dual-agent framework intended to improve spatial reasoning in vision-language models. The framework introduces a Perception Agent using a versatile perception toolbox (including a novel SOG (Situated Orientation Grounding) module) to extract 3D information, and a Reasoning Agent that iteratively prunes and augments this information to curate a minimal sufficient set (MSS) required for reasoning about a given question. Experimental results on two challenging benchmarks (MMSI-Bench and ViewSpatial-Bench) demonstrate state-of-the-art performance. The framework can also produce interpretable reasoning paths as training data for future models.", "questions": "1. How is the quality of the constructed Minimal Sufficient Set ensured? Are there scenarios where the Reasoning Agent might prune critical information or request unnecessary information?\n2. What are the common failure modes of the model? Does it struggle (e.g., the Reasoning Agent never makes the final decision) when the question is very complex?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The designed dual-agent framework is meaningful and effective.\n2. The proposed method is training-free, making it easy to integrate seamlessly with existing models.\n3. The method achieves state-of-the-art performance, outperforming all proprietary, open-source, specialist, and agentic models.", "weaknesses": "1. The paper only selects a proprietary LLM (GPT-4o) as the backbone to demonstrate the effectiveness of the method. Additional experiments and analysis on open-source LLMs (e.g., Qwen2.5-VL) could further validate the efficacy and generalizability of the method.\n2. The paper only provides the inference time per iteration. A thorough efficiency comparison with previous methods and corresponding analysis could further enhance the soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)