From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
Authors: Yifu Yuan
Haiqin Cui
Yibin Chen
Zibin Dong
Fei Ni
Longxin Kou
Jinyi Liu
Pengyi Li
YAN ZHENG
Jianye HAO
Venue: ICLR 2026 Poster (2026)
Keywords: Spatial VLMs
General Robotic Manipulation
VLM Reasoning
Spatial Chain-of-Thought
Primary area: applications to robotics, autonomy, planning
TL;DR: We proposed FSD, which enhances the generalization capability of robotic manipulation by enhancing spatial visual reasoning.

Abstract: Achieving generalization in robotic manipulation remains a critical challenge, particularly for unseen scenarios and novel tasks. Current Vision-Language-Action (VLA) models, while building on top of general Vision-Language Models (VLMs), still fall short of achieving robust zero-shot performance due to the scarcity and heterogeneity prevalent in embodied datasets. To address these limitations, we propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning, providing fine-grained guidance for robotic manipulation. Our approach combines a hierarchical data construction pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals. Through extensive experiments, we comprehensively validated FSD’s capabilities in both “seeing” and “doing”, achieving outstanding performance across 8 benchmarks for general spatial reasoning and embodied reference abilities, as well as on our proposed more challenging benchmark VABench. We also verified zero-shot capabilities in robot manipulation, demonstrating significant performance improvements over baseline methods in both SimplerEnv and real robot settings. Experimental results show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.

Published as a conference paper at ICLR 2026
FROM SEEING TO DOING: BRIDGING REASONING AND DECISION FOR ROBOTIC MANIPULATION
Yifu Yuan", 1 Haiqin Cui Yibin Chen 1.9 Zibin Dong', Fei Ni', Longxin Kou*
Jinyi Liu", Pengyi Li', Yan Zheng's Jianye Hao
Colleee of Intelligence and Computing, Tianjin University
'HC and YC are co-second authors.
"Corresponding authors: Jianye Hao (janye.ha@@tu..ddu
ABSTRACT
Achieving generalization in robotic manipulation remains a critical challenge, par- ticularly for unseen scenarios and novel tasks. Current Lision-Language-Action (VLA) models, while building on top of general Vision-Language Models (VLMs), still fall short of achieving robust zero-shot performance due to the scarcity and heterogeneity prevalent in embodied datasets. To address these limitations, we propose FSD (From Seeing to Doing), a novel vision-language model that gener- ates intermediate representations through spatial relationship reasoning, providing ane-grained guidance for robotic manipulation. Our approach combines a hierar- chical data construction pipeline for training with self-consistency mechanism that aligns spatial coordinates with visual signals. Through extensive experiments, we comprehensively validated FSD's capabilities in both "seeing" and "doing", achieving outstanding performance across 8 benchmarks for general spatial reason- ing and embodied reference abilities, as well as our proposed more challenging benchmark VA Bench. We also verified zero-shot capabilities in robot manipula- tion, demonstrating significant performance improvements over baseline methods in both SimplerEnv and real robot settings. Experimental results show that FSD achieves 40.6% success rate SimplerEnv and 72% success rate across real- world tasks, outperforming the strongest baseline by 30%. More visualizations and datasets are available on website.
INTRODUCTION
A driving force behind robotics research is the pursuit of generalization: creating agents capable of versatile action across diverse robotic platforms, extending beyond familiar tasks, objects, and environments while adapting to dynamic visual inputs. Current approaches (Kim al., 2024; Brohan et al., 2023; Ni et al., 2025) leverage pre-trained LasguaLanguuage Models (VLMs) and transform them into Lision-Language Action Models (VLAs) using large-scale embodied datasets. The intention is to capitalize on the generalization capabilities of VLMs pre-trained on internet-scale data, with the hope that resulting VLAs will adapt to novel scenarios. However, empirical evidence (Zheng et al., 2024; Zawalski et al., 2024; Liu et al., 2024e) demonstrates that this approach falls short of achieving strong zero-shot performance on completely novel tasks.
We attribute the limited generalization in existing VLA-based systems to two fundamental challenges: data scarcity and heterogeneity. Firstly, robotics data remains limited compared the vast scale of language and vision datasets, preventing the emergence of similar scaling laws (Kaplan et al., 2020; Lin et al., 2024a). Secondly, the heterogeneity of robotic embodiments (Wang et al., 2024) introduces significant variations in action spaces and physical interactions, making the end-to-end supervised learning of a direct vision-to-action mapping a potentially unrealistic path toward true generalization. To bridge this generalization gap, the community has explored several paradigms. End-fo-ent VLAS (Black et al., 2024; Brohan et al., 2023) attempt a direct mapping from multimodal inputs to low-level actions, but the disconnect between pre-trained cyberspace data and physical action modalities can lead to knowledge forgetting and task conflicts. Modular Methods (Huang et al., 2024; Liu et al., 2024a) chain together specialized models for tasks like object detection and grasping,

---

Published as a conference paper at ICLR 2026
Put the pot blue cloth
Zero-shot Manipulation
Encoder
Firetuned
Projectoo
SimplerEny
Colano
Large L Language Model
cieco citeeo
othins
contemn <onnams
Whencfffree
Visuol Ald Senerction
XArm Robot
workbench green sponge, anack packet, and blue cookies on top
object camera takinn this photo?
black rebotic
orm located top left the round con labeled BEANS
Top
"Books"
Region Caption
Spattai Reasoning
Spotial Groph Description
Spatial Affordance Box
Spatial Affordance Point
Visual Trace
Figure Overview FSD. FSD unlocks visual aids reasoning and generation through Spatial Relationship- Focused CoT, demonstrating exceptional generalization capabilities that enable zero-shot robot manipulation and achieving remarkahle performance across multiple benchmarks.
but these pipelines prone cascading failures, difficult to tune, and suffer from high inference latency and a lack of holistic scene understanding. Affordance Methods (Yuan et al., 2024b; Xu al., 2025; Zhou al., 2025a;b) offer a promising direction by predicting intermediate visual aids (e.g., grasp points). However, they remain limited, often providing aids that are not comprehensive enough for complex decision-making and predicting raw coordinates without an explicit reasoning process, which hinders their ability to anchor instructions to the correct semantic entities.
We argue that the key to generalization lies not merely in predicting visual aids, but in first conducting explicit reasoning over the spatial and semantic context to generate an expressive, embodiment- independent intermediate representation. This representation, composed of visual aids like spatial affordance boxes and visual traces, provides compact yet powerful information for downstream decision-making To this end, we propose FSD (From Seeing to Doing), a novel framework that generates these visual intermediate representations through structured spatial reasoning (Fig. 1). FSD (SrCoT):
comprises three key components: Spatial Stationshii-focusse visual chain-of-thoug This core mechanism treats visual aid generation as a reasoning process. It first performs a step- by-step textual analysis of the scene to identify relevant objects and infer their spatial relationships. Only after this semantic and spatial understanding is established does it generate a comprehensive set of object-centric visual aids. Hierarchical data construction pipeline: We integrate large- scale embodied datasets with commonsense data to build the foundational capabilities required for embodied reasoning, enabling a weak-to-stront capability enhancement during training. 3 Self- consistency mechanism: This aligns the model's understanding and generation by binding spatial coordinates with specific visual signals. ensuring the generated aids are well- rell-grounded. Besides, manually annotated 300 images from diverse real-world scenarios to create the realistic and complex Visual Aids Generation Benchmark (VABench) evaluate our approach.
FSD is an enhanced affordance-basee VLA that generalizes effectively to new instructions and scenes through its reasoning abilities. Our contributions include: 1) A novel paradigm where VLM reasoning generates versatile visual aids, enabling either direct open-loop control or serving as the high-level planner in hierarchical closed-loop policy; The SrCoT method with self-consistency mechanism for multi-step reasoning and guiding zero-shot manipulation; 3) Our weak-to-strong spatial reasoning and visual aids datasets, along with VABench; 4) Superior performance across 8 benchmarks in spatial reasoming, free space reference, and visual aids generation, achieving 40.6% zero-shot success in SimplerEnv and 72% in 8 real-world tasks, outperforming the RoboPoint baseline by 30%.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Spatial Understanding and Reasoning with VLMs Spatial understanding and reasoning, the ability to infer spatial relationships beyond 2D images, is crucial for embodied Al applications like naviga- tion (Song et al., 2024; Hong al., 2023) and manipulation (Yuan et al., 2024b). Recent advances such as SpatialVLM (Chen et al., 2024). SpatiaIRGPT (Cheng et al., 2024), and SpatialBot (Cai et al., 2024) have enhanced geometric reasoning by explicitly modeling spatial primitives and relationships. FSD specifically targets embodied manipulation by introducing a novel SrCoTT mechanism and sophisticated self- ceff-consisteney alignment technique to advance spatial reasoning for complex tasks.
Visual Chain-of- -hoouht Reasoning Chain-of -fo---uugtthtttttt reasoning (Wei et al., 2022) has significantly enhanced LLM performance by structuring step-by-step thought processes. This paradigm has been extended to multimodal domains to tackle complex visual reasoning challenges (Mitra et al., 2024; Zheng et al., 2023a). Approaches like Shikra (Chen et al., 2023) and VoCoT (Li et al., 2024d) anchor reasoning in specific visual regions, while EmbodiedCoT (Zawalski et al., 2024) pioneered its application in embodied Al by fine-tuning VLAs to generate intermediate reasoning steps. In contrast, FSD advances this area by uniquely employing spatial relationship graphs reasoning anchors, creating a more structured and grounded visual-spatial reasoning process.
Visual Aids Empowered Robotic Manipulation Extracting embodiment- aggnosiii visual aids to enhance training efficiency has emerged as a promising paradigm in robotic manipulation. Numerous studies (Bharadhwaj al., 2024; Wen al., 2023; Xu al., 2024; Zheng et al., 2024; Yuan et al., 2024a; Ni et al., 2024) have explored robotic policy learning based on visual traces. though most remain confined to specific tasks with cross-embodiment applicability. LLaRVA (Niu et al., 2024) advances this field by predicting visual traces to better align visual and action spaces for robot learning, compiling an impressive large visual trace dataset, yet struggles to generalize to novel downstream tasks without task-specific fine-tuning. Spatial affordance represents another effective visual aid, with several works (Yuan et al., 2024b; Mo et al., 2021; Qin et al., 2020; Song et al., 2024; Ji et al., 2025; Yang et al., 2025; Li et al., 2025; Yuan et al., 2026) demonstrating its utility in robotic manipulation tasks. Consequently, FSD adopts a reasoning- -riven paradigm, leveraging the VLM's general world knowledge for visual aids generation, thereby enhancing zero-shot generalization to unseen scenarios.
BRIDGING REASONING AND DECISION VIA VISUAL AIDS GENERATION
To harness VLM perception for cross-domain generalization, we propose FSD (From Seeing to Doing). Our approach generates embodiment-agnostin visual aids (spatial affordances and traces) via novel reasoning mechanism: the Spatial Relationship-Fos Focused CoT (SrCoT). SrCoT uniquely leverages object coordinates and their spatial relationships as reasoning anchors to produce precise visual aids. Then, FSD trained with a progressive weak-to- to-stonn data construction pipeline, further refined by self-connistency alignment mechanism.
3.1 DEFINITION OF VISUAL AIDS
As shown in Fig. 2, FSD utilizes three visual
aids of increasing complexity, all defined within normalized image coordinates x (p. q) C IR Spatial affordance boxes 8 = [x1, 31, 22 V21 define target placement regions, including un- marked free space beyond standard detector capabilities; Spatial affordance points P ((*,, y;) i= = 1, 2, ... n provide more precise and flexible targets; and Object-centric visual traces T {x, t-
Put the sushi into the silver pot
Spatidl Affordance Bounding Box
Spotial Affordonce Paints
Visual Tracs
Figure 2: Definition of visual aids with increasing com- plexity. FSD leverages spatial affordance boxes for region-level guidance, points for coordinate-level pre- cision, and visual traces for temporal sequences in an object-centric coordinate system.
which are ordered coordinate sequences describing com- plex, cross- eenssimmbdiment manipulation trajecto- ries. Our choice of an object-centric, rather than agent-centric, approach is crucial: it circum-
3

---

Published as a conference paper at ICLR 2026
Hew drow visual trace for the task?
I L'Where is the broccel? (Start Peint)
2 Which area is the pat? (End Point)
3 How cornect the stort and and points to forn troce?
(A stroight Iine the smpplest)) x
& How to avoid collitions?
(Pick the vegetables with ourve.)
Task instruction: put the brecooli from the left burner into the silver pot
<Descnption
Braccali
Silver
cffforccoll///eeeeeee setbboxx./bbb/ ./bose is positioned spredbleft oft/pred
Left of
onefrallver .. (Spatial Relationship)
</bescription
(Reosoning
Behind
To the broccoli from the burner to inside the silver pot, start by identifying spoints[ (561, 369]]//point, to the right, moving towards the pot. reaching approximately spoint[[633 319]]//point> Continas moving over the pot ot (ponnts[776. 304 [|//points, Finally, lower the broccoli into the pet, ending of the terget position (boxe 377, 876, 478l]//boo with the finel point <point:[[83S, 443])/ppinnt.. </Reasoning:
pesition broccoli
broccoli diggttly upwards
Next to
Left Burner
Knife
moving broccali pat [rppanns
(761
Figure 3: Inspired by the process of human reasoning, FSD uses a spatial relationship graph as an anchor to derive a visual chain of thought reasoning process for visual trace generation.
vents the limitations of heterogeneous embodied data and leverages general visual datasets to achieve robust generalization.
3.2 SPATIAL RELATIONSHIP- VISUAL CHAIN-OF-THOUGHT
To enable VLMs to generate spatial visual aids, direct approach is supervised fine-tuning (Niu et al., 2024; Yuan et al., 2024b; Li et al., 2024b; Zawalski et al., 2024) using these aids new action space or employing generative models (Shridhar et al., 2024; Xu et al., 2024). However, the heterogeneity and scarcity of embodied datasets limit this method. Directly aligning RGB images with coordinate points is challenging and prone to overfitting, hindering generalization. How can we stimulate VLMs' spatial reasoning abilities guide the generation rather than merely relying on extensive demonstrations? Inspired by human cognition (Fig. 3 (Top)), when executing tasks like "putting broccoli into a pot," humans first locate relevant objects, then plan movement paths based on relative positions while considering feasibility and obstacles. During this process, humans construct reasoning chains, repeatedly referencing object positions and establishing spatial relationships.
Based on these considerations, we introduce Spatial (SrCoT). This approach guides VLMs to generate visual aids through structured reasoning based on spatial relationship graphs. SrCoT consists of two essential phases: Description: We generate object-centric region captions establishing a task-relevant spatial relationship graph where nodes represent objects with their coordinates and edges denote relative relationships (above, below, left, right, etc.). Reasoning: Using the spatial relationship graph as anchor points, we determine start and end coordinates through object references and free space reasoning, then iteratively derive intermediate points with explicit logical connections between steps. Thus, we prescribe a templated reasoning path for VLMs, enabling FSD to perform analogical reasoning in the spatial domain. While VLMs struggle to directly map future actions to image coordinates, our method leverages known object relationships Fig. 3 (Bottom) demonstrates a complete reasoning sequence. This step-by-step SrCoT approach, though powerful, fundamentally depends on precise spatial understanding capabilities. To improve the stability and reliability of reasoning paths and reduce model hallucinations, SrCoT requires the model generate coordinates specified format and bind them objects while performing object-centric reasoning (Wang et al., 2025; Li et al., 2024d). We use <ref> to mark objects, and <point> and <box> mark points and boxes, respectively, ensuring that each object is strietly bound to its corresponding coordinates. This explicit visual-spatial coordinate alignment enhances the FSD 's understanding of the spatial positions and relationships of objects. All coordinates are treated as text and discretized integers normalized between 0 and 999.
Sanntiinnshi-Fooocuse Visual Chain-of-thought
reference points for multi-hop analysis, simplifying the reasoning process.
3.3 WEAK-TO-STRONG CAPABILITY DATASET CONSTRUCTION PIPELINE
The SrCoT mechanism places significant demands on VLMs, requiring capabilities like precise grounding, spatial understanding, and complex instruction following where mainstream models (Liu
4

---

Published as a conference paper at ICLR 2026
5
Level 1: Region Grounding Caption
Capabilities
Level Spatial Graph Description
Baid
DROID
RT-X
300k
Level 3: Spatial Reasoning
QAPairs
Level 4: Spatial Affordance
Embodiedments
Level 5: Visual Troce
Task Instruction: move the or orange cleth on the right side of the metal pot
Left behind
sushi, with an orange cloth on its right.
netal pot in
Template
behind
<>
silver round 1id and spoon are nearby.
robotic am is hovering above. red sushi, silver 1id, spoon, robotic
Query
Objects: metal.
right
ams
Dende Spatial Groph
Interaction:
orange cloth metal. pot
Figure 4: FSD screens data from large-scale embodied datasets, generates ground truth spatial relationship graph. We finally collected 300K data for 10+ embodiments with 5-level capabilities.
al., 2024c; Lin et al., 2024b) show limitations. To address this, developed progressive, weak- to-strong data pipeline to cultivate these abilities hierarchically. FSD encompasses five hierarchical capability levels: Region Grounding enables robots to focus on key objects in scenes. Although grounding capabilities have been broadly integrated into current VLMs (Chen et al., 2023; You et al., 2023), understanding various small objects and complex scenes for embodied tasks still limited; Spatial Relationship understanding establishes prerequisite knowledge for spatial reasoning, forming the anchor points for SrCoT; Spatial Reasoning builds upon these foundations to perform multi- hop analysis of object positions and relationships; and finally, 4 Spatial Affordance Generation and Visual Trace Generation create actionable spatial guidance. Notably, SrCoT functions as general visual-spatial reasoning mechanism applicable beyond visual traces to diverse spatial reasoning tasks. Through hierarchical spatial capability training, we enhance VL.Ms* general spatial reasoning abilities, extending well beyond just embodied domains.
Automatic Dataset Construction: We constructed a 300K Supervised Fine-Tuning (SFT) dataset by processing demonstrations from large-scale sources like BridgeData (Walke et al., 2023), RT- (O Neill et al., 2023), and Droid (Khazatsky et al., 2024). It first produces grounding data (Level by using a VLM to nominate objects and vision model to extract their bounding boxes. Next, to establish spatial relationships (Level @). ) we reconstruct 3D scene graphs using Metric3Dv2 (Hu al., 2024) and WildCamera (Zhu et al., 2024) infer the relative positions of objects. While precise depth accuracy was not required for inferring positional relationships, to ensure high-quality data, we only processed object pairs exhibiting a relative depth gap of least 20%. Then, these 3D scene graphs serve as a basis for a VLM to automatically generate complex spatial reasoning Q&A pairs (Level A core aspect of the FSD dataset the visual aids generation. We employ simple method with successful human demonstrations from embodied datasets and infer the process from the results. Spatial affordance represents the designated completion area for manipulation tasks. create spatial affordance labels (Level 4 Dataset), we extract the manipulated object's 's final position from the terminal frame, combine with reference object positioning, calculate the precise affordance region, and re-render this information onto the initial frame. For visual trace generation (Level 5 Dataset), we employ a two-stage approach: first applying self-supervised keypoint extraction (Huang et al., 2024) to identify grasp points on manipulated objects, then utilizing Cotracker (Karaev al., 2024) to capture temporal dynamics from human demonstrations, subsequently projecting these trajectories onto the initial frame. Throughout this process, employed strict rule-based filters and continually validated our approach against manually annotated test sets, iteratively refining our filtering criteria based on empirical feedback ensure the resulting dataset met our quality requirements. The dataset presentation, data filtering process, and prompts used to generate the data are provided in App.A.
3.4 SELF-CONSISTENT ALIGNMENT FOR SPATIAL UNDERSTANDING AND GENERATION
High-quality SET datasets enable VLMs to generate visual aids, yet these models struggle to under- stand the physical meaning of such annotations since coordinate spaces never appeared in pretraining data. The alignment between image coordinates and actual spatial positions presents a significant challenge. Therefore, we propose self-consistency mechanism to further align FSD capabilities in
5

---

Published as a conference paper at ICLR 2026
spatial understanding and generation. We frame generation tasks inversely understanding problems: the forward task requires inferring visual trace T from an image X, and task instruction X,, i... Xg) - T, we construct the inverse task of predicting possible instructions given an image and visual traces (X.. +) - x.- This bidirectional approach helps the model comprehend spatial coordinates' meanings and aligns coordinate space with image-text modalities, unifying visual aids as both understanding and generation signals while enhancing FSD spatial reasoning capabilities.
TRAINING AND ACTION EXECUTION OF FSD
Training: We adopt the instruction tuning pipeline from LLaVA-1.5 (Liu et al., 2024c), as illustrated in Fig. FSD's architecture features a frozen CLIP-ViTH (Gao et al., 2024) image encoder and a Vicuna-13B (Zheng et al., 2023b) LLM, which are connected by a trainable linear projector. This projector maps visual tokens into the language embedding space. We initialize our model from ASMv2 (Wang et al., 2025), which provides a strong foundation with its inherent grounding and relation conversation abilities. The training process unfolds in two stages: General Spatial Reasoning Enhancement: In the first stage, we use our Level 1-3 data to cultivate the model's is core embodied spatial reasoning. To prevent catastrophic forgetting and retain pre- pre-trained knowledge, employ a mixed-data strategy, combining our robotics data with a diverse 1 1.4M sample of general visual question answering (VQA) and internet data. This joint training crucial for developing robust embodied capabilities. Visual Aids Generation and Understanding: Using data from levels 4-5 with the self-consistency mechanism, we specifically train visual aids generation and understanding abilities. FSD predicts a fixed set of 8 points for simplification when generating spatial visual traces. Additional training details and the summary of mixture datasets are provided in App.B.
Action Execution: FSD can reason from initial or intermediate task steps, freely selecting needed visual aids. When using bounding boxes, we sample the center as the target point; with affordance points, we directly sample one point. For visual trace execution, we first generate 2D visual traces T and obtain preliminary depth information from depth cameras. Following the pinhole camera model, we employ depth-based back projeprojection to map these to 3D space, yielding 7d = {x² t == 1, T.). Next, based on the spatial position of the first point X1, we query GraspNet's 's (Fang et al., 2020) grasp candidates G to match the nearest grasp pose G*. For relatively fixed scenes, we may also use predetermined grasp poses. Subsequently, we optimize the path trajectory using gradient descent-based interpolation, generating complete motion trajectories in SE(3) space, enabling the robotic arm to follow the 3D visual trajectory. When using only spatial affordance, we utilize CuRobo (Sundaralingam et al., 2023) as the motion planner to determine execution trajectories T based on the target position P. More details are provided in App.C. Unlike methods such as LLARVA (Niu et al., 2024) and EmbodiedCOT (Zawalski et al., 2024) which also utilize visual auxiliary aids, FSD transforms prediction tasks into reasoning tasks, better leveraging visual-spatial common knowledge without requiring -epeaii fine-tuning.
VISUAL AIDS GENERATION BENCHMARK
Existing datasets for evaluating visual aid generation are scarce, with benchmarks like Where2Place (Yuan et al., 2024b) being limited to simple instructions and a notable absence of benchmarks for visual trace prediction. To address these gaps, we introduce the Visual Aids Genera- tion Benchmark (VA Bench), comprising 300 problems manually annotated from diverse world datasets (OXE (O'Neill et al., 2023). BridgeData (Walke et al., 2023), and Droid (Khazatsky et al., 2024)). VABench requires models to infer visual aids from natural language instructions that mimic everyday commands and is evaluated across two main tasks. For spatial affordance (VABench-P), we measure the proportion of points falling within target regions, sampling uniformly from predicted bounding boxes when necessary. For visual traces (VABench-V), we compute the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) against the ground truth, normalizing coordinates to a 1000x1000 space and using interpolation to ensure consistent evaluation. Recognizing that many instructions admit multiple valid solutions, we supplement these quantitative metries with a qualitative LLM Score (1-10) assigned by an MLLM based on detailed assessment criteria that simulate human judgment. A complete description of our evaluation procedures is provided in App.D.

---

Published as a conference paper at ICLR 2026
new
ner is me
Figure FSD directly generates visual aids based on instructions for novel tasks and scenarios. 1st row: affordance bounding boxes; 2nd row: affordance points; 3rd and 4th rows: visual traces.
Table Performance comparison on 5 spatial reasoning benchmarks. Bold underlined values show best and second-bes performance among open open-source models.
CVBendh
CRPE
SAT
BLINK RelDepth
EanhSp Text
Rank
Coust
ZDRe
3DDeg
3DDis
Asg.
Exit.
Suly
Pied.
Olg.
Avg.
Val
Real
Coust
MV
Spkel
Arg.
Clorghorare modely
71.1
68.5
44.5
62.2
36.1
GP544
65.5
87.4
16.2
79.4
71.8
73.
95.3
49.4
$7.5
60.3
16.2
66.3
63.
49.1
Opre model
LLaV 1...38
58.2
46.6
33.0
478
51.4
65.7
57.4
54.2
55.2
63.5
51.4
41.6
45.0
41.4
53.2
69.0
52.4
35.1
48
$1.5
76.2
28
ASM622 13B
68.9
56.3
57.4
FSD 16
62.4
56.5
88.8
96.7
80.9
94.8
15.2
68.1
10.5
76.2
25.1
63.3
96.6
10.2
18.3
635
633
1,3
6 EXPERIMENTS
We evaluated FSD across two dimensions: Seeing and Doing. For Seeing, we tested its general spatial reasoning and visual aids generation capabilities. For Doing, we conducted zero-shot manipulation experiments in both SimplerEnv (Li et al., 2024c) simulation and real-world xArm robotic platforms assess its practical generalization performance.
6.1 EVALUATION OF SPATIAL UNDERSTANDING AND REASONING CAPARILITIES
Benchmarks and Baselines. General Spatial Reasoning Capabilities: We evaluated general spatial reasoning capabilities using five popular benchmarks: CVBench (Tong et al., 2024), BLINK (Fu et al., 2024), CRPE (Wang et al., 2025), SAT (Ray et al., 2024), and EmbSpatial-Bense -enchh (Du et al., 2024). These benchmarks encompass 15 subtasks measuring various spatial competencies. We included two leading closed-source models: GPT-4o and GPT-4V as performance reference. Subsequently, we conducted comparative analyses against other open-source spatial enhanced MLLMs, including LLaVA-1.5 (Liu et al., 2023a), SAT-Dynamic (Ray et al.. 2024), RoboPoint (Yuan al., 2024b), and ASMv2 (Wang et al., 2025), all with 13B parameters. Object and Free Region Reference Capabilities: We assessed embodied spatial capabilities using the RoboReflt (Lu al., 2023) and Where2Place (Yuan et al., 2024b) benchmarks. We compared mainstream closed-source models and MILM for enhancing spatial abilities (SpatialBot (Cai et al., 2024), Space VA (Chen et al., 2024), and RoboBrain (Ji et al., 2025)). We used the proportion of predicted points within specified regions as the accuracy metric. For models without point output support, we asked models to output bounding boxes of target regions, then sampled evenly within these bounding boxes. Spatial Visual Aids Capabilities: We utilized our VABench to evaluate the capabilities. We found few models with this capability for Visual Trace prediction,
also trained end-to-end prediction baseline
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison object/frree space reference benchmarks. The best results are highlighted.
Benehmark GPT-40
Saacel.LaVA LLaYA-NeXT:348
SpatialHot-33
ASMv2-138
RoboBrain-78
RoboPoint- 13B
FSD-133
RoboRell:
15.3
48.4
Where2Place
29.1
11.8
15.0
15.0
22.0
16.6
46.0
45.8
Table Performance comparison on VABench. best results are highlighted bold.
(a) VABench-Point
(b) VABench- VisualTrace
Model
Accuracy 1
Model
RMSEL
MAE]
LLM Score
GPT4o
9.30
GPT4o
136.13
113.53
4.37
ASMv2
10.07
DINOv2 Predictor
128.32
117.49
4.01
RohoPoint
19.09
RoboBrain
121.6
103.8
4.5
RoboBrain
7.00
FSD
78.26
63.44
6.21
FSD
61.82
w/o SrCoT
99.53
80.06
5.07
w/o SrCoT
26.21
w/o Alignment
80.48
66.80
5.92
w/o Alignment
55.92
model using pre-trained DINOv2 (Oquab et al., 2023) encoder coupled with transformer (Vaswani al., 2017) architecture to predict visual trajectories, trained on the same visual trajectory data, named DINOv2 Predictor. We conducted this comparison to demonstrate the advantages of our reasoning-based FSD approach. More experimental details are provided in App. E.
FSD exhibits superior general spatial reasoning capabilities. As shown in Tab. 1, FSD achieves leading average rank of 1.3 across 15 tasks from spatial benchmarks, significantly outperforming other 13B open-source models and rivaling the closed-sone source GPT-4o. The model particularly excels in 3D depth perception (88.0%), distance estimation (86.7%) and spatial relationship understanding (78.3%). These results validate our data-centric approach for building robust spatial reasoning, critical foundation for advanced embodied intelligence.
FSD excels in object reference and free space localization. The results in Tab. demonstrate FSD's ability to accurately identify objects and free spaces from language instructions. For object reference (RoboReflt). FSD achieves 56.7% accuracy, surpassing both GPT-4o (15.3%) and specialist models like RoboPoint (49.8%) by significant margins. On the more challenging free space reference task (Where2Place). FSD performs competitively with RoboPoint while substantially outperforming other models. This improvement stems from enhanced spatial understanding through our SrCoT mechanism. See App. for more visualizations.
FSD demonstrates breakthrough performance in visual aid generation. As shown in Tab. 3, FSD significantly outperforms all baselines in generating precise spatial affordances and visual traces. Specifically, FSD achieves 61.82% accuracy on VABench-P, over 3x higher than RoboPoint (19.09%) and attains significantly lower error rates with a better LLM Score on VABench-VV V. Ablation studies validate the critical contributions of both SrCoT and ceff-consistency alignment, confirming that our reasoning-bess based approach enables more accurate predictions than purely data-driven methods. A detailed comparison with RoboBrain (Ji et al., 2025) is available in App.
Visualization of visual aids generated by FSD. We visualize the visual aids generated by FSD in Fig. 5, showcasing its effective adaptation diverse scenes, perspectives, and tasks. Fig. illustrates how the CoT reasoning guides the generation process. Additional visualizations and complete output examples available in App. F.
6.2 EVALUATION OF THE DECISION CAPABILITY
SimplerEnv Evaluation. To assess the zero-shot generalization capability of FSD, we deployed this approach on the WidowX robotic arm and conducted experiments using SimplerEnv (Li et al., 2024c). Since end-to-end VLA methods require fine-tuning, for fair comparison, all end-to-end baseline methods were trained using the BridgeData Walke et al. (2023) dataset and evaluated under the same settings. We benchmarked mainstream VLAs, including end-to-end models (Octo, the Tp series, VLA), the modular-base based method MOKA (Liu et al., 2024a), and the affordance-based RoboPoint (Yuan et al., 2024b). As shown in Tab. 4, although the latest
ensured that
8

---

Published as conference paper at ICLR 2026
P Enstruction:
Description
tasks instruction performing
mage ieefrgneen platev/nefssoool
[i/booo The
monipuiated
complete
green
Reasoning fertifying <points[[60h, ]08]1=/polnts
Reosonity First, Lift 1863, 6631 663)1 s/points clear costacles.
slightly upwards
Recsonin
the on Final 2338 le/points, <boss
visual trace placing the cerrot the
plate ([oie 7081, (663.663) (607 660), (547, 491), (482, 438) (416, 365), (590, 396), [390, 41ojn/
Figure Example of FSD Reasoning Process FSD performs point-by-poimt point reasoning localization, ultimately generating feasible visual trace.
Pick up strouberry
Place egg green pot
Move egg near pot
Remove egg from plate
Picc Serww
Samovi
Rooi Fas
Moni Figo
Move cucumber between pot and boul!
Move strauberry left of brown bosket
Pick up sponge and place outside plate
towel
Figure 7: Real-world robotic manipulation tasks performed by XArm 6 robot (left) and performance comparison of FSD against baseline models GPT-4o, MOKA, and RoboPoint (right).
Table 4: SimplerEav Evaluation on WidowX Robot. Each task is tested 24 episodes. Most of the results for end- to-end VLAs are sourced from Chen et al. (2025), while the results for the remaining models are reproduced in accordance with the official code.
Put Spoon Towel
Put Carrot Plate
Stack Green Block on Yellow Block
Put Eggplant in Yellow Basket
Type
Model
Avg
Octo (Team 2024)
41.7
8.2
56.7
26.7
(Black al., 2024)
29.1
0.0
16.6
62.5
27.1
End-to-end VLA
fast (Pertsch et al., 2025)
29.1
21.9
10.8
66.6
48.3
OLA (Kim et al., 2024)
4.2
0.0
0.0
16.7
5.2
OpenVLA-OFT (Kim et al., 2025)
34.2
30.0
30.0
72.5
41.8
Modular VLA
MOKA (Liu et al., 2024a)
45.8
41.6
33.3
12.5
33.3
Affordance
RohoPoint (Yuan et al., 2024h)
16.7
20.8
8.3
25.0
17.7
VI.A
FSD
41.6
50.0
33.3
37.5
40.6
end-to-end VLAs (such wo-fast and Open VLA-OFT) perform well in this limited generalization scenario, FSD still achieves an average success rate of 40.6%, significantly outperforming standard zero-shot baselines such RoboPoint (17.7%). Without dedicated fine-tuning, end-to-end VLAs may suffer from severe performance breakdowns (with success rates approaching zero) when faced with substantial variations in backgrounds and instructions. In contrast, FSD demonstrates robust zero-shot generalization. Therefore, we suggest that an important direction for future research is to explore closed-loop policy VLAs explicitly guided by visual trajectories, order combine robust planning capabilities with precise execution abilities.
Real World Robot Evaluation As shown in Fig. 7, we conducted zero-shot tests with FSD on an xArm 6 robot across tabletop manipulation tasks. The setup included Intel RealSense L515 LiDAR camera. To test the capabilities different visual aids, we used visual trace for the sponge and folding tasks, while affordance points were used for other tasks. We compared against GPT-4o, MOKA, and RoboPoint baselines. Robopoint often makes incorrect predictions tasks involving spatial understanding. The primary cause of MOKA failures stems from the cascading errors of multiple submodules. Under zero-shot conditions, FSD achieved 72% success rate, outperforming the strongest baseline by more than 30%. Notably, FSD successfully completed complex tasks with

---

Published as a conference paper at ICLR 2026
visual trace generation, e.g. towel folding, which beyond baseline capabilities. Full results presented in Fig. 7. We refer to App. G for detailed setup and visualization.
Generated Visual Trace
Task
Demonstration
Figure 8: Qualitative results challenging manipulation scenarios. We evaluate FSD on contact-rich task (Top: Wiping the whitchoard) and an articulated object task (Bottom: Opening the drawer). The leftmost column visualizes the visual trace generated by FSD, followed by key frames of the real-world execution.
Qualitative Experiments on Challenging Tasks We conducted a preliminary exploration and attempted to assess FSD's capability in handling scenarios that require continuous contact and kinematic understanding. "opening the drawer." As shown in Fig. 8, FSD was able to infer the visual traces required for these tasks. For the contact-rich whiteboard task, the model generated a smooth wiping trajectory across the surface. For the articulated drawer task, the model successfully predicted the trajectory needed to pull the handle. Although the downstream action controller requires further adjustment and optimization for these more challenging tasks, these qualitative results verify the potential of FSD to bridge perception and complex control.
introduced two new challenging tasks: "Wiping the whiteboard" and
Real World Execution Latency We evaluated FSD's ex- ecution latency against the end-to-end VLA and mod-
Table Real-world Execution Time (s)
Model
Sponge
Cucumber
ular MOKA pipelines, with results reported Tab. 5. FSD achieves the lowest latency due to its single-pass. pass, single- model architecture that generates the entire visual trace zero-shot manner. This approach avoids both the costly
Open (FT)
23s
12s
MOKA
28s
18s
FSD
14s
10s
fine-tuning and step-by-step inference required by OpenVLA and the system overhead multi- component pipeline like MOKA. FSD thus effectively balances powerful zero-shot manipulation capabilities with practical, low real-world latency. We primarily use open-loop control, but FSD can also be integrated into hierarchical learning-base based pipeline, using its visual trace to guide and improve closed-loop control. See App. for more details.
CONCLUSION
In this paper, we introduced FSD (From Seeing to Doing), bridging visual reasoning and robotic manipulation through intermediate spatial representations. Our approach overcomes the critical chal- lenges of scarcity and heterogeneity through three key innovations: a Spatial Relationship-oocne Focused Visual Chain-of- Thought for multi-step reasoning, hierarchical weak-to-stree soak-to-strong data pipeline, and a self-consistency mechanism to align spatial coordinates with visual signals. Experiments demon- strate FSD's superior performance across multiple spatial reasoning and visual aid benchmarks. In zero-shot robotic deployment, FSD achieved an impressive 72% success rate across diverse tasks, outperforming baselines by 30%. We acknowledge limitations, such as the reliance on 2D trajectory generation and constraints from training data quality. More limitations and future works are in App.J.
ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 62533021) ) and the National Natural Science Foundation China Youth Student Basic Research Project (Grant No. 625B2128). Yifu Yuan is also supported by the Young Science and Technology Scientists Sponsorship Program by CAST Doctoral Student Special Plan. This work is also supported by the National Natural Science Foundation of China (Grant No. 92370132) and the National Key Research and Development Program China (Grant No. 2024YFE0210900) Pengyi Li is supported by the Basic Research Project (Grant No. 624B2101). We would like to thank
10

---

Published as a conference paper at ICLR 2026
Zhongwen Xu, Liang Wang, Shuyang Gu, and Han Hu for their participation in the discussions of this paper and for providing valuable insights. In addition, we would especially like to thank Yiyang Huang for the constructive suggestions improving the figures in the manuseript.
ETHICS STATEMENT
This paper is dedicated to advancing the field of robotic manipulation towards the creation of more effective and versatile robotic assistants, Our research strictly adheres to responsible practices and aligns with the ICLR Code of Ethics. All training data was sourced from large-scale, open-acces, robotics datasets, with all assets utilized full compliance with their original licensing and terms of service. We recognize that while the intended applications of this research are positive, the long-term societal impacts of increasing robotic autonomy warrant careful consideration.
REPRODUCIBILITY STATEMENT
To promote transparency and reproducibility within the scientific community, we provide detailed training parameters and resources in the appendix. The complete code for both training and in- ference has been uploaded to an gitHub repository: https: /ttt:://gthhbb com/ pi pickiiuupi/ Embod i eede We have made datasets and model checkpoints publicly available.
REFERENCES
Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet vidcos enables generalizable robot manipulation. arXiv preprint arXiv:2405.01522 2024.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi. Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A vision-language-action flow model for general robot control. arXiv preprint 1rXiv:2410.24111
2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-actinn models transfer web knowledge robotic control. arXiv preprint MXXv:23077.18/1 2023.
Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang. Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 13642, 2024.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306. 15/95, 2023.
Xinyi Chen, Yilun Chen, Yanwei Fu. Ning Gao, Jiaya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiangmiao Pang. Yu Qiao, Yang Tian, Bin Wang, Bolun Wang, Fangjing Wang, Hanqing Wang. Tai Wang, Ziqin Wang, Xueyuan Wei, Chao Wu, Shuai Yang, Jinnuu Ye, Junqiu Yu, Jia Zeng, Jingjing Zhang, Jinyu Zhang, Shi Zhang, Feng Zheng, Bowen Zhou, and Yangkun Zhu. Internvla-ml: A spatially guided vision-language-actonn framework for generalist robot policy, 2025. URL httpss:/aaee /ttp:///aavvvv//v wrgg/bb////eoooooooo /
An-Chich Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifci Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01555 01844 2024.
Cheng Chi, Zhenjia Xu. Siyuan Feng, Eric Cousineau, Yilun Du. Benjamin Burchfiel. Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal Robotics Research, pp. 02783649241173666 2023.
11

---

Published as a conference paper at ICLR 2026
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yi Ma, Pengyi Li, and Yan Zheng. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. Advances in Neural Information Processing Systems, 37:86899 -1888999--996,, 2024.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-bench: Bench- marking spatial understanding for embodied tasks with large vision-language models. arXiv preprint MxXv::2406005766, 2024.
Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-Ibillion: A large-scale benchmark general object grasping. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11444-11453 2020.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pp. 148-166. Springer, 2024.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vasion-language models with feature adapters. International Journal of Computer Vision, 132(2):581-599 2024.
Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2nd edition, 2004. ISBN 0521540518.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023.
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404.15506, 2024.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal rea- soning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024.
Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, Xinda Xue, Qinghang Su, Huaïhai Lyu, Xiaolong Zheng, Jiaming Liu, Zhongyuan Wang, and Shanghang Zhang. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. arXiv preprint 2522: 2025.
Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy Oliver Wang, Kevin Slackburn-Matzer Matthew Sticha, and David F Fouhey, Perspective fields for single image camera calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17307-17316. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini urXiv:2001.08360 2020.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.1 1831, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246. 2024.
12

---

Published as a conference paper at ICLR 2026
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-acoi models: Optimizing speed and success. arXiv preprint arXiv:2502. 19645, 2025.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, et al. Llara: Supercharging robot learning data for vision-language policy. arXiv preprint urXiv:2406.20095 2024b.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 240.:2405500941, 2024c.
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, et al. Hamster: Hierarchical action models for open-word apendworld robot manipulation. arXiv preprint arXiv:2502.05885, 2025.
Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, and Zhongyu Wei. Vocot: Unleashing visually grounded multi-step reasoning in large multi-modal models. arXiv preprint 2rXiv:2405.169) 2024d.
Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprint arXiv:2410.1864 2024a.
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689 226999, 2024b.
Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark- based visual prompting. In First Workshop Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024b. URL htttts:s//llaa/// github. ht/pic//////////// 2004
01 - 30 1220-11a0 1lava -nxt/.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprint arXiv:2402. 14245, 2024d.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint +xXiv:2003.0549 2023b.
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Xu, Hang Su, and Jun Zhu. Rdt-1b by diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864. 2024e.
Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. VI-grasp: a 6-dof interactive grasp policy for language-oriented objects cluttered indoor scenes. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 976-983. IEEE, 2023.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thoughh prompting for large multimodal models. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420-14431, 2024.
13

---

Published as a conference paper at ICLR 2026
Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6813-6823, 2021.
Fei Ni, Jianye Hao, Shiguang Wu, Longxin Kou, Yifu Yuan, Zibin Dong, Jinyi Liu, MingZhi Li, Yuzheng Zhuang, and Yan Zheng. Peria: Perceive, reason, imagine, act via holistic language and vision planning for manipulation. Advances in Neural Information Processing Systems, 37: 17541-17571, 2024.
Fei Ni, Min Zhang, Pengyi Li, Yifu Yuan, Lingfeng Zhang, Yuecheng Liu, Peilong Han, Longxin Kou, Shaojin Ma, Jinbin Qiao, et al. Embodied arena: A comprehensive, unified, and evolving evaluation platform for embodied ai. arXiv preprint arXiv:2509.15273: 2025.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406. 110000611811.. 1815, 2024.
Jake O'Neill, Abraham Arthurs, Fábio Avila Belbute-Peres, Julian Balaguer, Sarah Bechtle, Gemma Bidoia, Kyle Burden, Erwin Chang, Sheila Chen, Todor Davchev, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08800 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/990 2023.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for visionlangguage-cttonn models. arXiv preprint arXiv:2501.09747 2025.
Zengyi Qin, Kuan Fang. Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference Roboties and Automation (ICRA), pp. 7278-7285. IEEE, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 67, 2020. URL https:
pwtppsaapp/s///2 //jmlr. //5mm//.orr//ppprr///2222
074. .hmml.
Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat Spatial aptitude training for multimodal language models. arXiv preprint arXiv:2412.07755, 2024.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14152 2024.
Mohit Shridhar, Yat Long Lo, and Stephen James. Generative image as action models. arXiv preprint arXiv:2407.07875 2024.
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. arXiv preprint arXiv:2411.1653 2024.
Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, et al. Curobo: Parallelized collision-free robot motion generation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 8112-8119, IEEE, 2023.
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Xiansheng Chen, Minglan Lin, Zhiyu Li. Zhou Cao, Pengwei Wang, Enshen Zhou, et al. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.02022 2025.
14

---

Published as a conference paper at ICLR 2026
Octo Team, RT-X Team, Anthony Brohan, Noah Brown, Lauren Chen, Michael Cheng, Krzysztof Choromanski, Eamonn Cullina, Gabe Dalal, Chelsea Fu, Florian Golemo, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2403. 10164, 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: fully open, vision-centric exploration of multimodal Ilms, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems. 30, 2017.
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learming, pp. 1723-1736. PMLR, 2023.
Lirui Wang. Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive- visual learning with heterogeneous pre-train pre-trained transformers. arXiv preprint arXiv:2409.20553 2024.
Weiyun Wang. Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. In European Conference on Computer Vision, pp. 471-490.
Springer, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24888 2022.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025,, 2023.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208, 2024.
Rongtao Xu, Jian Zhang, Minghao Guo, Youpeng Wen, Haoting Yang, Min Lin, Jianzheng Huang, Zhe Li, Kaidong Zhang, Liqiong Wang, et al. AO: An affordance-awar hierarchical model for general robotic manipulation. arXiv preprint 2tti::500.126336 2025.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng. Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. arXiv preprint arXiv:2502. 13130, 2025.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any gramularity. arXiv preprint 2r1102210000770, 2023.
Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. arXiv preprint arXiv:2401. /1439, 2024a.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 10721, 2024b.
Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. arXiv preprint 2402:2440.02222. 2024c.
Yifu Yuan, Haiqin Cui, Yaoting Huang, Yibin Chen, Fei Ni, Zibin Dong, Pengyi Li, Yan Zheng, Hongyao Tang, and Jianye Hao. Embodied-rl: Reinforced embodied reasoning for general robotic manipulation. The Fourteenth International Conference on Learning Representations, 2026.
Michat Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning, arXiv preprint arXiv:2407.0800 2024.
15

---

Published as a conference paper at ICLR 2026
Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty- Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5 1455168-5199 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arema. Advances in Neural Information Processing Systems, 36:46595-46623 2023b.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprini M4Xx::24122.0345. 2024.
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejum Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint 2563502506.04308, 2025a.
Enshen Zhou, Cheng Chi, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Yi Han, Yuheng Ji, Mengzhen Liu, Pengwei Wang, et al. Robotracer: Mastering spatial trace with reasoning in vision-language models for robotics. arXiv preprint arXiv:2512.13660 2025b.
Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu. Tame wild camera: in in-the-wit monocular camera calibration. Advances Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2026
A WEAK-TO-STRONG DATASET CONSTRUCTION
Region Grounding Dataset Generation Regional grounding enables robots to focus on key objects within a scene. embodied AI, robots need concentrate on task-relevant local scenes in the image according to task instructions and be able to accurately locate objects. In Level 1 of our dataset, integrate grounding with captioning, aiming for the agent to provide positional information when describing images. Unlike traditional image captioning, objects in embodied scenes often cluttered, and we only seek to extract captions for task-relevant regions, avoiding redundant information. We prompt GPT-4o exclude task-irrelevant regions based task instructions and generate both image captions and the corresponding object names. Subsequently, we use Grounding DINO (Liu et al., 2023b) to capture the locations of various objects in the image and embed them into the caption sentences, forming captions with object location information. Below is an example from the dataset:
Level 1: Region Grounding
Give short and clear explanation the subsequent image. GPT: A <ref>robotic arm</ref><box> ( [[338, 126, 994, 861]]</box is positioned above <ref>blue bettles/ref><bort [[257, 201, 381, 413]]</box>, with <ref>grey toy</ref><box> [[911, 413, 538, 518]]</box> to the left and <ref>green oval object</ref><bo [[592, 481, 702, [01]]</box> to the right on wooden table.
Human:
Spatial Relationship Dataset Generation To accurately infer object spatial relationships from RGB images, a multi-stage pipeline is employed: encompassing object detection, instance segmentation, 2D-to-3D mapping, and subsequent relationship calculation.
Initially, for ieeeiddntified objects interest within the scene, GroundedSAM (Ren et al., 2024) is utilized to perform instance segmentation, yielding precise object masks. Subsequently, the 2D RGB information is elevated a 3D spatial representation. This transformation begins with leveraging Per- spectiveFields (Jin et al., 2023) to estimate the 7-axis orientation, serving coarse approximation for camera extrinsics. Concurrently, the WildCamera (Zhu et al., 2024) model is employed to estimate intrinsic camera parameters, including focal length and resolution. Metric3Dv2 (Hu et al., 2024) is then used for robust depth estimation. By integrating the RGB image, estimated depth image, and computed camera intrinsics and extrinsics, the 2D RGB image is transformed into 3D point cloud. Given the prior object segmentation, the specific 3D spatial position and size of each individual object be precisely derived from the generated point cloud. These comprehensive 3D data then enable the calculation of relative spatial relationships between objects, which are subsequently exported as a spatial relationship graph. We present the 2D RGB-D images before transformation and the visualized point clouds after transformation in Fig. 9.
RGB Imags
Depth Image
Point clouds after instance segmentation
Figure 9: RGB-D images before transformation and the visualized point clouds after transformation.
It is important to note that for inferring spatial relationships, only relative depth sorting data is generated, which means stringent accuracy requirements for absolute depth are relaxed. To further enhance data quality, particularly for robust relationship inference, objects exhibiting relative depth gap of at least 20% are preferentially selected for subsequent generation. An example from the dataset:
17

---

Published as a conference paper at ICLR 2026
Level 2: Spatial Relationship
Tell me about the items in your view and their relationships to each other. Answer the question with scene graph.
Human:
GPT: <ref>front left burner</ref><box> [ 422, 817, 596]]</box>> is positioned <pred>to the right
df</pred><box><1557) 422, 817, 596]]</bbx>><bxxx[[11]] 275, 446, 409]]</box> <ref>metal pots/reef><bo>>11171, 275, 446, 409]]</box>. <ref>metal pot</ret><box>1111) 275, 446, 409]]</box> appears <pred>to the left of</pred><box> [[171, 275, 446, 409]]</box><boxx<(113333 474, 715, 608]]</box> <ref>green spatoles/reff>xbox][[[4]]]
474, 715, 608]]</boxd <ref>metal pot</ref><tox>>111111 275, 446, 409]]</box> can be seen <pred>to the 1eft oft/preeo<box>(([11]
275, 446, [09]]</box><bbx>>[138) 408, 600, 490]]</box> <ref>orange cheese slice</ref><box> ([519, 408, 600, 490]]</box>>
Spatial Reasoning QA Generation After generating the spatial relation diagrams, we can easily create various template questions for spatial reasoning (Cheng et al., 2024; Du et al., 2024), such as: How are [A] and [B] positioned in relation to each other in the image? or From your perspective, which object in the image is at the shortest distance? In addition to template- based QA. we also com- bine task instructions, existing spatial information, and images to query GPT-4o, thereby generating more diverse multi-turn dialogues to enhance the model's generalization ability in spatial reasoning.
Spatial Affordance and Visual Trace Dataset Generation Next, we provide a detailed description of how to extract the required controllable points/boxes and visual trajectories from the embodied dataset like BridgeData (Walke et al., 2023), RT-X (O' Neill et al., 2023) and Droid (Khazatsky et al., 2024). Our methodology for extracting visual aids, specifically Affordance Boxes, Affordance Points, and Visual Traces, involves a multi-stage process leveraging state-of-the oftee-of-hhearrrr vision models. We also incorporate a rigorous data validation procedure to ensure high-quality output.
First, we acquire the initial and final frames of the video sequence. To determine the Affordance Box, we utilize GroundingDINO (Liu et al., 2023b) and GroundedSam (Ren et al., 2024) to detect the mask of the m ed ob ject in the final frame. The bounding box of this detected object in the final frame defines the Affordance Box, representing the ultimate spatial location of the manipulated object. Subsequently, we extract Affordance Points. This involves performing obtained from the final frame. Erosion reduces the
erosion operation
on the mask of the manipu peuulater aallaateeebeecco mask's area, facilitating the sampling of points that are more central or internal to the object, thereby mitigating the risk of sampling edge points. From this eroded mask, we uniformly sample points, which constitute the Affordance Points. For Visual Trace extraction, we begin by detecting the mask of the man i pul ated_ob ject in the first frame. From this mask, we sample points, which serve as the initial query points for the CoTracker (Karaev al., 2024) model. Selecting multiple points enhances the robustness of trajectory tracking. The processed video sequence and these sampled points are then fed into the CoTracker model, which outputs the predicted trajectory for each query point across every frame of the video sequence. We calculate the total distance of each trajectory and select the longest trajectory as the representative trajectory for the manipulater chosen trajectory is then subjected to cubic spline interpolation for smoothing. Finally, 8 equidistant points are uniformly sampled from the smoothed trajectory, forming the Visual Trace.
ob ject. This
A critical aspect of this process is addressing potential prediction errors from the pre-trained visual models, such as incorrect object identification or incomplete tracking of object motion. To mitigate these issues, we implement stringent rule-based filtering using hyperparameters like size thresholds and trajectory length thresholds. Before annotating each dataset, we iteratively adjust these rules and conduct manual inspections 100 examples. Only when the accuracy of these filtered results surpasses 95% do we deem the filtering rules robust enough to proceed with the full data generation pipeline. This meticulous validation process ensures the high quality of the generated data.
18

---

Published as a conference paper at ICLR 2026
Following the generation of these visual aids, also pre-generate the thinking processes for CoT reasoning. We input templates, questions, and answers into GPT-4o, querying it to complete the thinking process. The complete query prompt is constructed accordingly:
19

---

Published as a conference paper at ICLR 2026
Prompt and In-context Example for Completing Thinking Process
You are an visual assistant that can analyze a single image. You receive one image and correspending caption, task instruction, manipulated object in the task and target place to finish the task, and bounding box position and relative position of these objects. Also, you receive the answer of points and bbox. Now, you need to generate <Description>.. </Descriiitiiiiiionnn <Answer> <Answer>...</Ansser. format.
<Reasoning>... sraoooiiin.....//snnnnnnnnnn.
<Description> First, using the provided caption, task Instruction, describe the scene. If there errors in the caption, please ignore them and do not point them out in your description. Instead of directly mentioning the bounding box coordinates, utilize this data to explain the scene using natural language with its bounding box in the format like "<ref>object</ree>>oooo yi, x2, y2] //]]x//x>>". When mentioning the predicate between two objects, you should mention it in the format like "apredopreedccctee//preexxxxxxx" [xx1, yl, x2, y2]]
[xx1,
y3, x4, yd]]</box>", where "<box> [xl, denotes the bounding box coordinates of
yl, x2, the subject and "<box> [Ix3, y3, x4, y4]]</box>" denotes the bounding box coordinates of the object. </Description>
<Reasoning> According to the task instruction and the answer of points and bbox, provide chain-of-thoughts logical explanation of the problem. </Reasoning>
<Answer> State the final bounding box answer and point answer a clear and direct format here. Bounding box answer in the format like "<box> [x1, yi, x2, [2]]</box>" Point answer is in the format like "<point>[[<) yl], [x2, y2), </Answer>
[x3, y3l, ...]</point>".
Here is an example:
Caption: 7up can is on the table beside the bowl, drawer is open.
Task instruction: place 7up can into top drawer Manipulated object: 7up can<box> [[686, 174, 788, 3 </box> Target place: top drawer <box>[[135, 411, 852, 920]]</box> Current Relation: (7up can, [686, 174, 788, 357], top
drawer, [135, 411, 852, 920], above)
Bounding Box Answer: [569, 553, 651, 679]
Point Answer: [[603, 609], [593, 6371, [596, 643], [615, 596], [612, 593], [590, 637], [603, 606], [615, 631]]
<Description> The scene describes a <ref>7up can</ref><box> [[686, 174, 788, 357]]</box> that is positioned on a table. It is situated <prec>abbrre//pred>>oxx/ ([686, 174, 788, 357]]</boxx><box> [135, 411, 852, 920]]</box> the <ref>top drawer</ref><<ox> [135, 411, 852, 920]]</box> which is open. </Description> <Reasoning> The 7up can is above the top drawer now and should be placed into the top drawer. So the target position of the 7up can might be similar to the position of the drawer's interior. </Reasoning> <Answer> complete the task, the target position is <box> [1569, 553, 651, 679]]</box> defined by the following several free points spoint>[[60s 609], , [593, 637], [596, 643), [615, 596], [612, 593], [590, 637], [603, 606], [615, 631]]</point>, which can outline the position of the 7up can after it is placed in the top drawer. </Answer>
The new position can also be roughly
20

---

Published as a conference paper at ICLR 2026
Next, present two examples from the dataset: one involves generating Affordance Boxes/Points (Level 4), and the other involves generating visual trace (Level 5). We consider Affordance Points samples within the box; thus, generating the box first and then the points is refinement process from coarse to fine granularity. Therefore, both types are generated for each data, and one can choose which
needed. In the task of generating visual trace, since we performed equidistant interpolation in advance, the visual trace in the dataset is fixed at points. Consequently, models trained on such dataset will also generate 8 points the visual trace.
Level 4: Spatial Affordance Box/Points Generation
Human: "You are currently a robot performing robotic manipulation tasks. Your task instruction: Moved the to the left side of the green cloth. Observe the image, use 2D points and bounding box mark the target location where the manipulated object will be moved. In your answer, use <box> [x1, yl, x2, </]]</box> to present the bounding box of the target region, and use <point>[ yl], [x2, y2), ...1</toint>> mark the points of the free space."
GPT: "<Description>
The image shows <ref>white soup can</ref><box> [[544 177, 700, 3/8]]</box> positioned to the right of a <ref>green towel</ref><box> [244, 389, 664,
table.
small bowl containing a yellow object and brush is next to the towel.
</Description>
<Reasoning>
The task is to move the white soup can to the left side of the green towel. Currently, the can is to the right of the towel, so it needs to be zepositioned to the left of the towel's s current location. </Reasoning>
<Answer>
The target position for the white soup can is <box> [250, 181, 400, 392]]</box> defined by the following points: <point> ([346, 248], [302, 365], [377, 251], [330, 295], [357, 291], [354, 362], [329, 3551, , [312, 3 </Answer>"
The new position can also be roughly
21

---

Published as a conference paper at ICLR 2026
Level 5: Visual Trace Generation
Human: "You are currently a robot performing robotic
manipulation tasks. Your task instruction: put the green spatula in the silver pot. Observe the image, use 2D points mark the manipulated object-centric waypoints to guide the robot to manipulate the object. Typically, the waypoints consist of an ordered sequence of eight 2D points. The format is <point>[(xl, yll, [x2, y2], ...]</point>>"
GPT: "<Description>
The image shows a <ref>green spatula</zef><bo>> [1762, 536, 856, ]1i]]</box> positioned to the right of orange towel on a table. A <ref>silver pot</ref><box> [[0, 723, 242, 8721 </box>> is located above robotic arm. Other objects, such as a gray can and a mushroom, are also present in the scene.
</Description>
<Reasoning>
To move the green spatula into the silver pot, start by identifying the initial position of the spatula at <point> ([802, 613]]</point>, to clear the table, moving to <point>[1774, 501]]</point>, Then, move it horizontally towards the left, passing through <point> [[685, 394]]</point>, Finally, lower the spatula into the silver pot, ending at <point>[165] 401]]</point> within the target bounding box <box>[1225, 529, 407, 640]]</box>> </Reasoning>
First, lift the spatula upwards
<Answer>
The visual trace for placing the green spatula into the silver pot is <point>[[802, 613], [780, 582], [774, 501], [744, 465], [685, 394], [657, 349), [668, 354], [657, 401]]</points
</Answer>"
B TRAINING DETAILS AND DATASETS
The training of FSD is two-stage process, building upon the LLaVA (Liu et al., 2024b) architecture through continued fine-tuning ASMv2 (Wang et al., 2025). In order to train a VLM with powerful spatiotemporal reasoning abilities that also generate visual aids, we leveraged comprehensive dataset of approximately 1.4M samples from various sources. Note: The position coordinate format used by FSD normalizes the image coordinates a range 0-999 after padding the image to a square shape. For all datasets mentioned below, the coordinates have been pre-processed in this manner.
Stage 1: Spatial Reasoning Enhancement The primary goal of the first stage is enhance the model's iss spatial reasoning capabilities. This dataset is primarily composed of two parts: common- sense image QA and conversations data and spatial reasoning data. The inclusion of GeneralQA is crucial for FSD to maintain broad instructionffolloviii selected some datasets and randomly sampled around 838k data from L.LVVA-65k and ASMv2-4M instruction- contuuttion-folloiing data. Next, incorporated approximately 295k samples of spatial reasoning data from the LLa VA-One VA-O Vision (Li al., 2024a), RoboPoint (Yuan et al., 2024b), SpatialBot (Cai al., 2024) and SAT (Ray al., 2024) training datasets. Finally, the first three levels of FSD's collected embodied spatial reasoning datasets (250k samples) were also integrated into this training phase. A summary of the datasets used can be found in Tab. 6.
abilities after fine-tuning. First, we
Stage 2: Visual Aids Generation and Understanding We specifically focused on training for visual aids generation. Here, "generation" refers to generating visual aids based given question, while
22

---

Published as a conference paper at ICLR 2026
Table 6: Details of the training data for FSD
Stage
Task
Datasets
Samples
Data Sources
GeneralQA (Caption,
Share VQAv2,
838k
LLaVA (Liu et al., 2024b), ASMv2 (Wang et al., 2025)
VQA, OCR, RegionVQA. Conversation, Grounding,
OCR-VQA, Visual7W, ST-VQA, RefCOCO/+/g.
Text)
VG, AS-Core, AS-V2, Te Visual7W
Stage
Spatial Reason- ing
General
KITTI, 2D3DS, Objec- 295k (Ref, RegionRef, VSR, CLEVR. CLEVR-Math, SUPER-CLEVR, RAVEN
SpatialBot (Cai et al., 2024), RoboPoint (Yuan et al.. 2024b), SAT (Ray al., 2024). LLaVA- One (Li al,, 2024a)
Embodied Spatial Reason- ing
FSD Level 1 (145k), Level (86k) and Level (19k)
250k
FSD
Spatial Affordance Gener-
FSD Level (24k)
24k
FSD
ation Understanding
Stage
Visual Trace Generation & Understanding
FSD Level (26k)
FSD
"understanding" represents its inverse problem. We excluded some inverse problems with ambiguous semantics. The final total amount of data used is shown in Tab.
Training Configuration We used exactly the same hyperparameters in both stages of training. We utilized global batch size of 128 and the AdamW optimizer, configured with 31 0.9, 33 0.999, and a weight decay coefficient of 0. The learning rate was set to X 10 Both training stages employ a linear warmup for the first 3% of training steps, followed by cosine decay strategy to minimum learning rate of O. We simultancously train both the vision-language connector and the language model. The image resolution was set to 336 x 336, and the visual encoder remained frozen throughout the entire training process. In the first phase, we train for epoch on the complete dataset, while in the second phase, train for 3 epochs. The FSD model has approximately 13B trainable parameters and we conducted training using 8 A100 40G GPUs, with Stage requiring approximately 72 hours and Stage requiring hours.
C DETAILS OF ACTION EXECUTION
When utilizing FSD for robotic manipulation tasks, can select from various visual aids. As described in Fig. 2, these include spatial affordance boxes (B), spatial affordance points (P), and object-centric visual traces (T). The choice of visual aids dictates the subsequent motion planning strategy.
Motion Planning with Spatial Affordances For spatial affordance boxes (B), the target point for manipulation is derived by sampling the center of the box. In the case of spatial affordance points (P), point is directly sampled. When relying spatial affordance information, whether from boxes or points, we employ CuRobo (Sundaralingam et al., 2023) as our motion planner. CuRobo responsible for generating collision-free paths that guide the robot's end-effector to the inferred target affordance point.
Motion Planning with Object-Centric Visual Traces When leveraging object-centric visual traces (T), the process involves mapping 2D visual traces into 3D space and then interpolating these discrete points form a complete motion trajectory SE(3) space. The detailed procedure as follows:
We directly acquire 2D keypoint information, denoted as k; (u, uy) R² where u, and v; represent the x and coordinates of the i-th point in the image, for i E [1,T]. Initial depth information, d, E R, is obtained from a depth camera. Using the Pinhole camera model (Hartley & Zisserman, 2004), we can transform these 2D keypoints into 3D Cartesian coordinates P. (w4, V4, 24) E R via:
23

---

Published as a conference paper at ICLR 2026
14 8/
far
0
Cx
0/
=
0
Cy
yi
1
Z
Here, 8; is the normalized depth, calculated as 8; = d./depth_scale The intrinsic camera parameters, Sax ly: Cz. Cy. and the depth scaling factor, depth_scale, are all camera-specific.
However, naive use of the raw depth values d; obtained from the depth camera often results in trajectories that closely hug the object's surface, which is undesirable for robust robot manipulation. To address this, we formulate an optimization problem to address it. We fix the depth values for the start and end points of the path (dl and dr) and optimize the intermediate depth values d2:-11 to minimize the total Euclidean distance between consecutive points in Cartesian space.
The objective function for this optimization is:
d; = arg min Σ Pii) 22:7-1
where d(P,, P.+1) represents the Euclidean distance between points P. and P.+1- We employ gradient descent method from scipy library to optimize this objective function. This refined approach allows for more robust and practical robot motion planning by addressing the limitations of raw depth data and providing a structured framework for integrating various visual aids.
D DETAILS OF VISUAL AIDS GENERATION BENCHMARK
To address the gap in evaluating visual assistance signal generation, we established VABench, comprehensive benchmark. Benehhh comprises two distinct tasks: VABench-Point and VA Bench- VisualTrace, each featuring 300 meticulously hand-annotated questions. These tasks require models to infer visual auxiliary information solely from natural language instructions, mimicking everyday human commands.
For VABench-Point, we provide ground truth bounding boxes for each question. The model's performance is then assessed by calculating the proportion of predicted points that fall within the target region. For models that only output bounding boxes, we explored two scoring methods: Intersection Over Union (IOU) and uniformly sampling points within the predicted box. We ultimately opted for the latter approach, as the physical interaction between robotic arm and an object in real- world tasks is fundamentally determined by specific point coordinates. For Bench- VABBench-VisualTraaee we provide ground truth trajectories consisting of eight points. When the predicted trajectory length deviates from the ground truth, we employ interpolation to align their lengths. To ensure consistent uniformly normalized to range of
evaluation across varying image resolutions, all coordinates 0 to 1000. Subsequently, we employ a combination of Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and GPT Score comprehensively evaluate performance.
Here, we detail the design philosophy and evaluation procedure of the GPT Score. Given that multiple valid solutions can exist for each task instruction, relying solely on trajectory similarity to the ground truth as a scoring criterion can be one-sided. Therefore, we established detailed evaluation criteria simulate realistic human assessment. Based on these rules, we introduced visualized trajectory scoring method leveraging Multimodal Large Language Models (MLLMs), termed GPT-score.
Specifically, we designed an evaluation prompt guide GPT-4.1 in assessing predicted object manipulation trajectories based on both task instructions and visual inputs. The prompt provides clear instructions and criteria, positioning the model as an expert evaluator in robotic manipulation and visual reasoning. Each evaluation instance consists of a task instruction and an accompanying image that visualizes the predicted trajectory, where a red circle indicates the start point and a blue diamond marks the end point. The model is instructed to assess the trajectory according to three key criteria: (1) task alignment and success, determining whether the predicted path correctly fulfills the instruction by starting and ending in appropriate locations; (2) feasibility, evaluating the physical plausibility and smoothness of the motion; and (3) obstacle avoidance, considering whether the trajectory avoids potential collisions. The prompt emphasizes that completing the task correctly is the most important
24

---

Published as a conference paper at ICLR 2026
factor; any major deviation in goal achievement results in low score, even if the trajectory appears smooth or feasible. Then, the model returns a structured response consisting of a numerical score from to 10, along with concise explanation. Scores are interpreted based on task success and quality: low scores (1-4) indicate failure to accomplish the task, mid-range scores (6-8) reflect successful but imperfect trajectories, and high scores (9-10) are reserved for trajectories that both accurate and high-quality. This scoring scheme allows for nuanced, human-like evaluation that integrates both semantic understanding and visual reasoning. By leveraging this multimodal prompt framework, GPT-score enables robust and interpretable evaluation process that aligns closely with human judgment, overcoming the limitations of purely geometric or distance-basee metrics. The complete prompt is presented as follows:
25

---

Published as a conference paper at ICLR 2026
Complete Prompt for GPT Score Metries
You are an expert evaluator in robotic manipulation and visual reasoning. Your job to assess the quality of predicted trajectories based on task instructions and visual inputs.
You are given:
A task instruction describing an object manipulation task.
An image showing predicted trajectory.
**Note:**
In the image, the red circle indicates the start point, and the blue diamond indicates the end point.
'The trajectory represents the predicted movement path of the manipulated object, not the robot end-effector.
You should *vaaluaee the predicted trajectory as proposed motion for the object that is supposed be =oved** based on the task instruction **n based on the static positions of objects in the image*s, The objects have not actually moved.
**Evaluation Criteria (listed in order of importance):**
**Task Alignment and Success (most important)*** Does the trajectory clearly and accurately fulfill the task instruction? **The trajectory must start the correct location and end at a target position that aligns with the task goal. Large deviations in the starting or ending point (e.g., wrong object, wrong destination, stopping short of the goal) should result in low score, even if the rest of the trajectory is smooth. If the task is not accomplished (due incorrect goal interpretation or spatial execution), the score should be low regardless of other qualities.
# Feasibiiltyy** Is the movement physically plausible, smooth, and continuous? Are there any unrealistic discontinuities, sharp turns, or impossible transitions? Even if the movement is feasible, should not receive a high score if the task is not completed.
3. **Obstacle Avoidance / Safety** Does the trajectory reasonably avoid collisions with surrounding objects? Minor risks may be tolerated the task completed successfully, but major or clear collisions should reduce the score.
**Scoring Guideline:**
If the task **not accomplished** or if the start or end point is significantly incorrect, the score should typically be **4 below**.
If the task is completed but the trajectory has issues (c.g., roughness, minor risk of collision), score the #*6-8** range appropriate.
A score of 9-10** should be given only when the trajectory clearly completes the task, with good start/end accuracy, smooth motion, and reasonable safety.
Based on these criteria, provide single overall score from (very poor) to (excellent), reflecting how well the task is accomplished.
Respond strictly in the following format: Score: <1-10>
Explanation: <brief justification>
The task instruction is: (task jnstrrintttction)
Please give your response.
E DETAILS OF BENCHMARKS AND BASELINES
For general spatial reasoning tasks, the answers typically multiple-choie questions with clear options. However, some spatial reasoning models show reduced eusruction-followiii ability after fine-tuning, preventing them from directly outputting the correct option. To address this, we
26

---

Published as a conference paper at ICLR 2026
co1e car
"'ádle araner
You are currently a robot performing rebotic manipulation tosks. The task instruction is: place coke can into drawer. Use 20 points to mark the manipulated object-centrie waypoints the task.
the robof to successfully complete
Image Input
Generation
<Description>
The image shows <refscoke canc/ref==cbor<I444 309, 550, //1r]/ placed top of cobinet. The cabinet hos an open <refamiddle drowerc//refsctoor[[25 427, 798, 898]]</boxx.
</Description>
<Reasonings
move cake can into the middle drawer, start by identifying the initial position the coke can
<point>([493, 343]]</point>. First, lift the coke slightly upwards clear the edge of the cabinet, moving to <point=][490, po2]]</point>.Net) lower the coke con towards the middle drawer, passing through <point>[[990, 3961]</point> and <polnt>[[490, 465]]<point>> Continue lowering it into the drawer, reaching <point>[[498, 533]]</pointz. Finally, place the cake con in the middle drower at the final position <paint>[[524, 705]]</point>.
<boxx>[[77,, 674, 741]]</box>, ending the
</Reasoning>
<Answer>
The visual trace for placing the coke into the middle drawer <point>[[495, 3431, (490, 3021, [490. 3961. [490, 4651, (493, 583), [505, 6021. [515, 6771, [524, 7/pont/
</Answer>
Figure 10: example visualization of generating visual trace.
lenient matching rule, considering an answer correct if it includes either the correct content or the corresponding option.
For object/region reference tasks, carefully fine-tuned and used a tailored prompt for each model. Most models, such as GPT-4o and ASMV2 (Wang et al., 2025), cannot directly output specific points. Similar to the validation process for RoboPoint (Yuan al., 2024b), we also found that using in-context learning to specify point output formats resulted in worse performance compared to directly outputting bounding boxes. Therefore, for these models, we instructed them to output bounding boxes directly. From these bounding boxes, either uniformly sampled nine points or took the midpoint. Then, we calculated the proportion points within the specified region to determine the final average accuracy.
For the VA Bench- VABeenh--iisuaTrrre task, due to a lack of strong baselines, we developed an additional Transformer-base based prediction model, trained from scratch using the same data, which named DINOv2 Predictor. the DINOv2 Predictor, the visual encoder uses a pre-trained DINOv2 (Oquab al., 2023), encoding images output (196, 768) feature vector. The language encoder uses pre-trained TS-Base (Raffel et al., 2020), outputting (32, 768). These are concatenated with learnable embeddings (8, 768) and passed through Transformer encoder base. The output the learnable embeddings is then read and passed through linear layer to predict eight points. During training, we keep the language encoder fully frozen and train the visual encoder along with the other remaining parameters.
F MORE VISUALIZATIONS AND EXAMPLES
The visualization of the output of FSD is shown in Fig. 10 and Fig. 11. Besides, we present the prediction results of FSD on Where2place (Yuan et al., 2024b), Roborefit (Lu et al., 2023), and VABench in Fig. 12, Fig. 13, and Fig. 14, respectively.
G REAL WORLD EXPERIMENT RESULTS
In our real-world desktop manipulation tasks, used xArm 6 robotic arm for evaluation. This setup included an Intel RealSense L515 LiDAR camera and force-torque sensor on the xArm to enable compliance control, which improved interaction with the environment. A computer running Ubuntu 24.04 and equipped with an NVIDIA GTX 1660 was directly connected to the arm and camera to execute low-level control policies. Notably, single RealSense 1.515 depth camera was
27

---

Published as a conference paper at ICLR 2026
You are currently robot performing robofic montoulation fasks." Your fask instruction: Place spoon the towsll Obserue the image, bounding target location. your enswer, use <boxs![ja,, y1, x2, ///oo/ present the bounding bex target region and use spoints|[kd, [Id, gt). (x2, 42) Je/point>> mark the points of the free space.
Image
Bbox Generation
Point Generation
:Descriptions
shous crefored spanchhefsccoro(((777 833, 751, jc/oor/ placed to crefzarange (ruelc/raaactbooxx(16.
329, 384, 488]]</boxo. middle,
yellow chicken leg.
</Descriptinn
<Reusoning>
speon <paint><[488, 576n]</point> top green <box>>11133, 329, 4/8x)//box. The current position of the spoon above the fowel, so it needs to be moved to overlop with the towel's position choxrs[]665 529,
384, 488])</boxs.
</Reasonings>
<Answer>
To complete the task, the torget pasition for the red spoon is <box>](16,, 371, 418, 435]j</borc The new position con diso be
roughly defined the following several free points spoint>[201,, 4021, [204, 4021, [204, 404], [202, 4021, [204, 4051, [206, 405), 404], [204, 40sits/points,, which the position spoon placed orange
</Answer>
Figure 11: An example and visualization FSD for generating affordance box and points.
Locate
right
Find paints
TAC Find seew poines VROUE vaide DIDNE
THBC FINS BORK DIICEE WYTH VaCHRESpecs TO the FigEt The COTE FERTON
NOW
Figure 12: Visualization of visual aids generated by FSD in the Where2Place benchmark.
sufficient for task completion, especially when performing visual trace execution. This approach eliminated the need for object segmentation and 3D mapping; instead, directly mapped 2D visual trajectories to 3D for execution, with no strict requirements on depth information accuracy. Demonstrations available in Fig. 15 and website.
28

---

Published as a conference paper at ICLR 2026
Testr Soed grece Erran mITH bOrtA
Teaks pen white cancelt byntee bocter clanneer
Tesk wd plecer pestree storic teaihoru
Tenh: lueat grea othibhe lage
Figure 13: Visualization of visual aids generated by FSD in the RoboRefit benchmark.
H PROMPT FOR USING FSD MODEL
Generate Spatial Affordance Points & Bounding Box
You are currently robot performing robotic manipulation tasks. Your task instruction: (Task Instruction] Observe the image, use 2D points and bounding box to mark the target location where the manipulated object will be moved. In your answer, use <box>[[x11 yl. x2, y2]]<<box>> to present the bounding box of the target region, and use <point>[[x1, yl], [x2, y2]. //.//nttnt>> to mark the points of the free space.
Generate Visual Trace
You are currently robot performing robotic manipulation tasks. Your task instruction: { (Task Inst ruction], Observe the image, use 2D points to mark the manipulated object-centric waypoints to guide the robot to manipulate the object.Typically the waypoints consists of an ordered sequence of eight 2D points. The format is <point>[[x1, yl], [x2, y2], ... 1/poonts>
I COMPARISON OF FSD AND ROBOBRAIN
Both FSD and RoboBrain (Ji et al., 2025; Team et al., 2025) have the capability to generate visual trace. However, RoboBrain tends to produce agent-centric visual trace, whereas FSD generates object- centric visual trace. FSD adopts task-centric design principle, allowing it to perform effectively even in more heterogeneous ontological scenarios, including those that completely lack robotic arms in the image, thus exhibiting stronger generalizability. Due to the differences in the methods of generating visual trace, we conducted several sets visual trajectory visualizations for qualitative analysis, shown in Fig. 16. Under the same zero-shot setting, the visual trace generated by FSD has higher accuracy compared to those generated by RoboBrain, confirming that FSD's 's reasoning- based pipeline possesses greater generalizability when facing unknown tasks.
29

---

Published as a conference paper at ICLR 2026
Visual Trace
Affardance Point
Fos apoica c0054
pro the ctonge
Affordance Bounding Box
Figure 14: Visualization visual aids generated by FSD in the VABench benchmark. FSD generate three types visual aids based on task instructions for novel tasks scenarios. 1st-2nd row: visual trace; 3rd-4th row: affordance points; 5th-6th rows: affordance bounding box.
J FUTURE WORKS
We have made preliminary attempts to use visual aids intermediate states in FSD, achieving promising results in object/target/reginn reference and actual manipulation task execution. Future work can focus on the following aspects to further enhance the applicability of this paradigm: 1. Task Decomposition for Complex and Long-Horizon Instructions: The current version of FSD primarily targets clear and explicit language instructions. When dealing with long- hong-horizon tasks or anbiguous/complex instructions, the model needs to decompose them into atomic, executable sub-tasks. We believe that decomposing instructions into sequence of visual aids to guide each sub- task execution is promising avenue. 2.Downstream Execution and Visual-Aid Ainll-Aid-Guieed Control: Currently, FSD relies mainly on training-free motion planning methods for downstream execution. In extremely complex or dynamic scenarios, this may lead to a bottleneck in success rates. A potential improvement to use the generated visual aids as explicit guidance for downstream VLA models. replacing language- conditione training. Several preliminary studies (Zheng et al., 2024; Bharadhwaj al., 2024; Li al., 2025) have shown that, for robotic manipulation tasks, affordance and visual trajectories provide more effective guidance than language prompts. 3. Extending from 2D to 3D Visual Aids: Zhou et al. (2025b) At present, FSD focuses on predicting 2D visual aids, similar to the representation used in Referring Expression Comprehension (REC) and related tasks, which leverages the general visual understanding and reasoning capabilities VLMs. However, as task and scene complexity increase, predicting 3D visual traces may prove to be more effective solution, and we identify this as an important direction for future research. 4. Inference Efficiency of FSD: While FSD demonstrates acceptable efficiency for open-loop execution, the inference latency associated
30

---

Published as a conference paper at ICLR 2026
Pick up struwberry
More cucumber between and bowd
Remove egg from plate
Place in
Fold the towell from right left
Pick up spouge and place outside plate
Part Spoon on Towel
Stack Green Block Yellow Block
Put Fggplaut in Yellow Basket
Figure 15: Visualization of FSD executing tasks. The first six are real-world experiments, and the last four are from SIMPLEREDv.
with its reasoning capabilities can become a bottleneck in dynamic environments, necessitating frequent re-pla re-planning. Consequently, optimizing inference speed to enable more responsive closed- loop adaptation remains key priority for future work. 5. Tisual-Aid-Mediatee Human Feedback and Alignment: Traditional alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), often rely language-based evaluations Yuan et al. (2024c); Liu et al. (2024d), which can be spatially imprecise and ambiguous for complex manipulation tasks. We envision using visual aids transparent and interactive interface for human intervention. This visual-aid- aid-mediated feedback loop could significantly improve the sample efficiency of alignment and ensure that the agent's 's execution is more consistent with human intent and safety constraints.
31

---

Published as a conference paper at ICLR 2026
place chip bag into white bowl
RoboBrain
RoboBrai
ploce blue chip bag into white
bowl
Figure 16: Comparison of generated visual traces between FSD and RoboBrain.
K INTEGRATION WITH LEARNING-BASER PIPELINE
Experimental Setup. To evaluate FSD as a complementary component for learning-based methods, conducted experiments 10 tasks from the LIBERO-spatial benchmark based on CleanDiffuser codebase Dong et al. (2024), each with 50 demonstrations. Our baseline is a Diffusion Policy (Chi al., 2023) using T5-base for text encoding. The policy is conditioned solely the current RGB image to predict action chunks of length 16. For FSD integration, we adopt the ATM (Wen et al., 2023) approach: visual trace generated from the initial frame and task instruction, and we then train a trajectory-guided policy that leverages this visual guidance.
Results and Analysis. The results in Tab. show that FSD improves the average success rate by nearly 10% over the baseline. We believe this is because the visual trace provides the policy with richer information by clearly indicating the object's movement path, thereby enhancing ability to distinguish between different task instructions.
Table 7: Performance the LIBERO-spatial benchmark. FSD provides significant improvement when integrated with Diffusion Policy (DP) baseline.
Tasks
DP
DP + FSD
Libero Spatial (10 Tasks Avg.)
76.1 + 1.6
85.3 1.2
L SUPPLEMENTARY EXPERIMENTS
L.1 ABLATION ON HIERARCHY LEVELS
We designed the 5-level capability hierarchy logical weak-to-strong curriculum, where Levels 1-3 (Spatial Understanding) serve as prerequisites for Levels 4-5 (Visual Aid Generation). To validate the effectiveness of two-stage learning, conducted comparative experiment called "FSD w/o Stage 1". In this experiment, we skipped the foundational training (Levels 1-3) and instead trained the model directly visual aid data (Levels 4-5). All other experimental settings remain unchanged compared to the FSD (Full).
The results are presented in Tab. 8. The results show that the FSD significantly outperforms the FSD w/o Stage variant. The first stage provides essential spatial understanding and coordinate anchoring abilities. These essential for the reasoning process and the generation of visual aids.
32

---

61.8 Published as a conference paper at ICLR 2026
Table 8: Ablation study the impact of Stage I training. We compare the full FSD model against variant trained without foundational spatial understanding stage (Levels 1-3).
Model
Where2Place
VABench-Point
FSD (Full)
45.8
FSD w/o Stage 1
33,2
42.3
L.2 ABLATION ON TRAJECTORY POINT DENSITY
In this section, we ablated the number of generated trajectory points. We fixed the model trained on Stage data as the base model, and then adjusted the training data for Stage using resampling to support the generation of 4. 8, and 16 points, respectively. We present the ablation results on VABench-V.
The comparative results are detailed in Tab. 9. The results indicate that using points yields the lowest score due the loss prediction accuracy, while the prediction performance with 16 points slightly degrades and is inferior to that with 8 points. This may be attributed to the fact that greater number of predicted points increases the difficulty both instruction following and reasoning processes. We argue that more complex tasks require predicting additional points to characterize more intricate visual trajectories; however, for most tasks, 8 points can provide sufficient granularity to generate smooth trajectories, achieving a balance between performance and speed. Therefore, for the majority of manipulation tasks, we adopt 8 points as a design choice to balance granularity and token complexity.
Table 9: Ablation results on VABench-V with different trajectory point densities. We compare the performance metrics for trajectories generated with 4, 8, and 16 points.
Method
RMSE
MAE
GPT Score 1
FSD-4p
102.1
82.6
5.0
FSD-8p (default)
78.3
63.4
6.2
FSD-16p
85.2
67.1
5.7
L.3 MORE COMPARISON REAL- WORLD TASKS
To further evaluate the robustness of our approach, we conducted additional comparative experiments with Open VLA on two challenging real-world tasks using the xArm6 platform. Specifically, for VLA, report results after fine-tuning (FT) with 20 demonstrations per task, its zero- shot success rate was 0%. In contrast, both MOKA and FSD were evaluated under zero-shot (ZS) conditions. The success rates for each method summarized in the Tab. 10.
Table 10: Comparison of success rates real-world xArmó tasks. OLe was fine-tuned (FT) with 20 demonstrations, while MOKA and FSD evaluated in zero-shot (ZS) setting.
Task
OpenVLA (FT)
MOKA (ZS)
FSD (ZS)
Pick up sponge and place outside plate Move cucumber between pot and bowl
100%
40%
80%
40%
60%
80 %
Notably, even after fine-tuning, OpenVLA (FT) performed poorly the second task (40%), indicating inadequate generalization to positional variations within the test distribution. In contrast, FSD demonstrated consistent and robust performance (80%) across both tasks under zero-shot conditions. These results suggest that, compared to end-to-end methods, FSD's paradigm of reasoning driven SrCoT, combined with intermediate visual aids, offers significant advantages both generalization capability and execution accuracy.
33

---

Published as a conference paper at ICLR 2026
L.4 VISUALIZATION OF COLF-CONSISTENCY ALIGNMENT
To provide a more intuitive understanding of the impact of alignment, we present a qualitative case study in Fig. 17. As illustrated, the model without alignment (grey path) generates correct textual reasoning plan but fails to map it to accurate coordinates, leading to "spatial hallucination." " In contrast, the model with alignment (colored path) successfully anchors its reasoning the correct pixel coordinates, ensuring precise task execution.
Instruction:
block
w/o Alignment: <Reasening> To place the blue block an top of the red block, start by identrlying the current postiion of the blue block at point>([460 487]]- st//it/pointt- Lift the blue block upwards to dear any abstadles ...x/ </Ressningg>
-Reaoonnng-
the red black, deetifying the postion the blue black spoint- 40tjj=/pcint> Lift the blue block upwards to clarr block, moving to </Reasoning-
w/ Alignment
wio Alignment
Figure 17: Visual comparison demonstrating the effectiveness of Seff-Consistenyy Alignment.
worth noting that without self-consistent alignment, the model's textual reasoning process is logically correct (for example, it can accurately describe the target position relative to the red block). However, fails to ground this reasoning spatially. The generated output coordinates deviate from the described location, resulting in trajectory that does not align with physical reality. This indicates that without aligmment, the model treats coordinates merely abstract linguistic symbols, rather than mapping them to actual visual features, thereby leading to spatial hallucinations. In contrast, proposed bidirectional training mechanism with self-consistency alignment enforces a strict correspondence between spatial coordinates and visual features. This compels the model to focus on specific pixels corresponding to the generated coordinates, ensuring that the generated points accurately land the intended semantic targets and mitigating coordinate drift.
M USE OF LLMs
We utilized Large Language Model (LLM), specifically, as writing assistant during the preparation of this manuscript. The use of the LLM was strictly limited to language enhancement tasks, such improving grammar, rephrasing sentences for better clarity, and ensuring stylistic consistency. The LLM was not used for generating scientific ideas, experimental results, or conclusions. All authors have carefully reviewed and revised the LLM's suggestions and take full responsibility for the scientific integrity and content of this work.
34


Reviews:
[{"review_id": "v0JfmOb0pV", "reviewer": "Reviewer_aF4E", "summary": "This work presented FSD (From Seeing to Doing), a framework that connects visual reasoning with robotic manipulation using intermediate spatial representations. FSD tackles key challenges of data scarcity and heterogeneity through three core innovations: a Spatial Relationship-Focused Visual Chain-of-Thought for multi-step reasoning, a hierarchical weak-to-strong data pipeline, and a self-consistency mechanism that aligns spatial coordinates with visual signals. Experiments demonstrate FSD’s effectivenss across multiple spatial reasoning and visual aid benchmarks.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear, by identifying limitations of existing VLA approaches, i.e., data scarcity, and heterogeneity, and a principled solution is proposed through object-centric intermediate representations of visual aids.\n\n2. Extensive experiments across 8 benchmarks for spatial reasoning, plus zero-shot manipulation in both simulation and real-world settings demonstrate thorough validation.", "weaknesses": "1. The approach generates 2D trajectories that must be lifted to 3D using depth cameras. This is a significant limitation acknowledged by authors but I still have some questions:  \nA.\tThe depth model needs to predict the depth of 2D trace, which is tricky in my understanding, as this kind of trace could be out of distribution for the depth model. How robust / accurate is the depth estimation?\nB.\tHow does a 2D trace avoid the collilsion, as there is uncertainty when it’s unprojected to 3D? \n\n2. Ablation study of parameters: No ablation on the number of hierarchy levels. No comparison of different VLM backbones. No study on how performance changes with trajectory point density\n3. The model is built based on HVicuna-13B and CLIP-ViT-L, how about the performance of a naïve VLA model implementation, without visual aid? This can give us an insight about the improvement with visual aid.\n4. It uses GPT-4o to generate reasoning chains for training, then evaluates using GPT-4o for scoring (VABench GPT-score), which potentially leads the model to overfit to GPT-4o's biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMrE2tuMjC", "reviewer": "Reviewer_ezj7", "summary": "This paper proposes FSD, a vision-language model that provides intermediate visual representations for robotic manipulation, such as affordance regions and visual traces. It presents detailed information on how to construct the dataset and train the model. Extensive experiments demonstrate the strong performance of FSD across multiple benchmarks, as well as its zero-shot open-loop capabilities. The model, along with its detailed data generation pipeline, dataset, and training strategy, offers valuable contributions for general robotic manipulation.", "questions": "1. In Table 3b, the DINOv2 predictor is compared. How many parameters of this model? Is it the same model size as FSD?\n3. In Table 3, the results show that self-consistency alignment helps improve performance. Do you have any failure cases without alignment and success cases with alignment, so that we can clearly see the effectiveness of the self-consistency alignment?\n4. How long does it take to generate the dataset?\n5. Table 5 shows the execution time. Is this time for the whole task execution? What is the FSD inference time? OpenVLA is the closed loop model, which call the model constantly. What is the time cost for each component in your pipeline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides a large dataset with visual aids, which is essential for robotic manipulation tasks. The data generation pipeline is detailed and well-designed. The experiments are extensive, and the model demonstrates strong performance.\n2. The paper proposes weak-to-strong training and a self-consistency alignment strategy, both of which are effective for model performance.\n3. The evaluations on real-world manipulation tasks and simulations are thorough, and the performance is strong. Both open-loop and closed-loop settings are tested.", "weaknesses": "1. Inference time may become a bottleneck for real-world applications.\n2. The open-loop inference pipeline relies on complex and specifically designed submodules such as GraspNet and depth estimation. The overall success rate is influenced by compounded errors from both FSD and the downstream modules. \n3. This paper presents promising results on closed-loop robotic manipulation in Appendix K, where visual aids are generated by FSD at the beginning of the task. It would be beneficial to update these visual aids every few steps to make the entire pipeline more practical and effective in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9zu1fHuVv6", "reviewer": "Reviewer_vmwj", "summary": "The paper proposes FSD, a VLM-based framework designed to generate visual aids for robotic tasks. These aids—spatial affordance boxes/points and object-centric visual traces—are produced via a Spatial-relationship focused Chain-of-Thought (SrCoT) and a self-consistency alignment that binds coordinates to visual references. The model is trained using a hierarchical, weak-to-strong data construction pipeline, creating 300k SFT samples from multiple robot datasets to teach five distinct capability levels (grounding → spatial relations → spatial reasoning → affordances → traces). Experiments report strong results on five spatial-reasoning benchmarks, good object/free-space referencing, high accuracy on the new VABench for visual aids, and successful zero-shot manipulation in SimplerEnv and on an xArm across eight real-world tasks.", "questions": "Cost Attribution and Scaling Analysis: Provide a detailed table with parameters, FLOPs, and p50/p90/p99 latencies per module (ViT, projector, SrCoT, aid decoding, planning). This analysis should be benchmarked on the same hardware used for comparisons (eg, vs. OpenVLA/MOKA) and show throughput vs. image resolution and token budget.\n\nGeometric Robustness Stress Test: Conduct simulation experiments that quantify the drop in task success rate against applied noise. This must include sensitivity analysis for: (i) camera intrinsic perturbations (±Δf, ±Δcx,cy), (ii) synthetic depth noise, and (iii) mis-calibration. Includes a qualitative gallery of failure cases where 2D traces misproject.\n\nData Quality Audit: Report spot-check precision/recall of the auto-generated grounding and relation labels against a manual ground truth. For VABench, report the inter-annotator agreement (eg, Cohen's Kappa or Fleiss' Kappa) to validate the 300 manual items. Release data filtering scripts for reproducibility.\n\nClosed-Loop and Granularity Ablations:\n\nImplement a simple closed-loop variant (eg, visual servoing or re-planning at the next waypoint) and compare its success and correction frequency against the open-loop baseline.\n\nJustify the choice of T=8 waypoints by ablating performance (VABench and SimplerEnv) vs. T ∈ {4, 16} and an adaptive T.\n\nHuman-in-the-Loop Evaluation: Strengthen the VABench claims by complementing the LLM-score proxy with a human preference study (eg, ≥300 pairwise prompts) on visual trace plausibility, reporting 95% confidence intervals.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Clear Problem Decomposition: The paper's primary strength is its clean decomposition of the problem: reason first, then produces compact, embodiment-agnostic visual aids (boxes, points, traces). This abstraction (§3.1) translates well to zero-shot action by simplifying the downstream tasks of 2D→3D back-projection and motion planning.\n\nMethodological Novelty: The Spatial-relationship focused Chain-of-Thought (SrCoT) is a novel and promising approach. By anchoring reasoning on spatial relationship graphs with strict object and coordinate bindings (<ref>, <box>, <point>), the method plausibly reduces hallucination and improves grounding—a common failure point for VLMs.\n\nSystematic Data Pipeline: The development of an automatic, multi-source 300k SFT dataset is a significant engineering contribution. The five-level capability curriculum, which uses external geometry tools (Metric3D v2, WildCamera) and rule-based filters, is well-conceived. Ablations effectively demonstrate that both the SrCoT and the alignment components contribute to performance.\n\nComprehensive Evaluation: The paper is thoroughly evaluated. FSD achieves state-of-the-art or near-SOTA results across a wide range of benchmarks (CVBench, CRPE, SAT, BLINK, EmbSpatial) and shows strong performance on grounding tasks (RoboRefIt, Where2Place). The large margins on the newly proposed VABench, combined with a 72% success rate on 8 real tasks, provide compelling evidence for the method's efficacy.", "weaknesses": "Incomplete Compute and Latency Attribution: The paper makes claims about deployability but fails to provide a rigorous analysis of its computational cost. A single small latency table is insufficient. The review lacks a module-level breakdown of FLOPs, parameters, and wall-clock time for essential components (projector, LLM tokens, SrCoT decoding, visual-aid generation, 2D→3D lift, and planning). Without this, it is impossible to judge the method's cost-benefit trade-off against modular or end-to-end VLAs under matched hardware and batching conditions.\n\nUnexplored 2D Trace Limitations: The core robotic primitive—a 2D trace lifted to 3D via depth and pinhole back-projection—is sensitive to error. The paper does not quantify how errors in camera calibration (focal length, principal point), depth estimation noise, or occlusions accumulate and affect the final 3D trajectory. While 2D limitations are acknowledged, the lack of robustness studies is a significant gap.\n\nUnknown Data Quality and Bias: The 300k auto-constructed dataset is a potential source of unmeasured error. The paper does not report label noise rates, inter-annotator agreement for the 300 manual VABench items, or how errors from the data pipeline (eg, the 20% relative depth gap filter) propagate to the quality of the final SrCoT reasoning.\n\nLimited Scope of Robotic Interaction: The evaluation, while broad, is shallow in its robotic scope. Most executions appear to be open-loop, with closed-loop feedback mentioned but not rigorously evaluated. There is no analysis of drift, re-planning, or recovery from grasp failures. Furthermore, the tasks do not explore multi-step long-horizon sequences, heavy clutter, moving distractors, or significant camera pose changes, all of which are critical for real-world autonomy.\n\nWeak Justification for Design Choices: Several key hyperparameters are fixed without ablation (eg, T=8 trace points). The evaluation for visual aids on VABench relies on a mix of metrics (MAE/RMSE) and an unvalidated LLM-score proxy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sC5cOfufXm", "reviewer": "Reviewer_Kh3L", "summary": "This paper introduces FSD, a vision-language model designed to bridge visual reasoning and robotic decision-making. FSD employs a Spatial Relationship-focused Chain-of-Thought (SrCoT) mechanism to generate intermediate visual representations (e.g., visual points and visual traces), enabling a structured transition from perception to action. It further integrates a hierarchical weak-to-strong data pipeline and a self-consistency alignment mechanism to enhance relation understanding and generation.", "questions": "Q1. It is recommended that the authors explore more challenging tasks, such as articulated object manipulation or contact-rich manipulation scenarios.\n\n\nI think this paper is currently borderline, but I’m willing to give it a borderline accept, provided that the authors address my concerns. If there is any misunderstanding on my review, I would appreciate it if the authors could clarify.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. Using visual prompts for zero-shot manipulation is a promising research direction.\n\nS2. The benchmark proposed by the authors can help the robotics community evaluate the generation and reasoning capabilities of VLM models.", "weaknesses": "W1. Generating such a large number of visual prompts may affect inference efficiency and potentially obscure critical information. Moreover, I am not convinced that this model can achieve faster inference than end-to-end VLA models. The authors are encouraged to compare execution efficiency with Pi_0.\n\nW2. How does the method address the closed-loop problem, for example, when the scene dynamically changes after generating visual prompts? Would redoing all prompt reasoning in such cases be too inefficient?\n\nW3. Regarding the automated dataset construction, does it only build image-level planar relationships? Could the authors consider incorporating spatial relationships in the 3D Cartesian coordinate system?\n\nW4. The end-to-end VLA scores reported on SimplerEnv (WidowX Robot) appear to be taken from SpatialVLA (Qu et al., 2025). It is recommended that the authors reproduce existing VLA models for fair comparison, include more recent VLAs, and report fine-tuned results as well.\n\nW5. Although multiple visual and language prompts are generated, the accuracy does not reach the current SOTA. The authors should compare FSD with fine-tuned end-to-end VLAs, since such models are trained through imitation learning rather than a purely zero-shot paradigm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)