RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation
Authors: Hao Li
Ziqin Wang
Zi-han Ding
Shuai Yang
Yilun Chen
Yang Tian
Xiaolin Hu
Tai Wang
Dahua Lin
Feng Zhao
Si Liu
Jiangmiao Pang
Venue: ICLR 2026 Poster (2026)
Keywords: Robotics
Manipulation
VLM
LLM
Benchmark
Primary area: applications to robotics, autonomy, planning
TL;DR: Introduces a unified data-benchmark-model suite of per-frame intermediate representations of robotic manipulation, driving the exploration of plan-then-execute VLAs.

Abstract: Advances in large vision-language models (VLMs) have stimulated growing interest in vision-language-action (VLA) systems for robot manipulation. However, existing manipulation datasets remain costly to curate, highly embodiment-specific, and insufficient in coverage and diversity, thereby hindering the generalization of VLA models. Recent approaches attempt to mitigate these limitations via a plan-then-execute paradigm, where high-level plans (e.g., subtasks, trace) are first generated and subsequently translated into low-level actions, but they critically rely on extra intermediate supervision, which is largely absent from existing datasets. To bridge this gap, we introduce the RoboInter Manipulation Suite, a unified resource including data, benchmarks, and models of intermediate representations for manipulation. It comprises RoboInter-Tool, a lightweight GUI that enables semi-automatic annotation of diverse representations, and RoboInter-Data, a large-scale dataset containing over 230k episodes across 571 diverse scenes, which provides dense per-frame annotations over more than 10 categories of intermediate representations, substantially exceeding prior work in scale and annotation quality. Building upon this foundation, RoboInter-VQA introduces 9 spatial and 20 temporal embodied VQA categories to systematically benchmark and enhance the embodied reasoning capabilities of VLMs. Meanwhile, RoboInter-VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants that bridge high-level planning with low-level execution via intermediate supervision. In total, RoboInter establishes a practical foundation for advancing robust and generalizable robotic learning via fine-grained and diverse intermediate representations.

Published as a conference paper at ICLR 2026
ROBOINTER: A HOLISTIC INTERMEDIATE REPRESEN- TATION SUITE TOWARDS ROBOTIC MANIPULATION
Hao Li Ziqin Wang Zi-han Ding' Shuai Yang Yilun Chen Yang Tian Xiaolin Hu Tai Wang Dahua Lin? Feng Zhao Si Liu Jiangmiao Pang21 2
UUniversity of Science and Technology of China, SShangaii Artificial Intelligence Laboratory, BBeihang University, *Nanyang Technological University, ZZejjanng University, oTsinghua University, "The Chinese University of Hong Kong Project page: https://lihachts g thub. io/ Robo Inter .wtcer.ggiten github.
ABSTRACT
Advances in large vision-language models (VLMs) have stimulated growing in- terest in Dision-language- ever, existing manipulation datasets remain costly to curate, highly embodiment- specific, and insufficient in coverage and diversity, thereby hindering the gener- alization of VLA models. Recent approaches attempt to mitigate these limita- tions via plan-then-execute paradigm, where high-level plans (e.g., subtasks, trace) are first generated and subsequently translated into low-level actions, but they critically rely on extra intermediate supervision, which is largely absent from existing datasets. To bridge this gap, we introduce the RoboInter Manip- ulation Suite, a unified resource including data, benchmarks, and models of in- termediate representations for manipulation. It comprises Robolnter-Tooe a lightweight GUI that enables semi-automatic annotation of diverse representa- tions, and RoboInter-Data, a large-scale dataset containing over 230k episodes across 571 diverse scenes, which provides dense per-frame annotations over more than 10 categories of intermediate representations, substantially exceeding prior work in scale and annotation quality. Building upon this foundation, RoboInter- VQA introduces 9 spatial and 20 temporal embodied VQA categories to system- atically benchmark and enhance the embodied reasoning capabilities of VLMs. Meanwhile, RoboInter- VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants that bridge high-level planning with low-level execution via intermediate supervision. In total, RoboInter estab- lishes a practical foundation for advancing robust and generalizable robotic learn- ing via fine-grate greigea and diverse intermediate representations.
(VLA) systems for robot manipulation. How-
INTRODUCTION
The remarkable generalization of large language models (LLMs) and vision-language models (VLMs) through large-scale pretraining has inspired efforts extend this paradigm to robotics, giv- ing rise to end-to-end Dision-language-ace
(VLA) models (Brohan et al. 2023 Kim et al. [024] Black et al. (0244) Bjorck et al. [2055] Lu et al., 2025). Although web- scale multimodal data enables
broad semantic reasoning, existing large-scale robot datasets (et al., 2023 Khazatsky et al.) 2024
Wu et al. 2024 Bu et al. 2025) remain costly and tightly coupled to specific embodiments despite
massive efforts in data collection (Fu et al., 2024b), leaving significant gap in generalization.
To address this gap, recent research has explored frameworks that separate planning from execution. Modular approaches (Huang et al. 2023 Belkhale et al. 2024] Huang al.
2024a Liu et al.
2024a Nasiriany et al. (2244) infer level structures before translating them into low-level ac- tions, enabling better generalization but often relying rule-based design. Mcanwhile, many end-
to-end VLAS (Zhou et al. 2025b Yang et al. 2025b) Zawalski et al. 2024 Shi et al. 2025] Lin
al. 2025 Deng al. 2025 Cen al. 2025
Wu et | 2025 Du al. (023) introduce interme-
diate representations (e.g., subtasks, visual traces, 2D/3D grounding, etc.) as extra input conditions
"Equal contributions, ordered by coin toss. PProject leader. 'Corresponding authors.

---

Published as a conference paper at ICLR 2026
Robolnter
10- Typea Interftp Data
Par-Frame Denso
Spatial
Temporal
230k Epizode Annotation
Diverse acenw/arm typer
Transfer General to Robotics
Types InterRop Q4
20 Typea Interllep QA
1,2M Training Sample
1,7M Training Sampla
Bunchmank Support
Benchmart Support
General Data
Robotic Data
Sea meentoring
Lopsp Rose
VLMs VLAs
Locatia
Gropper
Robalnter
Drnnact Feamhe
@: What la the nout step?
Afferdance
@: Generate gripper trace.
Robolnter-Date
Asscerge
Robointer-VQA
SpurtiOg
Obj/Gripper
Annotation
Subtasks Annotation
Kay Frame Annotation
Dens kug:
Instructions
Planner
Object Box? Afferdence?
Tosk Plans?
Intermediate
Moue Trace?
Asynchronous
Offline
Tranefer Pretrain Knoastedyn
Check &Complle
Data Intra
Robolii Planstng
Executor
Intermealate Gustance
Robolnter-Tool
Excoution
Robointer-VU VLA
Figure 1: RoboInter manipulation suite includes annotation tools, annotated data, curated VQA dataset, and their applications in VLMs and VLAs. RoboInter provides a dataset with over 230k episodes (mainly from Droid (Khazatsky et al., 2024) and RH20T (Fang et al. 2023)) and 10+ types of intermediate representation annotations, named Robolnter-Date a curated embodied VQA benchmark and dataset covering 29 spatial- and temporal-level categories, Robointer- VQA; and an integrated plun-then-execute framework for training VLM and VLA models, Robolnter-VLA.
or supervision signals. Despite their differences, both directions converge on introducing planning intermediate representation, which we summarize as the plan-then-execute paradigm.
The effectiveness of this paradigm critically depends on high-quality intermediate representations. Existing datasets (et al. 2023 Khazatsky al. 2024) typically pair visual inputs with overall in- structions and robot actions, but they rarely provide the fine-grained intermediates required for plan- then-execute. Collecting and annotating fails to leverage the advantages of community-drivee open-source data and makes unavailable for large-scale pretraining. Recent efforts (Li et al., 2025b, Yuan et al. 2024) have explored automated annotation for existing datasets, yet with limited success. For example, LLARVA (Niu et al.) 2024) leverages a pretrained gripper detector to generate large-scale traces, but it is sensitive distribu- tion shift. ECoT (Zawalski et al. 2024h annotates neeudo-nterrmddite textual planning with object grounding via Gemini (Team) 2023). ShareRobot (Ji et al., 2025) combines automated annotation with manual verification, but only at small scale and with labels misaligned to step-wise actions. Overall, current datasets lack lcage-scale, high-quality annotations, which limits their value for ad- vancing research intermediate representations for VLMs and plan-then-cxecute VLAs.
data remains costly and infrastructure-ineenst
which
To address this gap, we propose the RoboInter Manipulation Suite, illustrated in Figure Built on RoboInter-Tool, lightweight GUI for cemi-automatic per-frame annotation of embodied videos, we introduce RoboInter-Dee -Dtta, a large-scale, per- framframe annotation dataset of intermediate represen- tations for robotic manipulation. As shown in Table Robolnter-Data provides
230k episodes across 571 distinct scenes, surpassing LLARVA (Niu at. al. 2024). ECoT (Zawalski et al.) 2024). and ShareRobot (Ji al. 2025) in both scale and diversity. Unlike prior datasets that either cover lim- ited number of scenes (Chen et al. (024a) or rely solely on automatic annotation (Li et al. 2025bb Zawalski et al. 2024). Robolnter-Dac Data uniquely combines large-scale coverage with human-in-ther loop verification. To the best of our knowledge, this is the first real-world manipulation dataset that provides dense, per-frame annotations of intermediate representations spanning more than ten categories, including subtasks, primitive skills, segmentation masks, gripper/object bounding boxes, placement proposals, affordance boxes, grasp poses, traces, and contact points. All annotations are temporally synchronized with executed actions and robot states, together with two-view observa- tions (one third-person and one wrist-view camera), enabling end-to-end action learning.
Leveraging these fine-grained annotations, we introduce RoboInter- VQA, which exposes and benchmarks existing VLMs from the perspectives of embodied generation and understanding, com- prising 9 spatial and 20 temporal embodied VQA categories to enhance the embodied abilities of VLMs. Built upon the high-level VLM planner trained on these curated VQA data, we introduce Robo VLA, integrated elae-uhen-execute framework that supports both modular and end-
2

---

Published as a conference paper at ICLR 2026
Table Comparison of embodied annotations datasets. Emb.-VQ2 denotes the availability of cu- rated embodied VQA benchmarks and datasets; E2E-ACT indicates whether the dataset temporally aligned with executed actions; Curated-Col specifies nulti-intermediatee chain-of ohain-of-thought support.
Dataset
iVidee
#Scene
Dense
-VQA
Emb.
-ACT
EZE
Curuted -CoTT
Subtusk &Skill
Allor- -dance
Contact Point
Gripper Box
Object Box
Trace
Annotation
Type
LLARVA
311
X
x
X
X
Auto
RH20T-P
38k
V
X
V
X
X
Humam+Ault
ECOT
60k
12
/
X
/
J
V
X
X
V
V
X
Auto
AgiBot World
1M
106
X
V
/
X
Human
VLA-O5
V
V
ShureRobot
51k
X
V
Human+Auto
Robo2VLM
176k
463
X
/
X
J
X
X
Auto
VeRmin
12k
X
X
X
X
Human
Ours
230k
571
V
V
Human=Auto
to-end VLA variants, enabling rapid adaptation from planning to execution, and then we systemati- cally investigate the impact intermediate representations on the generalization and controllability of VLA variants. Through extensive experiments, we show that RoboInter-Date substantially im- proves the reasoning and grounding capabilities of VLM planners, particularly in understanding and generating various embodied intermediate representations for manipulation, thereby strengthening VLMs' foundational embodied abilities. Open- and closed-loop evaluations of manipulation tasks further demonstrate that the pretrained planners and intermediate representation data provide performance and generalization gains to the VLASs. By analyzing the trade-offs among different VLA variants, we establish a unified foundation for leveraging these data in future research. We release RoboInter to the community, including its corresponding data, benchmarks, and models, and hope they can pave the way for applications of intermediate representations for Embodied Al.
RELATED WORKS
Embodied intermediate representation. Embodied intermediate representations have been widely
explored. Prior work leverages 2D trace (uu et al. 2023). optical flow (Xu et al., 2024). sub-
tasks (Zhang et al. 2024 Belkhale al., 2024). key points Wenn al. al.] (023)) future images (Zhao
et al. 2025 Lv et al., 2025 Cai et al. (2026) and grounding box (Sundaresan et al. 2023 Huang
al.
2025) to guide action generation. While some representations (e.g., grounding box flow)
can be estimated by vision foundation models (Ravi et al. [2024] Dong & Fu 2024). the results are not always reliable across different embodied scenes. Web-scale resources remain under-aligned with manipulation tasks, embodied alternatives such as RoboBrain (Team et al. 2025) and Ve- Brain (Luo et al. 2025) curate task-specific embodied datasets and train VLMs to generate interme- diate representations. Another line (Li et al., 2025b; Yuan al. 2024) collects large-scale datasets annotated with a single type of intermediate representation and leverages them to generate actions.
Robotic manipulation datasets. Numerous prior works (Fang et al. 2023 Khazatsky al. 2024 et al., 2023) Gao et al., [0255] Tian et al.) 2025) have introduced real-world or simulated robot datasets encompassing diverse sources, scenarios, and skills. While these data do not provide native intermediate representation labels for real-world manipulation tasks, subsequent efforts have pro- vided some annotations. RH20T-P (Chen
2024a) augments RH20T with primitive-level labels of subtasks, RT-H (Belkhale et al. 2024) provides language descriptions of grasp poses and motions. LLaRVA (Niu
2024) and Hamster (i Li al. 2025b) leverage gripper tracking to generate large- scale 2D trajectories for pretraining. High-level planning approaches, such as Robo2VLM (Chen) ct al. 2025a), sample frames and then supply diverse VQA annotations to enhance embodied scene understanding. RoboInter introduces pre-frame dense annotations data across varied intermediate representations to advance both embodied understanding and end-to-end action learning.
Embodied planning for execution. Embodied planning is important in diverse and complex real- world settings. Plan-then-execute systems can be categorized into implicit and explicit forms. Im- plicit methods operate black boxes, where these VLA methods (Black et al. 2024 Li et al.) 2023a 2025a) primarily rely on implicit reasoning by fine-tuning pretrained VLMs of various pretraining paradigms. Explicit methods generate interpretable intermediate representations and directly uti-
3

---

Published as a conference paper at ICLR 2026
Robolnter-Date
A7OP Scace Typea
Object
Took: Purolo bottle into dio
Subtlokk:
Piokt bottl
Prinnóvv Skalt Plat
Embodiments
Robointer-Tool
Objects
16 Primiiiteee Saills
Gripper
Langunga
Starbaan
Grgparkoypotr
Cantall home
Step? Data Collection
Step
GPT Pre-annotation
Record Video Subaut
Annotation Lang
Select Skills
Crosa Validation
Step 4 Generate VQAx
Video check
Record Contact
Generate
Select Trackable
Frame
Manipulated
Mask image
Results
Formatting
Gripper
Outtiers
Spatial VQA
Temporal
Episodes
Feaar wwweooo reglts.
Skill
Env
Kreto
Prane Fare (4000000
Riceess
Track EHICES
Statistics
Robolnter-VQA
Figure 2: Overview of Robolnter-Data and Robolnter- VQA. We collect and annotate 230k ma- nipulation episodes to obtain 10 types of intermediate representation annotations through Data Col- lection and Annotation & Check. We further construct large-scale, diverse of VQA spanning spatial and temporal dimensions. Statistics of raw episodes and the curated VQA are also provided.
lize the knowledge from VLMs, such as To.5 (Inteliignnce et al. 2025) and Rekep (Huang et al. 2024b). ECoT Zawalski al. 2024) introduces explicit CoT into VLA by prompting a VLM to autoregressively generate both COT text and discrete action tokens. VLA-OS (Gao et al. 2025a) fur- ther explores various model designs to combine planning and action generation. We offer large-scale intermediate representation data and the pretrained VLM Planners, joinnll enabling both implicit and explicit reasoning VLA variants through efficient information aggregation.
3 DATASET
3.1 ROBOINTER-DATE
As illustrated in Figure 2 Robolnter-Daie builds upon extensive manipulation datasets and provides large-scale, high-quality annotations diverse intermediate representations.
Data collection. To enhance dataset diversity, we collected two types of raw manipulation data: (1) In-the-Wille setting (i.e., diverse indoor scenarios), emphasizing the diversity of scenes and instructions. mainly collected from Droid (Khazatsky et al. 2024) and OXE fet (023). (2) Table- Top setting (i.e., tabletop interaction scenarios), highlighting the high quality and skills diversity, collected from RH20T (Fang et al. [023]. By integrating raw teleoperated video recordings of these datasets, followed by rigorous screening and preprrcesssig, we constructed high-quality, large-scale database consisting of 230k manipulation episodes (third-person videos) in total.
Annotations Check with Robointer-Tooe For accurate and comprehensive labels, RoboInter- Tool is employed to perform the following annotations: (I) Task decomposition & key-frame anno- tation. Each manipulation video decomposed into clips using 15 predefined primitive skills, and ChatGPT (OpenAl 2023) is employed produce preliminary references for language annotations.
Human annotators utilize Robolnter- Tool to segment video clips, assigning each clip to a primitive skill, and simultaneously completing clip-level and video-level language annotations. The contact
4

---

Published as a conference paper at ICLR 2026
frame where the robot arm contacts the manipulated object is also recorded. (2) Recognizing the manipulated object. After recording the object that the robot arm interacts with, Robolnter- Tool au- tomatically transports the annotation to SAM2 (Ravi et al. 2024) for object segmentation and track- ing, and the result is asynchronously returned for review. A re-annotation and inspection mechanism reduces the impact of segmentation or tracking errors on the quality. (3) Locating the end-effector. As many raw recordings lack reliable camera parameters, directly projecting 3D coordinates to ob- tain accurate 2D end-effector traces is often infeasible. We estimate a calibration matrix improve projection accuracy and complement parameter- miinnegn-missing episodes via gripper detection and point tracking, enabling reliable reconstruction of the 2D trace. Details are provided in Appendix [A.6]
Post- proe-peoeesed annotations. By reorganizing the above annotations, we derive additional inter- mediate representations: (1) Grasp annotation. The grasp affordance box is inferred from the 21) end-effector location at the annotated contact frame. The contact points are the pre-defined key points of the gripper at the moment of contacting, while the corresponding robot state (i.e., the 61) end-effector pose) defines the grasp pose. (2) Placement annotation. The object position at the end of the subtask is treated as the target placement location. (3) Gripper annotation. Anchor points enclosing the gripper are identified and projected from 3D to 2D using camera parameters and robot states, forming the gripper bounding box on the 2D image coordinates.
Statisties of Robolnter-Datee As shown in Figure 2 provide high-quality, scene-diverse inter- mediate representation annotations for 230k manipulation episodes. This dataset includes 6 types of robot arms, 571 types of scenes, and 15 types of primitive skills. With the assistance of the Pabolnter-Tool.. we produce nearly 61M-frame object grounding annotations, about 70M-frame gripper trace annotations, 190k affordance box and placement proposal annotations, and nearly 760k language clip annotations. Despite this scale, the annotations maintain high quality.
3.2 ROBOINTER- VQA
VQA task construction. As illustrated in the Tobolnter-VQA section of Figure 2 we convert the annotations into diverse VQA tasks to enhance VLM capabilities. Tasks are organized along two axes: (i) intermediate representation type (spatial vs. temporal) and (ii) target capability (under- standing vs. generation). (I) Spatial VQA for understanding. We design three selection tasks and one judgment task to train spatial comprehension, including selecting the correct object bounding box or grasp pose, matching scenes to instructions, and determining whether contact occurs. (2) Spatial VQA for generation. It includes five prediction tasks that require generating spatial inter- mediate representations for downstream execution and complete specific content based on spatial reasoning, which includes object bounding box, grasp pose, placement proposal, key points, and the gripper bounding box. (3) Temporal VQA for understanding. To evaluate how the VLMS under- stand motion traces and the relationships between subtasks and observations, we design selection tasks and judgment tasks for temporal information. We design five selection tasks for movement directions of grippers, the matching of trace and description, subtask/primitive discrimination, and execution stage identification. Four judgment tasks assessing task success and next-step feasibility. (4) Temporal VQA for generation. We formulate tasks that require trace generation and multi-step planning under varying levels of contextual completeness. Prompts condition on different amounts of prior information (e.g., past subtasks overall instructions) and ask the model to predict the subsequent steps or multi-step planning. Video-based visual inputs are also used to summarize past events and predict feasible next steps. Trace generation evaluated under both easy and challenging settings (with or without initial waypoints). Details are included in the Appendix A.7.3
Statisties of RoboInter- VQA. Our VQA data is also large in scale, comprising approximately IM spatial generation entries, 172k spatial understanding entries, 131k temporal generation entries, and 935k temporal understanding entries. To prevent information leakage between training and valida- tion, we carefully designate 7,246 videos as the evaluation pool with the remaining data used for training, and sample validation sets for each question category from this pool.
ROBOINTER- VLA
In this section, as illustrated in Figure 3 we present BabolnterrVLA., a family of models following plan-then-execute paradigm, consisting of Planner and an Executor. Rather than monolithic
5

---

Published as a conference paper at ICLR 2026
Planner
Executor
Intermediate
Understanding
Generation
Subtwok, Skili, Object Roc, Pose,
ofouuhh
Planning
pur rotece
Intermadiate
Subtosk
Deseription: move the red thee tap the
VLM
Primitive
Temparal aeuthe
Guldance
999
Shill
Transfer: Pacce/ Pick/ Prenc....
Gripper
Box
Gebiial
Diverse Scenes
Information Aggregator
Object
Box
Difiision,
Contact
Malti-Ojjects
Points
Chunking
Graap
Porn
EaltleTrg Top
Trace
andaae
Cominne Actions
Planner
Exccutor
Planner
Exccutor
Planner
Exccutor
Winght
Wingin
Intornediats
VLM
VUM
Head
VIM
(2), Rebellter--C-EE
Rebolater-ECFFFE
(ch. Robubotee-Miaaker
Figure 3: Framework of Robolnter-VLA. Our model follows plan-then-execute paradigm with a VLM-based Planner and an Executor. The Planner exhibits enhanced understanding and gen- eration for manipulation, strong general grounding abilities, and robust perception across diverse scenes. The Executor shares the VLM backbone with the Planner. Three variants are supported, and intermediate representations in Flexible Chain-of-Thought (F-CoT) bridge planning and execution.
design, Robolnter- Veoln supports multiple variants and enables flexible adaptation from planning to execution. The Planner performs high-level decision-making by combining general and embodied reasoning to produce intermediate representations, which guide the Executor in translating multi- modal observations and language instructions into low-level actions.
4.1 MODEL ARCHITECTURE
VLMs Planner. The Planner model acquires embodied capabilities through visual question answering formulation with co-training strategy. To capture both spatial and temporal information, adopt VLM architectures that support single- and multi-image inputs, including the Qwen-VL VL series (Wang et al., (024b) and LLaVA- Vision (Li et al. 2024a). Each model consists of base LLM, vision encoder, and an MI.P-based vision-language projector. The Planner generates outputs autoregressively and is optimized using a cross-entropy loss.
VLAs as Executor. To systematically and lightweightly derive from the Planner, we build Executor on a Qwen2.5-VL. backbone with Diffusion Transformer (DiT) action head (Peebles & Xiel 2023). We further utilize an information aggregator that gathers the hidden states of all input and output tokens, well as intermediate representations, and compresses them into conditioning features with a controllable length. The Executor consumes multi-view visual observations (c.g., primary and wrist), language instructions, and intermediate representations (based on primary observation), and produces multi-step action chunks via a diffusion loss.
4.2 PLAN-THEN-EXICE PARADIGMS
As shown in Figure by leveraging the pretrained Planner, we provide three paradigms to en- hance downstream action execution: (1) Robolnter-IC-E26 Implicitly-Conditione End-to-End), which directly injects the VLM from pretrained Planner into the end-to-end Executor, using as stronger vision-language feature extractor. This approach can yield robust embodied percep- tion and more accurate task-relevant visual cues. (2) Robolnter-EC-
(Explicitly- Explicity--Condiionee Conditionee End-to-Endy where the Executor is initialized with the VLM of the Planner and jointly optimizes both mntemmediaterrpreesattttooo reasoning and action generation.
(3) Robolnter-Moduul (Mod- ular Planner-to-Exeecuorri Don-E2E hierarchical design that treats the Planner and Executor as

---

Published as a conference paper at ICLR 2026
Table 2:
Performance comparison on third-party benchmarks. Including Embodied, Ground- ing, and General benchmarks for general VLMs (upper) and embodied VLMs (lower).
Embodied
Grounding
General
Model Name
Where- Splacet
RoboRefTt-
Robo- VQAT
Refcocn- g-valj
Refenca+ val?
Refcoco-
Text- VQAT
CO-
OCR- bench?
MM- VETV
MME?
POPET
testt
vall
CO1
1ntemVLL-1BB
2.65%
7.0%
30.5
79.8%
73.2%
83.0%
75.1
23.7
798
1907
58.9%
90.7%
IntemVL3-28
1.86%
27.5%
27.7
87.6%
84.0%
85.8%
77.0
27.9
835
2186
62.2%
89.6%
VL..88VL.3.88
1.95%
27.7%
27.9
92.5%
26.5
580
81.3%
91.1%
QwenVL22.53B
11.8%
68.9%
37.6
85.2%
82.4%
89.1%
79.3
15.7
797
2175
61.8%
85.9%
OwenVL2.557B
18.9%
75.8%
38.4
87.2%
84.25
90.2%
84,9
15.0
864
2306
67.1%
85.9%
LLaVA-OV-7B
7.9%
10.4%
31.4
71.9%
69.7%
73.8%
71.1
8.4
882
2307
67.3%
86.4%
ReboBrainn2.0-33
59.8%
30.9%
30.6
55.0%
51.5%
50.9%
81.0
27.2
811
2126
59,4%
88.1%
RoboBrain-2.0-75
63.6%
8.8%
31.6
62.9%
70.1%
76.1%
75.9
25.2
857
2076
61.4%
86.2%
Rebeloter-Qwenn-B
58.3%
80.0%
43,3
87.9%
85. 8%
89.5%
78.9
15.6
787
2180
61.0%
90.5%
Seoolnter-Qwen-72
65.8%
85.6%
24.4
88.4%
86.6%
91.5%%
83.0
15.9
832
2281
62.3%
91,4%
Robolnter LLBVAOV-7B
66.3%
89.3%
74.5
87.3%
84.2%
91.33%
72.2
15.8
725
2217
61.4%
90.4%
independent modules. During training, the Executor conditions on ground-truth intermediate repre- sentations to generate actions, whereas during inference it uses the predicted results of the Planner.
Flexible chain-of-thought for intermediate representations. To support the explicitly conditioned and modular architectures, introduce F-CoT,, chain-of- -fthought composed of multiple interme- diate representations. F-CoT plays two roles: (i) as VQA supervision for training the Planner, and (ii) action-aligne guidance for the Executor. In Robolnter-IC-E2E, the VLM generates F-CoT content, which is directly consumed by the DiT head. In Robolnter-ooullar the Planner produces the F-CoT content and the Executor conditions on it. F-CoT flexibly combines representations such as subtasks, skills, object bounding boxes, affordance boxes, motion traces, etc., in textual or visual form, allowing users to select subsets tailored to specific embodied tasks. We denote textual F-CoT as RoboInter- Te-Modular and visual-prompted F-CoT as Sobolnter-Im-Modulla
5 BENCHMARKING AND EXPERIMENTS
5.1 BENCHMARKING THE PLANNER
Enhanced grounding and embodied capability. As shown in Table 2, we evaluated on third- party spatial reasoning benchmarks, including Where2Place (Yuan al. (024) and RoboReflt (Lu et al.
2023) (spatial point and grounding reasoning) and RoboVQA (Sermanet et al., 2024) (tem- poral task planning). Across all three benchmarks, our models substantially outperformed the base models (Qwen2.5-VL-33//
(Wang al. 2024b) and LLaVA-One -LaVVA-OneVisionn7B Vi (Li et al. 2024a)). Notably, RoboBrain2.0 (Team et al.) 2025 is also an embodied VLM (i.e., Planner). At the 3B scale, Robolnter- Qwen-3B achieved 49.1 % improvement over RoboBrain2.0
RoboRefft and a 12.7% improvement on Robo VQA. At the 7B scale, the corresponding gains reached 76.8% and 42.8%, respectively. For grounding, all three RoboInterVI.M variants exceeded their respective base models on Refcoco (Lin et al., 2014). Particularly, Robolnter- Qwenterrrr-enerB eventually ranked second overall, with a 27.4% relative improvement over RoboBrain2.0-78 On general benchmarks, our models remained relatively stable on most benchmarks, indicating that our curated VQA data en- hances the abilities of embodied reasoning and grounding. meanwhile general capabilities of our VLMs are slightly affected.
RoboInter- VQA benchmark the spatial and temporal level. As shown in Table for spatial- based generation tasks, closed-source and general VLMs without embodied experience rarely pro- duce accurate intermediates (typically below 40%), underscoring the importance of additional inter- mediate representation annotations. On simpler questions, Gemiini-2.5-Flash and RoboBBain-2.0--7 lead on Grasp Pose (choice) with 32.7% and 23.3% ACC, For Grounding Choice LLaVA-OV-7B achieves 31.9% 31.9% while Gemini-2.5-Flash is strongest at 69.4%; most other models remain near ran- dom choosing (25%) given limited understanding of manipulation scenes. For temporal, closed- source API and general VLMs largely fail to generate future traces or task planning; RoboBrain-2.0 attains a much better DTW in Trace Generation than Qwen- VL-2.5 (541 v.s. 1702). On Visual Trace Choice, Gemini-2. 2.5- remains competitive (49.4% (9.4%) For Planning Choice and T/F of Task Plan-

---

Published as a conference paper at ICLR 2026
Table 3: Results of RohoInter-VQA spatial and temporal benchmark. G.D. means grounding. A.F. denotes Affordance. Spatial generation uses ACC@IOU>0.1 (%f), multiple choice and T/F use ACC (% For temporal, Trace uses DTW (L), other metrics use ACC or average BLEU (1).
RoboInter-VO VQA Spatial
Robolnter- VQA Temporal
Model
Generation
Multiple Choice
T/F
Generation
Multiple Choice
T/F
Object
Grusp A.F.1
Place A.F.
Gripper G.D.
Grasp Pose
Grouding Choicet
Task Trace Planning
Visual Planning
Task
G.D.
Contact
Trace?
Choicet
Planning?
QweaVL2.5538
46.6%
12.2%
34.1%
6.1%
21.1%
21.9%
50.9%
2712
20.3
37.5%
60.0%
59.7%
QweaVL2.5-7B
51.2%
14.7%
38.2%
10.2%
27.3%
25.7%
52.5%
1702
22.4
39.0%
64.5%
60.5%
IntemVL33-1B
7.8%
2.3%
8.3%
1.2%
24.8%
25.9%
50.4%
10.5
28.9%
54.9%
55.8%
I..emVL.3.2B
20.6%
3.1%
17.9%
1.9%
25.5%
27.3%
50.1%
7.7
35.2%.
59.3%
59.4%:
32.7%
5.9%
28.2%
3.5%
25.1%
31.1%
52.9%
1035
34.0%
71.5%
60.0%
Llava-OV-7B
25.8%
5.5%
23.7%
1.6%
24.5%
31.9%
54.4%
11.0
37.7%
44.9%
63.5%
GPT4o-mini
6.8%
7.2%
1.1%
10.9%
16.8%
53.6%
1736
14.7
28.4%
66.6%
63.9%
Gemini-2.5-fash
1.7%
1.2%
32.7%
69.4%
65.5%
49.4%
RohoBrain 3B
15.2%
2.8%
25.5%
26.5%
50.4%
595
16.0
29.7%
48.2%
46.8%
RoboBrain 2.0-BB
2.5%
23.3%%
21.5%
49.2%
15.3
29.5%
57.8%
46.4%
RoboInter- Qwen-3eennnn 3B
76.1%
34.9%
52.7%
61.66%
74.0%
73.4%
75.2%
332
61.2
78.8%
82.2%
88.7%
RoboInter-Qsen--7
75.1%
37.8%
56.9%
%
75.7%
75.6%
323
63.4
81.9%
86.5%
93.0%
Robolnter-LaavaV--78
82.9%
46.3%
55.1%
70.1%
74.1%
79.7% %
76.3%
299
62.7
81.9%
81.8%
83.9%
ning, planning aligns closely with general LLM abilities, most models transfer common-sent knowledge and show better performance. Overall, current closed-source and general VLMs typi- cally lack enough embodied abilities. Curated from diverse annotations, Robolnter- VQA markedly improves the VLM abilities of understanding and generating intermediate representations.
5.2 OPEN-LOOP EVALUATION OF THE EXECUTOR
Experimental settings. In this section, we examine how a pretrained Planner improves the Ex- ecutor and compare different VLA paradigms. As annotated corpus spans more than 500 dis- tinct scenarios, comprehensive real-world validation across all scenarios is infeasible, as empha- sized by HPT (Wang et al.] 2024a). Following the discrete token-accuracy evaluation in Open- VLA (Kim et al. [024). we utilize an Open-Loop Score (OLS) to evaluate the generation of con- tinuous action chunks, in which per-step actions are assessed independently and compared with the ground-trum actions. OLS is computed as the average value over 100K transitions from eval- uation videos, ensuring statistical stability. More details in Appendix[A.44.3 Nine Executor vari- ants are evaluated: (a). Vanilla: omits any pretrained VLM from Planner, performing action learn- ing only; Robolnter--C-E EC-E2E, Te-Modular and Im-Modular are stated in Section (4.2, (f) Executor: not end-to-end, both training and inference are guided by GT intermediate representations; s).QQenVL++ Executor: training is GT-guided, and inference employs intermedi- ates from original Qwen2.5VL: (h). VLA-OS: we train VLA-OS (Gao et al.) (202aa) in setting and use as an additional baseline; Oi).Oracle+ VLA-OS: not end-to-end, VLA-OS are guided by GT in- termediate representations. Two open-loop evaluation settings: (1) In-the-Wild focus on scene and object generalization, we compare the convergence performance under identical training steps. (2) Toble-Top: focus on the tabletop environment and cross-embodiment ability, we mainly examine the evaluation curve during training. The annotated corpus is divided into In-the-Wiil and Table-Top subsets. We sample approximately 10% of episodes from each subset for Executor training (25k in total), of which 8% are reserved for evaluation.
Planner consistently improves the Executor's action generation. The In-the-Wild setting is shown in Table The Vanilla achieves a lower mOLS score than IC-E2E (0.3086 v.s. 0.3218), which incorporates intermediate representations, indicating that pretrained VLM Planner can en- hance the learning capability of the VLA Executor. The mOLS score of EC-E2E is higher than IC- E2E (0.3340 v.s. 0.3218), showing that the explicit intermediate representations are more helpful for action guidance than the implicit. For Executor,, the non-E2E architectures, utilizing ground- truth annotation in the Executor, achieve substantially the highest scores. This indicates that our annotations are informative and stable. Te-Modular surpasses EC-E2E (0.3543 v.s. 0.3340), implying that decoupling planning and execution facilitates dedicated optimization each capa- bility and mitigates mode conflict. Im-Modular performs slightly worse than Te-Modular (0.3439
8

---

Published as a conference paper at ICLR 2026
Table 4: Open-loop evaluation in In-the-Wild setting. We report OLS with different error thresholds (@0.1 to @0.01) and the mean value.
Bobelne Modata Bohdima--KC RC-376 E2E
Ondle-Exssterr
Method
Open-Loop Score (OLS)
mOLs
Opete Loop
@0.11
@80.55
60.03
@0.01
VLA-08
0.6180
0.3905
0.1928
0.0129
0.3035
Vanilla
0.6793
0,3608
0.1753
0.0189
0.3086
Robolnter IC E2E
0.6984
0.3810
0.1873
0.0204
0.3218
RohoInter EC E2E
0.7049
0.3930
0.2066
0,0314
0.3340
Training Steps (log scale)
0.6749
0.3582
0.1777
0.0298
0,3102
Figure 4: Open-loop evaluation in TopleTop
Roholnter-Te-Wee To- Medular
0.7124
0.4133
0.2332
0.0584
0.3543
setting. We show the curve of OLS@0.05 from
Robolnter- 1m- Modular
0.7056
0.4029
0.2240
0.0430
0.3439
Oracle+YLA-OS
0.7260
0.4928
0.2734
0,0200
0.3780
1K to 40k training steps. We mainly report the five variances of Robolnter-VLA.
Oracle-Executor
0.7511
0.4640
0.2705
0.0587
0.3861
v.s. 0.3543), as visual prompting embeds within information-dens images, diluting their relative contribution. QwenVL+Executte employs Qwen2.5-VV V zero-shot Planner, and its overall per- formance is not comparable with other Non-E2E models, showing that the embodied reasoning ability of our Planner is better than the general VLMs. The evaluation curves of the Table-Top Top shown in F as Table-Top scenes are easier to interpret, most methods eventually achieve high scores. Robolnter-Ti--Modle
Executor converge faster and reach higher performance. and Executo
EC-E2E converges more slowly but ultimately approaches the performance of Te-Modular. IC-E2E shows stronger early-stage results and maintains a consistent advantage over Vanilla after 20k steps.
Ablations on intermediate repre- sentations. We ablate combina- tions of different intermediate repre- sentations within the open-loop Or- acle+Exxcutt setting. As shown in Table coarse-grained represen- tations such as Subtask and Prim- itive Skill provide only marginal improvements, they offer stage- level guidance with limited action- able constraints during execution. In contrast, spatially grounded rep- resentations (Object Box, Gripper Box, and Affordance) yield substan- tially larger gains by providing finer- grained cues. The most significant improvement comes from Trace, which introduces dense, temporally grounded information and
Table 5: Ablation of intermediate representation. We port OLS under multiple thresholds. Six representations are evaluated, where finer- tner-grained categories yield larger gains.
Variant
OL.
mOLS
@0.1
600.0
Vanilla
0.6793
0.3608
0,1753
0.0189
0.3086
Subtask
0.6965
0.3676
0,1770
0.0171
0.3146
S. + Primitive Skill
0.6983
0.3681
0.1779
0.0194
0.3159
+ S. + P. Object Box
0.7025
0,3849
0.1988
0.0294
0.3289
+ S. + P.+ OB. + Gripper Box
0.7212
0.4032
0.2048
0.0272
0.3391
+ P. OB. + G.B. Affordance
0.7245
0.4083
0.2114
0.0297
0,3435
+ + S.+P++00 OB. + G.B. Aff. Trace
0.7511 0.4640 0.2705
0.0587
0.3861
achieves the strongest overall performance. Additional results are provided in Appendix A.2.3
5.3 CLOSED-LOOP REAL- WORLD EVALUATION OF THE EXECUTOR
Experimental setting. We study how our dataset and the pretrained Planner affect the closed-loop success rate. Experiments conducted in few-shot TableToop evaluation with a real-world Franka Research 3 arm. Observation input comprises a static third-person camera and a wrist-view camera, and no proprioceptive state. We focus on more practical E2E variants and evaluate five E2E models: (1) OpenVLA (Kim et al. 2024): Initialized from official pretrained weights and extended with an additional wrist-view input. (2) Pi-O (Black al. 2024): Fine-tuned from the official checkpoints
of Droid. (3) Vanilla: our baseline. (4) Robointer-Iee IC-EEE: Initialized from the Planner and further finetuned on in-distribution (ID) data. (5) RoboInter-EC-E22: Initialized from the Planner, and jointly optimize action and CoT generation, with 1:1 ratio of our annotated data and collected data (more details in A.I.I. We design four tasks: (a) Object Collecting: sequentially place three pens from a cluttered tabletop into cup, -ut-of-distribution (OOD) tests with novel objects, and alter spatial layouts. (b) Cup Stacking: Stack cups from left to right. OOD tests include novel objects, OOD positions, and continuous stacking. (c) Towel Folding: Fold two towels in sequence and stack

---

Published as a conference paper at ICLR 2026
867
180 900
31.
461
15.3
133
60.0
513
ALT
4AT
06.
In In-Distribasion
In Distributtionn
Out Distrribuii
In Distimbuttion
Ous-of-Disriuutoo
In Distributtion
Off-of-Distrizztio
Objects Collecting
Cups Stacking
Towels Folding
Clatters Cleaning
000 Pooilive
Nondl Ofjjct
000 Postine
Nond Object
Aovd Beckground
Nerdl Objjcct
Rynanik Jusenctioon
Naval Dijezz
Figure 5: Real-World Experiments. The top charts present results from 15 in-distribution (ID) and 15 out-of-distrbution (OOD) trials. The bottom panel illustrates the OOD test setup. Notably, the performance drop from ID to OOD reflects each model's generalization under distribution shift, where EC-E2E outperforms IC-E2E and exhibits a smaller ID-OOD degradation (8.3% vs. 19.0%), showing the consistent conclusion with the Open-Loop Evaluation. Key steps are marked with number, along with an end-execution thumbnail. Experiments of RoboInter-Modular are in Table
them. OOD tests vary in the towel category and background. (d) Clutter Cleaning: Clean all items from the board with brush. OOD tests introduce novel objects and disturbances.
Experimental results on and OOD testing. Across all tasks, Robolnter-IC-E2E consistently outperforms the Vanilla. In ID evaluations, IC-E2E attains average success rate of 77.3%, com- pared with 65.0% of Vanilla. Under OOD conditions, the gap widens, and IC-E2E achieves a 58.3% success rate, while Vanilla reaches only 38.3%, indicating the superior generalization of IC-E2E. The pretrained VLM from the Planner is pre-exposed to rich embodied data and therefore pro- Droid, also demonstrates
vides stronger perceptual priors. Although Pi-O, which is pretrained solid ID and OOD performance, the IC-E2E benefits from a broader representation training, thereby producing better overall results. EC-E2E records lower ID success than IC-E2E (68.3% vs. 77.3%). which seems to be misaligned with the open-loop results in which EC-E2E was superior. Actually, the open-loop protocol enforces strict decouple between training and validation, and there- fore functions more like an OOD test. Correspondingly, under real-world OOD conditions, EC-E2E exceeds IC-E2E Object Collecting (60.0% vs. 53.3%) and Cup Stacking (73.3% vs. 60.0%), and achieves a higher average success rate (60.0% vs. 58.3%). The ID-to-OOl drop is only 8.3% for EC-E2E, whereas IC-E2E declines by 19%. We attribute EC-E2E's weaker ID accuracy to the po- tential modality interference from the joint training of text generation and action prediction. Diverse OOD knowledge from our dataset contributes to superior 00D robustness and generalization. We provide more weal-world results in Appendix A.I
CONCLUSION
We presented the RoboInter Manipulation Suite, a unified platform designed to advance research on intermediate representations for the plan-then-execute VLA paradigm. At its core, Robolnter- Data offers over 230k episodes with dense, per-frame anmotations spanning diverse intermediate representations, establishing new scale and quality standard for real-world manipulation datasets. Built upon this foundation, RoboInter- VQA systematically benchmarks and improves the embodied generation and understanding capabilities of Robolnter- VLMs across rich spatial and temporal tasks. Robolnter- VLA further integrates intermediate representations into both modular and end-to-end frameworks, enabling a principled study of how representations influence execution performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is funded in part by the National Key R&D Program of China (20222D0160201)); Shang- hai Artificial Intelligence Laboratory; the National Key Research and Development Program of China (No. 2024YFB4709800) the Special Program of the Graduate School, University of Sci- ence and Technology of China; National Key R&D Program of China (2022ZD01 15502); Anhui Provincial Natural Science Foundation under Grant 2108085UD12: Ningbo Science and Technol- ogy Innovation 2025 Major Project (20252034); National Natural Science Foundation of China (No. 62461160308, U23B2010); "Pioncer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01 161). We acknowledge the support of GPU cluster built by MCC Lab of Information Sci- ence and Technology Institution, USTC. The AI-driven experiments, simulations and model training were partly performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
REFERENCES
Suncel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01822 2024.
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, al. GrOOt n1: open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. Wo! A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24111 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- Vision-language-actoon models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 158/8, 2023.
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng. Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint 25X3v22503.06669, 2025.
Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, et al. Internvla-al: Unifying understanding, generation and action for robotic manipulation. arXiv preprint ar wxXiv:2601.024555 2026.
Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, Deli Zhao, and Hao Chen. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21553 2025. URL htttps://arxiiii wwwpaawwwww....///////0000
21539
Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R Sanketi, and Ken Goldberg. Robo2vlm: Vi- sual question answering from large-scale in-the-wild robot manipulation datasets. arXiv preprint 1.Xi::5505.15517, 2025a.
Xinyi Chen, Yilun Chen, Yanwei Fu, Ning Gao, Jiaya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiangmiao Pang. Yu Qiao, et al. Intervlaamm:: A spatially guided vision-tanguage-action framework for generalist robot policy. arXiv preprint arXiv:2510.1377 13778 2025b.
Zeren Chen, Zhelun Shi, Xiaoya Lu, Lehan He, Sucheng Qian, Zhenfei Yin, Wanli Ouyang, Jing Shao, Yu Qiao, Cewu Lu, al. Rh20t-p: A primitive-level robotic dataset towards composable generalization agents. arXiv preprint arXiv:2403: 2403::243..9900 19622, 2024a.
Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic essual-lnguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.

---

Published as a conference paper at ICLR 2026
Shengliang Deng, Mi Yan, Songlin Wei, Haixin Ma, Yuxin Yang, Jiayi Chen, Zhiqi Zhang, Taoyu Yang, Xuheng Zhang, Heming Cui, et al. Graspvla: a grasping foundation model pre-trained on billion-scale synthetic action data. arXiv preprint arXiv:2505.03233, 2025.
Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 10068-19078, 2024.
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint ar arXhv:2302.001111 2023. httthtt/p//aaat
wwt.abr.////oooooo.eo....tt..eee wbwp/s/s..... 2302 00111
Abby O' O'Neill et al. Open X-Embodiment: Robotic learning datasets and RT-X models. https:
2023.
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.00595, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a. URLhtttss//arxxv
abs/
2306. 3394
Zipeng Fu. Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024b.
Chongkai Gao, Zixuan Liu, Zhenghao Chi, Junshan Huang, Xin Fei, Yiwen Hou, Yuxuan Zhang, Yudi Lin, Zhirui Fang, Zeyu Jiang, et al. Vla-os: Structuring and dissecting planming representa- tions and paradigms
aagion-languge-aan
models. arXiv preprint arXiv:2506. /7561, 2025a.
Ning Gao, Yilun Chen, Shuai Yang, Xinyi Chen, Yang Tian, Hao Li, Haifeng Huang, Hanqing Wang, Tai Wang, and Jiangmiao Pang. Genmanip: LIm-driven simulation for generalizable instruction- following manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12187-12198, 2025b.
Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.0197. 2023.
Haifeng Huang, Xinyi Chen, Yilun Chen, Hao Li, Xiaoshen Han, Zehan Wang, Tai Wang, Jiangmiao Pang, and Zhou Zhao. Roboground: Robotic manipulation with grounded vision-language priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22540 22550, 2025.
Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08248, 2024a.
Wenlong Huang, Chen Wang, Ruohan Zhang. Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint crXiv:2307.05973. 2023.
Wenlong Huang, Chen Wang. Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024b.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi-0.5: a vision-language--ationn model with open-world generalization. arXiv preprint arXiv:2504 16054, 2025.
12

---

Published as a conference paper at ICLR 2026
Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipula- tion from abstract to concrete. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 1724-1734, 2025.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better track together. In European Conference Computer Vision, pp. 18-35. Springer, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis. et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint 2rXiv:2403.12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint 2tt.:24006.09445, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint Ttt.:2408.033266 2024a.
Hao Li, Shuai Yang. Yilun Chen, Yang Tian, Xiaoda Yang, Xinyi Chen, Hanqing Wang, Tai Wang, Feng Zhao, Dahua Lin, et Cronusvla: Transferring latent motion across time for multi-frame prediction in manipulation. arXiv preprint arXiv:2506. 19816. 2025a.
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, al. Cogact: foundational assion-anguuageactionn model for syn- ergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411. 19650, 2024b.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01372 2023a.
Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards generalist robot policies: What matters in building Aision-language-action action models. arXiv preprint arXiv:2412. 14058, 2024c.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941 2024d.
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, et al. Hamster: Hierarchical action models for open-world robot manipulation. arXiv preprint arXiv:2502.05485, 2025b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- langualangguag models. arXiv preprint 10X55 2023b.
Fanqi Lin, Ruiqian Nai, Yingdong Hu, Jiacheng You, Junming Zhao, and Yang Gao. Onetwovla: A unified vision-language-action model with adaptive reasoning. arXiv preprint arXiv:2505. 2025. URL httlt:ttt////rttt org/ abs
hat////bb/255.... /2505. 11917
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic ma- nipulation through mark-based visual prompting. First Workshop on Rision-Languag Models for Navigation and Manipulation ICRA 2024, 2024a.
13

---

Published as a conference paper at ICLR 2026
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang. Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12). December 2024b. ISSN 1869-1919. doi: 10.1007/s1 4235-66 611432-024-4255-
URL http::/dx..dee. dtt./xxxdciio...oooooooooo/oooo. org/10. 1007/
Guanxing Lu, Wenkai Guo, Chubin Zhang. Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505. 18719, 2025.
Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. VI-grasp: a 6-dof in- teractive grasp policy for anguage-orientee objects in cluttered indoor scenes. In 2023 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 976-983. IEEE, 2023.
Gen Luo, Ganlin Yang, Ziyang Gong, Guanzhou Chen, Haonan Duan, Erfei Cui, Ronglei Tong, Zhi Hou, Tianyi Zhang, Zhe Chen, et al. Visual embodied brain: Let multimodal large language models see, think, and control in spaces. arXiv preprint arXiv:2506.0012. 2025.
Qi Lv, Weijie Kong, Hao Li, Jia Zeng, Zherui Qiu, Delin Qu Haoming Song, Qizhi Chen, Xiang Deng, and Jiangmiao Pang. Fl: A vision-language-actionn model bridging understanding and generation actions. arXiv preprint arXiv:2509.06955 2025.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowl- edge for vlms. arXiv preprint arXiv:2402.07872 2024.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406. 1 11815, 2024.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudcep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, 2024.
OpenAl. Gpt-4 technical report. arXiv:2303.08774 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1195-4205, 2023.
Karl Pertsch, Kyle Stachowiez, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- gaiionlanguage-actinn models. arXiv preprint arXiv:2501.09747 2025.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang. Jia Yuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual- language-action model. arXiv preprint arXiv:2501.15830, 2025.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint 10Xx::2408.007111 2024. URL htttss/::: / wttaaaiivv.rrrr abs 02.4000.0111
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Mul- timodal mongooorizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 645-652, IEFE, 2024.
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, al. Hi robot: Open-ended instruction following with hierarchical vision- sasio-language-acttoo models. arXiv preprint arXiv:2502. 19417, 2025.
14

---

Published as a conference paper at ICLR 2026
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 8317-8326, 2019.
Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh, and Jeannette Bohg. Kite: Keypoint-conditioned policies for semantic manipulation. arXiv preprini arXiv:2306. 16605, 2023.
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, et al. Robobrain 2.0 technical report. arXiv preprint DrXiv:2507.02029, 2025.
Google DeepMind Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023, /tts//////// http://wwwvv..orraa////112/2/1
Yang Tian, Yuyin Yang, Yiman Xie. Zetao Cai, Xu Shi, Ning Gao, Hangxu Liu, Xuckun Jiang, Zherui Qiu, Feng Yuan, et al. Interndata-al: Pioneering high-fidelity synthetic data for pre- training generalist policy. arXiv preprint arXiv:2511. 2025.
Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-vssall learning with heterogeneous pre-trained transformers. Advances in Neural Information Processing Sys- tems, 17:11244--14444 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.0000 2023.
Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju. Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:24/2./3877 2024.
Zhenyu Wu, Yuheng Zhou, Xiuwei Xu, Ziwei Wang, and Haibin Yan. Momanipvla: Transferring vision-language-actin action models for general mobile manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1714-1723, 2025.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow the cross-domain manipulation interface. In 8th Annual Conference on Robot Learning, 2024.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijic Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: foundation model for multimodal ai agents. arXiv preprint 1xx.:22522.13300 2025a.
Shuai Yang. Hao Li, Yilun Chen, Bin Wang. Yang Tian, Tai Wang. Hanqing Wang, Feng Zhao, Yiyi Liao, and Jiangmiao Pang. Instructvla: Vision-language-action action instruction tuning from under- standing to manipulation. arXiv preprint arXiv:2507. 17520, 2025b.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 2023.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A lsiounlanguaae model for spatial affordance prediction for robotics. arXiv preprint (tfi::2406.1022.. 2024.
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thoughh reasoning, arXiv preprint arXiv:2407.08692 2024.
Jesse Zhang, Karl Pertsch, Jiahui Zhang, and Joseph J Lim. Sprint: Scalable policy pre-training via language instruction relabeling. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 9168-9175. IEFF, 2024.
15

---

Published as a conference paper at ICLR 2026
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang. Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual th-ougt reasoning for vision- language-action models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1702-1713, 2025.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint arXiv:2412.10345, 2024.
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision- lanounlanguage models for robotics. arXiv preprint arXiv:2506.04:: 2025a.
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with siioa-aanguage-ccttonn model. arXiv preprint arXiv:2502. 14420, 2025b.
16

---

Published as a conference paper at ICLR 2026
A
APPENDIX
CONTENTS
A.I
Real-World Experiments for Executor
18
A.I.I
Experimental Setting
18
A. 1.2
Additional Real-world ID and OOD) Validation
19
A.1.3
More results and Visualization
19
A.2
Additional Ablation Experiments for Robolnter-VLA
22
A.2.1
Inference Time Analysis.
22
A.2.2
Experiment Results for Data Scaling Law
22
A,2.3
Ablation for Designs and Intermediate Representations Types of F-CoT
23
A.2.4
Detailed Results of Planner Temporal Robolnter- VQA
23
A.3
Cross-plations Validation
25
A.3.1
Open-loop Cross platsfom Evaluation
25
A.3.2
Close-loop Evaluation on SimplerEny
25
A.3.3
Close-loop Evaluation on Real-world Widow Robot
26
Additional Details Training and Evaluation,
27
A.4.1
Training Details of Planner and Executor
27
A.4.2
VQA Benchmarks
27
A.4.3
Metric Details.
27
A.5
Case Study
................................
29
A.6
Annotation Workflow
31
A.6.1
Robolnter-Tool
A.6.2
Principle of Semantic Segmentation Annotation
32
A.6.3
Principle of Language Annotation
32
................................
A.6.4
Keypoint Alignment and Annotation Pipeline
36
A.6.5
Quality Control Pipeline
37
................................
A.6.6
Postprocessing
38
A.7
Dataset Overview
40
................................
A.7.I
Dataset Statistics
40
A.7.2
Visualization for Intermediate Representations
40
A.7.3
Details of VQA Dataset
45
A.7.4
CoT Data for RoboInter-VLA
67
A.8
Further Application, Limitations and LLM Usage Statement
68
17

---

Published as a conference paper at ICLR 2026
A.1 REAL-WORLD EXPERIMENTS FOR EXECUTOR
A.1.1 EXPERIMENTAL SETTING
All Experiments are conducted in few-shot TableTop evaluation with real-world Franka Research 3 arm equipped with a Robotiq-2f-85 gripper. Controls are in the mode of delta end-effector mo- tions. Observation input comprises static third-person camera and wrist-view camera, and no proprioceptive state provided. Inference runs on an online A800 cluster and communicates with a local PC for robot control. To accommodate network latency, the control loop is limited to lower than 10 Hz, and demonstrations are collected using a SpaceMouse. We test: (1) Open VLA (Kim et al. Initialized from publicly released OXE (et al. 2023) (including Droid (Khazatsky et al.
2024):
2024 weights and extended to process both third-person and wrist-view images. (2) To (Black
al. 2024): Fine-tuned from the official JAX checkpoints of the Droid dataset. (3) Vanilla: A
from-scratch baseline without a pretrained Planner and without annotated intermediate representa- tions. Our model designs mainly follow Intern VLA-MI (Chen al. 2025b) and CogACT (Li al. 2024b). (4) Meoolnter-IC-E22 Initialized from pretrained Planner weights and further finetuned on In-Distribution (ID) data. (5) Robolnter-ECC EC----E-EE---EE Initialized from the pretrained Planner, and action and CoT generation are jointly optimised during finetuning, with a 1:1 ratio of our annotated data and collected data. At inference time, we utilize a shorter CoT (only subtask, affordance box, and gripper box), well as caching mechanism that stores slowly varying components (e.g., subtask, affordance box) of the CoT sequence, and refreshes them every 30 control steps. (6) RoboInter- Modular: We use 1:1 mixture of annotated and collected data to guide the learning of the Planner and Executor, respectively. The CoT sequence follows the same design as Robolnter- EC--EEEECEEEEE-- while the Planner and Executor operate through an asynchronous inference process. We design four long- horizon, contact-rich tasks:
Object Collecting. Sequentially place three pens from cluttered tabletop into cup, requiring the precise picking and placing: we collect 100 demonstrations. -utttr-distiibution (OOD) tests are to replace the pens with novel objects, alter spatial layouts, and substitute the container.
Cup Stacking. Stack cups from left right across three 22cm 22em zones, requiring precise spatial grounding ability; we collect 150 demonstrations. OOD tests include novel objects, OOD positions, and continuous stacking evaluate the ability of re-planning.
Towel Folding. Fold two towels in sequence and then stack them, evaluating the ability to ma- nipulate deformable objects; we collect 64 demonstrations. OOD tests vary in the towel category, material, and background, evaluating the robustness.
Clutter Cleaning. Clean all items from the board with a brush, and the task demands sustained operation and spatial perception; we collect 150 demonstrations. OOD tests introduce continuous streams of novel objects and random disturbances, evaluating persistent cleaning process.
Table 6: Real-world closed-loop performance. We report success rates for four tasks under ID/OOD settings and the ID-000 performance drop.
Model
Objects Collect.
Cups Stack.
Towels Fold.
Clutters Clean.
ID ID-:00 Drop
IDOOOD
IDIOOD
IDFOOD
IDIOOD
OpenVLA
53.3 /2000
/3.33
26.7
33.3
21.7
To
/467
85.0/53.3
53.3/404
46.7/40.0
18.3
Vanilla
/3.33
80.0 /6.7
46.7 20.0
66.7
26.7
IC-E2E
/3.3
86.7 60.0
60.0/46.7
73.3/73.3
18.4
EC-E22
73.3 60.0
80.0 73.3
46.7 /
73.3 66.7
8,3
Modular
73.7
53.3/40.0
73.3
IC-E2E performs bet- ter than other variances across the ID settings of most tasks. EC-E2E exceeds IC-E2E in Object Collecting (60.0% vs. 53.3%) and Cup Stacking (73.3% vs. 60.0%) and achieves a higher average success rate (60.0% vs. 58.3%). The ID-to-OOD drop is only 8.3% for EC-E2E, whereas IC-E2E declines by 19%. The Modular variant achieves strong real-world performance and competitive -itt---iiittiittionn (OOD) generalization. Its slightly larger ID ID-O0D drop compared to EC-E2E indicates that the asynchronous two-module design is somewhat more sensitive to distribution shift, yet still substantially benefits from our intermediate representation.
Detailed results of closed-loop real-world evaluation are shown Table 6
18

---

Published as a conference paper at ICLR 2026
Yerdte
KC-E2
16.
053
$33
26
26-1
inserting 1i--Distbbutioe-1)
insertnng (tt-Distiiuuiion2)
Object sorling (Qul-o-DDtriuuuu-))
sorting Ofu-o-DDstriuuliiee
Oal-a and baise lecterubaj
Figure 6: Additional real-world ID and OOD validation. (Left) Tool Inserting: precision ID task requiring accurate contact handling and slot alignment. (Right) Object Sorting: an OOD task that tests tnguage--giiee generalization using novel objects and containers.
A.1.2 ADDITIONAL REAL-WORLD ID AND OOD VALIDATION
To further disentangle in-distribution (ID) and out-of -isfr-diittiittionn (OOD) generalization, we intro- duce two additional real-world tasks that emphasize different aspects of control and reasoning:
Tool Inserting (ID-oriented). A precision-controlled manipulation task that evaluates the model's ability to fit fine-grained in-distribution actions. The robot must pull metal tool out of 1.5 cm x 1.5 cm slot and re-insert into another slot of the same size. Two ID variants with different initial positions and different training demonstrations used, each requiring accurate contact handling.
Object Sorting (OOD-oriented). A generalization task in which the robot must place objects into their corresponding target containers. Training demonstrations include only red, green, and blue objects and cups with simple pick-and-plat motions. The OOD setting introduces novel objects and cups with novel colors, shapes, or types, assessing whether models can generalize sorting behaviors according to language instructions beyond the training distribution.
The results in Figw complementary strengths of the two variants: EC-E2E achieves stronger OOD performance owing to its explicit reasoning, whereas IC-E2E exhibits superior ID robustness. Overall, both variants outperform the Vanilla and To baselines, demonstrating that intermediate representations significantly benefit both ID precision and OOD generalization.
1.1.3
MORE RESULTS AND VISUALIZATION
We provide some visualizations the real-world experimental process in Figures 7 and
19

---

Published as a conference paper at ICLR 2026
Rold the clothes
Fold chechs (genesalize:
new table texture)
Fold clothes (generalizee to new colort
Figure 7: Real-world evaluation of folding deformable cloths.
Stock the cups. (in-domain)
Stock the cups. (position generalization)
Stack the cups. (generalize more cups)
Put the cube into the cup, then stack the cups. (generalize to novel objects)
Figure 8: Real-world generalization evaluation of stacking three cups. Robolnter- Vobolnter-VLA general-
izes novel positions, four cups, unseen objects, and novel instructions.
20

---

Published as a conference paper at ICLR 2026
pens cup, Sollowing order red, blue, and black.
Cleam board
Clean the boart (generalize long-hortzon task with human Interaction)
Figure 9: Real-world evaluation. RoboInter-VLA demonstrates precise action generation (e.g., grasping a pen from the table while avoiding collision) and long-horizon capabilities, such as con- tinuously cleaning the board.
21

---

Published as a conference paper at ICLR 2026
A.2 ADDITIONAL ABLATION EXPERIMENTS FOR ROBOINTER- VLA
A.2.1 INFERENCE TIME ANALYSIS.
We report the inference-time characteristies of Robolnter-Vi VLA in this section. For open-loop eval- uation, our goal is to measure the accuracy of action prediction under comparable model sizes. As shown in Table 7 explicit reasoning significantly improves open-loop accuracy for both VLA- OS (Gao et al., 2025a) and Robolnter-VLA, but increases latency due to autoregressive CoT gen- eration. For real-world deployment, we apply practical acceleration strategies, including textual caching and chunked execution for EC-E2E, and asynchronous dual-frequency execution for the Modular variant. Table shows that these strategies substantially reduce latency while preserv- ing strong ID) and OOD performance. 'he general trend confirms that explicit reasoning enhances robustness at the cost of slower inference, motivating future work on more efficient execution.
Table 7: Open-loop inference latency comparison. Explicit reasoning improves accuracy but increases latency due to autoregressive generation.
Model
Size
Reasoning Tokens
Lateney (5)
OLS
VLA-08 (Vanilla)
0.1647
0.3035
VLA-OS-I-Explici
3B
64
2.2274
0.3290
Vanilla
0.3086
Robolneer-IC E2E
3B
64
0.1309
0,3218
2.2187
0.3340
Nobolnter-Modulan
3B43B
64
2.3594
0,3543
Table 8: Real-world inference speed with acceleration strategies. robustness but incurs higher latency.
Explicit reasoning increases
Model
Speed (Hx)
Latency (ms)
Score
Score
OpenVI.A (twrist)
3.8
263.2
45.0
23.3
Vanilla
7.64
130.9
65.0
38.3
RoboInter-IC-E2E
7.64
130.9
76.7
58.3
Robolnter-BC (sacheschunk)
RoboInter-Modular (async)
6.92
144.5
73.3
60.0
A.2.2 EXPERIMENT RESULTS FOR DATA SCALING LAW
Experiments for Robolnter-VOL VQA scaling law. As shown in Table Table 10 and Table [1 we conduct data sampling rate experiments on RoboInter-VOL VQA with QwenVL2..53B, varying the proportion of our data from 10% to 100%. The training consists of general VQA data combined with the corresponding subset of our annotated data. The results clearly show that the model follows scaling-law trend with the dataset size increases.
Table 9: The scaling low with data of all visual generation VQA data.
Data Sumpling Rute( %)
Object Grounding
Grasp Affordance
Placement Affordance
Gripper Detection
Truce Generation
10%
68. 71.9%
22.9% 35.0%
42.0%
46.1%
442
100%
72.8%
53.2%
61.5%
352
Experiments for F-CoT data scaling law. As shown in Tab. 12 we vary the proportion of paired F- CoT and action samples to study how data scale influences open-loop performance. We compare two As the sampling ratio increases, both models improve; however, Oracle+Executor exhibits larger gain (0.2347 0.2705) than Robolnter-Te Modular (0.2109 - 0.2332). This suggests that, with more data, Executor directly conditioned on oracle intermediate signals benefits more readily, while the VLM Planner improves slower rate.
Me-Modular. Ro-oonter-T--MModdlllr settings: Oracle+ Executor and Robolnter-
22

---

Published as a conference paper at ICLR 2026
Table 10: The scaling low with data of all visual understanding tasks.
T/F
Multiple Choice
Data Sampling Rate( %
Contact Decide
Grasp Pose
Object Grounding
Thace
Trace Dir
Trace Lang
10%
69.9%
30.6%:
74.8%
43.6%
53.3%
65.8%
50%
82.9%
34.4%
77.9%
63.9 %
73.6%
77.1%
Table 11: The scaling low with data of all planning tasks.
Planning
Data Sampling
Multiple Choice
T/F Planning
10%
63.5%
66.0%
$4.8
100%
72.1%
75.1%
62.1
Table 12: F-CoT data scaling law between Oracle+Executor and Robolnter-Te-Moduus across varying data sampling rates.
Data Sampling Rate (%)
Oracle-Executor
Robointen-Te-Moohhlar
OLS@/0.03
OLS@0.01
OLS$00.3
OLS@0.01
10%
0.2347
0,0236
0.2109
0,0249
30%
0.2590
0.0280
0.2225
0.0285
50%
0.2686
0,0400
0.2372
0.0429
100%
0,2705
0,0587
0,2332
0.0584
A.2.3 ABLATION FOR DESIGNS AND INTERMEDIATE REPRESENTATIONS TYPES OF F-COT
Ablation for designs. F-CoT serves flexible carrier that composes multiple intermediate rep- resentations from Robolnter-Datt We evaluate two forms: (1) Textual F-CoT: Intermediate repre- sentations serialized into autoregressive text tokens, used in RoboInter- Te-Moorra--Mdllarrr (2) Visual F-CoT: Intermediate representations are drawn as visual prompts on additional input images, used in Meci-nter-I---oieee As shown in Table 4 Textual F-CoT (RoboInter-Tie Teeinnee-----Moooe Te-Modular) consistently performs best among learning- branning-based models, indicating that structured textual reasoning may more stable and expressive interface for composing heterogeneous intermediate representations.
Ablation for intermediate representation types. As shown in Table 5 we have ablated the con- tribution of each F-CoT component using the Oracle+Executor setting. We further ablate how each intermediate representation individually affects performance in Table I3 The trend is consistent: coarse-graie grassed representations (Subtask, Primitive skill) give modest gains, whereas spatially pre- cise cues (Object Box, Affordance) provide much greater improvements.
A.2.4 DETAILED RESULTS OF PLANNER ON TEMPORAL. ROBOINTER VQA
We present more detailed results of the Robolnter VQA benchmark in Table 14 Table [5] Table 16
Table 17 including entries that omitted only summarized by average scores in Table
23

---

Published as a conference paper at ICLR 2026
Table 13: Ablation results of single intermediate representations. We report OLS under four thresholds and the mean value (mOLS).
Method
OLS
mOLS
@0.10
@0.05
@0.03
@0.01
Vanilla
0.6793
0.3608
0.1753
0.0189
0.3086
Subtask
Primitive skill
0.6925
0.3658
0.13
0.0156
0.3133
Object Box
0.7001
0.3793
0.1879
0.0202
0.3218
Affordance
0.7054
0.3958
0.2012
0.0263
0.3322
Table 14: Performance comparisons on temporal tasks of the Mobolnter-VQA benchmark. indicates the model could not be accurately evaluated for that metric.
Generation
Multiple Choice
T/F1
Model Name
Trace-Easy_
Harde
Tracet
Trace Dirt
Traj Lang
Contact
V...VL2..533
2752
26.5%
35.6%%
50.5%
QwenV1.2.5.78
1553
1850
26.4%
35.7%
54.8%
54.3%
Intern V1.31V
16.5%
28.2%
42.1%
Inter VL332B
1216
19.1%
30.8%
55.7%
53.5%
IntemVLL--888
656
1414
16.2%
29.6%
56.1%
58.7%
Ilava OV:7B
27.8%
25.9%
59.3%
57.8%
Robobrain-3BB
592
25.6%
35.3%
52.0%%
Robobrain 7B 2.0
540
541
30.7%
24.4%
33.4%
50.5%
GPTio-mini
1091
2381
9.49%
21.4%
$4.3%
53.6%
Gemini-2.5-flash
44.4%
38.7%
63.2%
65.5%
RoboInter- -eaeerr--nnnRBB
315
350
73.9%
81.5%
1%%
72.5%
Robolnter-Qsenn-1e
302
344
81.1%
80.4%
34.3%
74.4%
291
306
83.6%
83.5%
78.5%
76.3%
Table 15: Detailed performance on Multiple Choice of task planning (ACC%1).
Qwen V1.2.5-3B
Intern
Intern VI.32BB
Intern VI.3-BB
Robo brain-3B
Rebo brain-7B
Llava
RoboInter RoboInter RoboInter
Task
V1.255-BB
VI.3-IB
OV-7B Qwen-31 Qwen-7B LlavaOV-7B
Scene Understanding
38.3
55.5
28.8
40.8
70.0
41.6
59.0
53.0
83.2
86.2
83.9
Past Multi-takk Sel.
93.8
94.2
88.2
90.3
93.6
70.6
81.0
70.4
94.2
96.4
90.4
Past Primitive Sel.
64.3
65.0
53,0
68.8
48.4
53.2
56.0
84.2
88.9
81.7
Future Multi-task Sel.
83.4
85.9
82.0
81.9
84.2
52.2
68.1
25.2
90.5
93.6
90.1
Primitivv Sel.
Temporal Undersiand
23.1
23.7
26.4
26.1
52.2
31.4
45.2
25.9
42.9
50.7
60.1
Averuge (weighted)
60.0
64.5
54,9
59.3
71.5
48.2
57.8
44.9
82.2
86.5
81,8
Table 16: Detailed performance T/F (Decide) Tasks of task planning (ACC%
Task
Qwen VL2,5-3B
Qwen VL2.5-7B
Intern
Intern
Intern VL3-8B
Robo brain-3B
Robo
Llava Robolnter RoboInter RoboInter Qwen-3B Qweu-7B LlavaOv-7B
VL3-1B
VL3-2B
brain-7B
Success Nogative
96.7
68.1
0.0
Success Positive
14.3
11.4
46.4
37.2
50.3
98.4
100.0
46.2
89.1
91.6
85.7
Discrim. Allord. Neg.
92.4
97.0
64.5
67.8
71.2
1.5
0.2
75.4
93.4
95.9
95.2
Discrim, Afford. Pos.
34.4
26.3
58.3
63.1
67.5
99.2
100.0
76.2
94.9
95.6
92.3
Average (weighted)
46.4
Table 17: Detailed performance on Generation Tasks of task planning (Average results 1 of BLEU-1 to BLEU-4 Score).
Task
Qwen VL2.5-3B
VL2.5-7B
Qwen
Intern VL3-1B
VL,32B
Intern
Intern VL3-8B
brain-3B
Robo
Robo brain-7B
RoboInter RoboInter RoboInter OV-7B Qwen-3B Qwen-7B LlavaOV-7B
Llava
Planning Task
28.4
25.4
13.2
8.1
6.8
17.7
11.8
61.0
64.7
61.1
Past Description
8,5
14.4
3.3
2.0
2,8
6.5
6.9
0.8
46.1
51.7
48.9
Planning Context
Planning Remaining
20.5
24.4
13.4
12.5
13.2
19.3
22.3
19.5
73.2
73.5
74.0
Generative Allordance
6.3
12.9
4.0
3.0
2.0
5.6
5.6
1.1
43.3
45.8
45.3
Future Prediction
22.3
25.4
13.8
7.8
10.0
18.8
16.1
5.7
69.0
68.1
71.2
Average (weighted)
24

---

Published as a conference paper at ICLR 2026
A.3 CROSS-PLATFORE VALIDATION
A.3.1 OPEN-LOOP PLATFOR EVALUATION
Our dataset spans 570 scenes across multiple robotic embodiments, forming hybrid collec- tion that is both poss-platform and cross-scene. Because both the training and validation splits include diverse platforms, the open-loop evaluation naturally serves as cross-platform testing. We report platform-specific open-loop performance in Table [8] including five distinct embodiments. The results show that all RoboInterV VLA variants consistently outperform the vanilla baseline across platforms, with the Modular configuration achieving the best overall accuracy among learned mod- els. The Oracle+Executor serves the upper bound.
Table 18: Cross-platfone open-loop evaluation across different robot embodiments.
Model
Franka-Panda
URS+Robotiq
Flexiv+AG995
Knka-Robotig
Frankn+ Rookkig
OL.S
0.6756
0.3608
Robolnter-IC IC-F2F
0.7149
0.9929
0,7786
0.7564
0.3810
0.72476
Roboloter-BC-EB
0.8098
0.9876
0.7723
0.7748
0.3930
0.7475
Robolnier-Modula
0,8572
0.9962
0.7743
0,7609
0.4133
0.76238
Oracle+Execukar
0.8828
0.9942
0.7809
0.7848
0.4640
0.78134
A.3.2 CLOSE-LOOP EVALUATION ON SIMPLERENV
Evaluation setup in SimplerEnv. We conduct a close-loop experiment within SimplerEnv LLi al. 2024d), a benchmark to evaluate models across various tasks with the WidowX Robot (WR) and the Google Robot (GR). The GR environment includes two settings, Visual Matching (VM) and Variant Aggregation (VA). WidowX Robot environment only includes the VM settings. SimplerEnv covers more than 2k trails across different scenarios, objects, and tasks. We report the average success rate of each task. RT-1-X (et al. 2023), RT-2-X (et al., (2023), and Octo-Based (Octo Model Team et al. [024) are early baselines, Ope (im et al. 2024), CogACT (Li et al.) 2024b), and Magma (Yang et al., 2025a) are trained on subsets of OXE. To and To- are VLAs with additional robot state input. RoboVLMs (Li al.) 2024c). SpatialVLA (Qu et al. 2025). and
Trace VLA (Zheng et al. 2024) are all trained on the Fractal and Bridge- V2 datasets.
Table 19:
Performance comparison on SimplerEnv. The experiments are conducted across 12 tasks, including both Visual Matching and Visual Aggregation. The parameter size of LLM are shown behind the model name. % of success rate is omitted.
Google Robot
Widow X Robot
OpensClose Draver
Pat in Drawer
Pick Coke Can
Move Near
Put Spuon
Put Caroet
Stack Blocks
Put
Methods
Eggnlont
Avg
VM
VA
VA
VM
VM
VA
VM
VA
VM
30.2
0.0
4.2
20.6
77.9
Octo-Ras
Tram
1.1
0.0
4.2
3.1
1.2
15.8
12.5
0.0
41.7
17.5
RoboVLMs (28)
Liet 20246
44.9
10.3
27.8
0.0
76.3
50.7
79.0
62.5
$7.0
30.9
30
37.5
0.0
$3.3
42.7
43.5
Ugeret
2005
54.6
39.2
0.0
6,3
79.3
78.7
90.0
$3.0
$6.0
51.8
20.8
37.5
41.7
$3.3
45.8
51.2
(38) [Back
38.3
25.5
20.5
72.7
65.3
63.7
55.3
46.3
29.1
0,0
16.8
62.5
27.1
43.0
2025
29.1
21.9
10.5
OpenVLA
2.9
55.0
8.3
24.7
(7B)
71.8
28,3
50.9
46.6
91.3
89.6
$5.0
80.8
74.8
61.3
75.0
50,0
16.7
79.2
55.2
63.8
TIxcVLA (70
2004
63.1
61.6
11.1
12.5
45.0
64.3
63.8
6016
45.5
49.8
12.5
16.6
16.6
65.0
27.3
41.1
Magmo (88)
58.9
59.0
24.0
75.0
68.6
53.0
78.5
48.5
57.5
37.5
29.3
30.8
91.7
44.8
50.4
48.6
56.0
61.8
12.5
46.9
Rebolntto-CC-F33..
71.3
54.9
Results. shown in Table 19 we initialize our RoboInter-IC-ED2 model with the pretrained VLM of Planner and perform implicitly conditioned end-to-end training on Fractal and Bridge data. Because RobeInter-Data does not include action annotations for WidowX or Google robots, this constitutes strictly cross-embodiment evaluation. On SimplerEnv, our minimal Vanilla design out- performs common baselines (wys To -A-FT,, though it is slightly below CogACT (61.8 vs. 63.8). With stronger embodied VLM initialization. Robolnter-IC-E22 improves to 65.7. surpassing Co- gACT (65.7 vs. 63.8). These results validate the implicit planner-then-ee exeeeecutrr executor and highlight the transferability of our data.
25

---

Published as a conference paper at ICLR 2026
Pi-0
Vanilla
Robsinter- IC-FFE
60.0
60.4
26.1
20.0
13.3
Pick the spoon
Pick yellow Pikachu
Put Cup Plate
Put Pikachu Plate
Figure 10: Real-world WidowX Robot results.
A.3.3 CLOSE-LOOP EVALUATION ON REAL-WORLD WIDOWX ROBOT
To further assess real-world generalization capabilities, we conduct closed-loop zero-shot experi- ments on the WidowXX225 robotic arm. We evaluate three models: To (Black et al.) (2024), Vanilla baseline, and our Sobolnter-IC-E2E All models are pretrained on the Bridge V2 without further post-training finetuning prior to deployment, and all experiments are executed using the same real-world setup. Our evaluation focuses on a kitchen environment, where we design four manipu- lation tasks, each executed 15 times:
Pick the Spoon: The robot must grasp a metal spoon placed at arbitrary positions on kitchen board.
Pick Yellow Pikachu: This serves out-of-distribution (OOD) test, similar objects
(Pikachu) do not appear in the Bridge dataset. The small toy increases the difficulty.
Put Cup on Plate: A pick-and-place task where the robot must pick up cup and accu- rately place it at the center of a plate.
Put Pikachu on Plate: An OOD variant of the previous task, replacing the cup with the Pikachu toy, making the placement more challenging.
The experimental setup is illustrated in Figure 10 Across all tasks, Robolnter-IC-EEE model demonstrates the strongest real-world performance, benefiting from its more powerful pretrained representation and showing superior generalization in closed-loop control.
26

---

Published as a conference paper at ICLR 2026
A.4 ADDITIONAL DETAILS OF TRAINING AND EVALUATION.
A.4.1 TRAINING DETAILS OF PLANNER AND EXECUTOR
Planner training hyperparameters. The RobohntrVLLMQWeenL2222 7B) Planner trained for one epoch with global batch size of 128 and a per-device batch size of 4, without gradient accumu- lation. The base learning rate, as well as the learning rates for the vision tower and the multimodal projector, are all set to x 10 Training uses BF16 mixed precision, maximum gradient norm of 1.0, zero weight decay, and a warmup ratio of 0.03. More details will be released in our codebase.
Executor training hyperparameters. The Robolnter-IC-ED. Executor is trained with global batch size of 128 and a per-device batch size of The base learning rate X 10-6, with no warmup and weight decay. Action head, the LLM backbone, and the vision backbone all remain trainable. The Q-Former operates between layers 36 and 37, predicting actions of 7 dimensions over future action window of 15 steps. More details will be released in our codebase.
VLM training data. We partially follow the basic VLM training recipe of InternVL. (Chen et al,, 2024b), and shown in Figure[[] to ensure that the Planner acquires both strong general perceptual understanding and embodied prior knowledge, we carefully select broad range of VQA data for training.
Undended Undeebaann
Enboch Grandin
Stadeliie Male
Planner Training Data
Geneal Goanchn
Figure 11: Training data distribution for the Planner. We cover broad range of training data, including temporal/spatial understandingggeneral embodied reasoning, and spatial grounding.
A.4.2 VQA BENCHMARKS
Recent benchmarks for spatial referring and multimodal reasoning can be broadly categorized into embodied and non-embodied settings. Embodied benchmarks such Where2Place (Zhou al. 2025a), RoboRefIt (Lu et al. 2023), RoboVQA (Sermanet et al., 2024), and RefSpatial-Bench (Lu al. 2023) emphasize 3a-aware grounding, ambiguous object reasoning. and multi-step spatial un- derstanding in realistic robotic contexts, thereby capturing the complexities of embodied interaction. In contrast, non-embodied benchmarks, which include RefCOCO/+/g (Lin et al. 2014) for referring expression comprehension, TexiVQA (Singh et al.) 2019), COCO (Lin et al. [01 4). DCRBench (Liu ctt al. 2024b). MME (Fu et al. 2024a). MMVet (Yu et al. (023), and POPE (Li et al. 2023b). have driven advances in multimodal reasoning but remain limited in sequential and interactive dimensions essential for robotic deployment.
A.4.3 METRIC DETAILS.
Open-loop evaluation protocol Our annotated corpus spans more than 500 distinct scenarios and contains roughly 200k demonstrations for large-scale robotic training. However, as emphasized by HPT [Wang et al.) 2024a), comprehensive real-world validation across all scenarios is infeasible, and there is practical approach for evaluating large-scale pretraining. Our primary objective is to quantify how intermediate representations benefit downstream action generation, so we adopt an open-loop evaluation protocol in which per-step actions assessed independently: given the vision and language inputs, the model predicts a candidate action that then compared with the ground- truth action. HPT compares validation loss; however, for the diffusion- model, validation
27

---

Published as a conference paper at ICLR 2026
loss is sensitive to the per-step noise schedule. Following the discrete token- accuracy evaluation in OpenVLA (Kim et al. 2024), we utilize an Open-Loop Score (OLS) for continuous action chunks:
OLS 1 M
(1a 101 < 2 0
where M is the number of evaluation transitions, T is the length of each action chunking. 7 denotes the error threshold, E [0. 1] is the chunking-level tolerance (default 1). For each frame, we fully denoise Ttsppp action chunk. If the mean stepwise otherwise counts as 0 through (*). OLS is the average value over all transitions. The validation set independent of all training data and contains over 100K transitions, ensuring statistical stability.
stays below 0, the episode counts 1,
28

---

Published as a conference paper at ICLR 2026
A.5 CASE STUDY
We present qualitative examples in Figure 12 and Figure 13 Although RoboBrain, which trained on embodied datasets, achieves performance par with Qwen2.5VL, this observation highlights the inherent limitations of the embodied data it relies upon. In contrast, RoboInter (RoboInterVLM Qwen2.5VL 7B) consistently demonstrates superior performance across different settings. These representational reasoning and informed decision-man makinng, thereby serving as an effective tool for data selection. Moreover, as shown in Figure [4] when evaluated on our self-collected zero-shot dataset, the RobolnterVI.M, model continues to exhibit strong generalization ability, maintaining robustness even when encoun- tering entirely novel objects.
results indicate that our planner model is capable of intermediate-
Robobmin 2.0
Robobrrain
Qwen2.5VL 78:
Qwen2.5VL DB:
Robainter
The robot task le put the morker in the pot, and now you are performing We ofter
The robot Lask 'move the sail cellar to performing subtask "pick the selt callar on the table: We offer set images containing gi indicates the start of each trajectory, and red indicates its end-these trajectories
the table', and now
subtask 'pick the containing gradient-calored trajectories. Green indicates the start nach trajectory, and red indicates its end-these trajectories represent potential routes depicts the right trajector?? The available choice is (A, B, C, DJ.Plesse select option only.
images
trajectories, Green
gripper
subtask.
image
the potential routes for the gripper finish the subtask. Which image depicts right trajectory? The evailable choices are (A, B, C, D). Please select one option only
Qwen2.5VL 78:
Qwen2.5VL 7B:
Robolnter 78:
Robointer 7B:
The robot task is 'move the blue towel from the left the right,, and now performing subtask 'pick blue towel from left side table!. Among the images presant, radiant-cnloroo trojectorics are inclucidd green stands for trajectory's start, and red for its end. These trajectories ropresant potantial to finish the subtasic Which image has the correct correct trajectory? The available choices [A, Please select option
The robot lask 'out green bag of snacks the sink', and now you performing subtask 'pick green bag of snacks from the counter: We provide some images gradient-cclored trajectories. Green represents the start of each trajectory, red represents its end, and these trajectories
required for the gripper
the paths the accomplish the subtask. Which image shows the right trajectory? The available choices are [A, B, C, D). Please select one option only
Figure 12: Case study on RoboInter- VQA.
29

---

Published as a conference paper at ICLR 2026
Robobrain 2.0 7B:
Robobbrsin 2.0 7B:
Qwen2.5VL 78:
Qwen2.5VL 7B:
The robot taok "pick up the white cloth and put the tablei. We represent the objects involved robotic erm's interactions, which image accurately depicts bounding box of the object? The available choicas are [[A, C, DJJ. select option only.
robot taok 'remove the yellow object from pot and place it on the table! We present images containing purple bowes, which robatic maybe image corractly shows the bounding hox of the objact? The available choices
images
Please select option only.
Robobrain 2.0 78:
Robobrein 2.0 7B:A
Qwen2.55VL
Qwen2.5VL
Robolnter 7B:
Roboliner 7B:
another' some images that foature orange fork-liko gripper potterns; these grpppr needs to adoot pick up an object. Which the imagees below contains correct greeping pose? The available choice [A, B, C, D]. Please select one option only..
object
We with orange fork-like gripper patierns. These patterns correspond potential poses the gripper may adopt grabs object-so which image: has right grosping pose? Pleese select one aption only.
"play
images
patterns potential
cholces are (A, DJ.
Figure 13: Case study Robolnter- VQA.
<inage> \n The robot task is "pickup the center bottle', and you performing subtask 'grasp the left bottle' In'n Leveraging the abserved scene, return the nonipulated object's current bounding box. In Return in JS0N format: The box should formatted as [[x1, y1], [x2, y2]], which means the Left top points and right down points.
Answer: [0.5, 0.31, 8.69, 0.46]
is "pickup and now you are performing subtosk 'grasp the left bottle' Leveroging the observed scene, expected grioper-object contact region. An Return J50N format: The region should be formatted y2]], representing the points of the bounding box of the gripper at the noment of contact the object. Answer: [e.5, 8.31, 8.69, 0,46]
<image>
bottle',
the
top points [[x1, yl], [x2, right down
Figure 14: Zero-shot evaluation object grounding and grasp affordance tasks.
30

---

Published as a conference paper at ICLR 2026
A.6 ANNOTATION WORKFLOW
A.6.1 ROBOINTER-TOOL
To enable efficient and consistent annotation for large-scale, high-precision robotic manipulation datasets, we develop Robolnter-Tool, PyQr5-based multimodal annotation platform. The tool sup- ports muulti-graullann annotations the video, clip, and frame levels, including language descrip- tions, gripper keypoint labeling, segmentation mask generation, and keyframe selection. Its mod- ular design integrates annotation, visualization, and editing, allowing annotators to perform rapid validation and iterative refinement. Robolnter-Tool consists of two main components: semantic segmentation annotation interface and language annotation interface, with built-in cloud-based asynchronous quality verification.
Semantic Segmentation Annotation Interface. This interface focuses on frame-level precision, automated segmentation assistance, and multi-object flexibility (Figure 15). It contains three main components: (1)Video annotation and visualization area. The system supports frame-by-frame playback and review, enabling annotators preciscly identify keyframes of robot-object interac- tion. By clicking on video frames, annotators can label gripper keypoints and provide them to the SAM2 (Ravi et al., 2024), which then automatically generates segmentation masks of manipulated objects. In addition, annotators may specify contact frames and bind annotated objects to corre- sponding clips to facilitate consistent clip-level annotations. (2) Tool control area. The system pro- vides unidirectional and bidirectional SAM-based segmentation modes, multi-object management via object IDs, and flexible editing operations such undo, deletion, and object-specific modifica- tion. (3) Shortcut and progress management area. Keyboard shortcuts support frame navigation, playback control, contact-framt frame setting, and object binding. Real-time statistics and progress indi- cators display annotation status. Save-and-load functionality enables interruption and resumption for scale workflows.
Bay points
Polcs 54M
Obpech-Lvve Video
Contact Pranee
Figure 15: Semantic Segmentation Annotation Interface.
Language Annotation Interface. It supports both video-level and clip-level natural language de- scriptions of robotic manipulation videos. This interface is designed to enable multi-granularity annotations, ranging from global video descriptions fine-grained primitive skill descriptions. As illustrated in Figure T6] the interface organized into four main components: (1) Video visualiza- tion area. Similar the semantic segmentation annotation interface, it supports frame-by-frane playback and provides a keyframe bar for previewing designated keyframess (2) Language annota- tion area. Annotators can provide and modify global video-level language descriptions that provide high-level summaries of entire videos. For clip-level annotation, annotators define a video segment by marking its start and end frames and then select the corresponding primitive skills from pre- defined template set. These templates are further parameterized by object and location, ensuring structured and consistent action descriptions. (3) Shortcut and progress management area. The interface provides convenient operations, including frame navigation, playback control, annotation deletion, and language modification via keyboard shortcuts, ensuring flexibility and efficiency dur- ing annotation. (4) Pre/annotation results area. The interface presents available pre-annotations
31

---

Published as a conference paper at ICLR 2026
Atom Action Subtuck templade choose
Giobat Video Language Ancotation
Video-lased Annotation
Hntpott movelt
Wide Deserittien growp peper
Select the
Raxt (deedl Reo0oon
Clip based Language Arnotation
Iull (st/eet) to jooases
(Bpphwtscbbeetoddeeee -ste leqect
Pos (stect)) (ecatio))
Atvre action: Brase
deocription: table
Pleae contim lang anoo
(roact) locatoni
Inerl (stacc) into locaten
lsteatt (ecatien)
Prestonstation tecats
Video Olip, Longuoge Antotiation
(cbwet)) jicattont
Operato (séject) (scedoo))
Start Frame
Star: Flara
EndiFram
Plaflass
Fome 170460
ARASSAAAI 163 Semaiinn
Beckapace Duletv
Trunns
bitan Addl lang
Di Neo Friine
Veeiyy ling anno
Figure 16: Language Annotation Interface.
current results for review, enabling annotators to validate and refine them. This functionality reduces redundant effort in large-scale annotation workflows while enhancing consistency and reusability.
A.6.2 PRINCIPLE OF SEMANTIC SEGMENTATION ANNOTATION
The semantic segmentation annotation process mainly encompasses three components: interacted object segmentation, contact frames identification, and gripper keypoints labeling. (1) Interacted Object Segmentation. Annotators specify prompt points in the area of manipulated objects, which serve as inputs to the SAM2. Based on these prompts, SAM2 produces pixel-level segmentation masks corresponding to the interacted objects. In scenarios involving multiple objects, annotators label each object sequentially according to the order of interaction, while the associated object IDs are automatically recorded, thereby ensuring precise segmentation and consistent tracking across teractions. (2) Contact Frames. Annotators identify contact frames, defined as the keyframes when the gripper first establishes physical contact with the target object. If the same object manipulated multiple times within an episode, annotators use combine function to bind different contact frames to that object, thereby preserving semantic continuity across disjoint interactions. G3)Gripper key- points. To facilitate downstream keypoint- basedd optimization, our tool provides functionality for gripper keypoint annotation. Annotators label a predefined set of keypoints on the gripper a frame-by-fin br-me manner, capturing its motion trajectory. Only keypoints visible under the current camera view are annotated, thereby offering fine-grained geometric constraints that support subse- quent 3D-to-2DD projection optimization. Together, these three levels of annotation (from pixel-level masks to clip-level contact frames and geometric-level keypoints) form a comprehensive semantic segmentation annotation pipeline.
A.6.3 PRINCIPLE OF LANGUAGE ANNOTATION
In the language annotation component, we establish a multi multi-granularity framework tailored for robotic manipulation videos, covering representations from global video semantics to fine-grained primitive skill. The annotation interface supports three complementary forms of annotation: (1) Global video-level descriptions. Annotators provide and refine high-level summaries of the entire task. To improve efficiency and reduce cognitive load, we first generate GPT-based pre-annotations, which are then inspected and corrected by human annotators. (2) Clip-level descriptions. Annota- tors segment each video into subtasks by marking start and end frames and selecting primitive skill categories from a predefined template library. Each template is parameterized by objects and loca- tions, ensuring structured, consistent, and semantically aligned descriptions. As the data from
32

---

Published as a conference paper at ICLR 2026
DROID and RH20T often contain brief pauses, additionally label the first and last non-static frames to improve temporal precision. (3) Primitive skills. Building upon this interface, we define 15 primitive skills the most clementary and semantically coherent units of robotic behavior. As shown in Table 20 primitive skill covering canonical manipulation behaviors such as Pick, Place, Push, etc. Importantly, primitive skill is not limited the single moment of gripper-object tact but also includes its preparation and completion. For example, a Pick starts when the gripper approaches the target object and ends when the object is securely held, while Place covers the movement of the object to the target location until release and gripper withdrawal. Simple motions such as repositioning or empty approaches are not treated as independent primitive skills but are merged into neighboring clips. Typically, the first annotated clip begins when the robot initiates pur- poseful motion, while the final clip extends until the end of the episode, unless the terminal sequence involves static or invalid states, or the gripper is absent. Through this principled segmentation pro- cess, all valid primitive skills within each video are exhaustively annotated, ensuring that the dataset achieves completeness in temporal coverage, semantic fidelity, and structural rigor.
33

---

Published as a conference paper at ICLR 2026
Table 20: Definition and template of 15 primitive skills.
Primitive Skill
Definition
Template
Move with Object
The gripper holds object and transports it from one location to another.
transfer [object] from [position] to [position]
The gripper approaches and securely picks up a specific object.
Pick
pick up [object] [position]
The gripper relcases and gently sets down an object designated location, ensuring sta- bility (distinct from throwing).
Place
place [object] [position]
The gripper tool applies downward force to actuate or press an item such as a button, switch, or block.
Press
press [object] [position]
The gripper or tool applies force to move object away along surface specified direction.
Push
push [object] [position]
The gripper or tool draws an object closer or along path, e.g. opening drawer, refrig- erator door, or zipper.
Pull
pull [object] [position]
The gripper twists turns object clock- wise or countrrclockwise, such as unscrew- ing a cap or rotating a box.
Twist
twist [object] (cbokwiis/ccomterelockssss
The gripper tilts container to transfer liq- uids granular materials into another con- tainer onto surface.
Pour
pour [ohject] [position]
The gripper manipulates flexible material (e.g., fabric, paper) to fold along one or more creases.
Fold
fold [object] [position]
The gripper moves object laterally across surface without lifting it, such sliding items on table.
Slide
slide [object] [position]
The gripper places an object precisely into an opening or slot, e.g.. inserting a block into rod connecting parts.
Insert
insert [object] [position]
The gripper repeatedly oscillates shakes an object achieve desired effect.
Shake
shake [object] [position]
The gripper or tool delivers quick strik- ing motion to an object, e.g., tapping with hammer or stick.
Strick
strike [object] [position]
The gripper propels object by releasing during motion, causing it to he cast dropped abruptly (distinct from placing).
Throw
throw [object] [position]
Miscellaneous manipulations not covered by the above categories, including minor ad- justments rare operations.
Other
manipulate [object] [position]
34

---

Published as a conference paper at ICLR 2026
ChatGPT-40 Prompt to produce preliminary references for language annotations
You are highly skilled in embudiment task planning. breaking down intricate and long-serm tokk into distinct primitive skills, All primitive skills are experssed follows Lhans of [ebjjcct ject. at an objoet, mank [pss_t.ion] as specitie location in the cavirommens. We have standardized the please stricty
apolicable Skills:
each prunitive
transfer [object] from [position to [positio)]: Move the abject
position
already
gripper.
pick up [object] [pesitios]] Move gripper to specitie position grab pick up an object
place [object] [position]: Place the object at designated position smoothly without sudden reletse (ditlerens from throw).
press (object] [position): Apply peessure or pees an object (buttons, switches, blocks, esc.).
5. push [ohject] [position] Push object, suitablee for moving it ce hindle in direction.
6. poll [sbject] [position]: Pull an object (oppositc of push), opening drowers, doors, zippers.
7. twist jobject] cocttrrooo
Twist rotate objects (bottle caps, boxes, etc.).
8. pour [obyect] [poution]: Pour liquid granular moterial from container into another container position.
fold [object] [position]: Fold bendable materials (cloth, puper,
slide [object] position): Slade objects without lifting them sweeping sliding surface).
insert [object] [position]: Insest object precise position (e-p. play, block).
(object) [pociion):
object
specific
jobyect] [position]; an object using le.g.
hammer))
throw [object] [position]: Throw an object to tirget or rolesse it subdenly (diffirrent from place).
manipulate [object] [position]: Manipulate adjust an object in situations not covered above.
Input Information:
Tisk Description (in lengunge)
RGB Image Linitaal observation)
Instructions:
Understand both the Task Description and the
Idemify and sequence primitive skills required.
Replace ject and [pus it ian] specific.
avoid vague like "desired location".
only
primitive
using
prinitive
only special cases.
exectly the given formet
Tip: You may first itemize the tusk related objects to help plan. Fxomple?
Task: Task:"pict Answer.
"pick rkk rxber choogl lat
From bot em dreser and place councer"; RGB Imager (omitted)
"pu13 the handle the bottonn draver to outxide the counter", bottom dravor", "pick up rxbac chocolato the drzwer "crancier the "place Lhe sabas choculate the counter"
the countor",
Task: Fopen: Jhe Iridge, pat
in the fridge, close the Ea Eridg RGB Image: (ominte)
Answr:
"pu11 the handle of the feidge to open the feidge", "picx up the tonstoes on the top at the desk", "cranete"
"place tonateas fron
tridge",
shelf "pess the dooz of the zridas the position of closing the fridge"
fridze",
mag", Image:
Anveer:
"pick up the contal nier on the top of she desk" "Eransie: conta from top the deek to the top nug "pour water container into meg"
Current Tack
Task Description:..
RGB Image: ace the following
The follows exartly form sx5agon and In additio::
The should formatied list incloding the leading and tralling
Ourput the plan again in Fnglish isame formut).
At the end the ourput. translate the Thak Description into English, formatted as <X0X>
35

---

Published as a conference paper at ICLR 2026
A.6.4 KEYPOINT ALIGNMENT AND ANNOTATION PIPELINE
To accurately capture and leverage gripper positional information both simulation and real-world settings, design systematic keypoint annotation and projection pipeline (Figure [7p. The pipeline unifies keypoint definition, EE Link offset calibration, and 3D-to-2D projection, thereby providing consistent and extensible annotation framework that generalizes across different gripper models and offers a reliable foundation for downstream application.
Recioies Adiiaton
Parrs madion
URDF
Keypoint links base frame
Kaypoint @ local frame
Kayp seppintt linka loeal frams
end ieflcctor link offset correction
Joint Angle
Human Annotation
Reprojection Error-based optinizzaiion
[6x,Ay,
20 Projection
repojjectoon
20 Koypolnts Projection
Figure 17: Keypoint Annotation Pipeline.
Keypoint Definition. We first define set of representative keypoints for each gripper type accord- ing to its geometric characteristics, such as fingertips and the central points of the inner surfaces of the jaws. The locations and numbers of keypoints for different grippers are illustrated Fig- T8. These keypoints are designed to capture the geometric constraints and functional features that are most critical when the gripper interacts with objects or the environment. The process of keypoint definition and annotation consists of two steps: (I) Computing keypoint coordinates on the mesh. We first merge gripper link meshes in MeshLab to align their spatial positions with the URDF. From the unified mesh, precise 3D vertex coordinates in the gripper base frame are extracted, and the desired keypoints are annotated on the corresponding links. This ensures geometric accu- racy and structural consistency with the real gripper. (2) Updating the URDF with keypoints. The computed 3D coordinates are added to the URDF additional links. Since these coordinates are defined in the base frame rather than local link frames, they cannot be directly driven by joint kinematics. To resolve this, apply coordinate transformation convert them into the parent link frames. The updated URDF then correctly propagates keypoint poses during simulation according joint configurations. (3) 3D-to-2D projection. To faithfully reproduce each recorded trajectory in simulation, the robot's joint angles and gripper open-close states are used to reconstruct the corre- sponding motions, from which the 3D coordinates of predefined keypoints are obtained. Leveraging camera intrinsics and extrinsics, 3D keypoints are projected onto the image plane to generate the 2D keypoint annotations.
EE Link Offset Calibration. In real robotic systems, a fixed structural offset exists between the gripper and the end-effector link. If the original URDF is used directly in simulation, the recon- structed and projected keypoints deviate significantly from the annotated ground truth. Since this strat-
offset is not directly observable during data collection, we adopt an annotation-optimizztiio egy that estimates and corrects it by minimizing reprojection error. The procedure mainly involves two steps: (1) 2D keypoint annotation. For each robot configuration, 15 episodes are randomly sampled and keypoints are manually annotated across all camera views, providing supervision for offset estimation. (2) Offset optimization. The BE-gripper offset is parameterized as 3D trans- lation vector and incorporated into the EE-gripper transformation. Given the recorded joint states, robot motions are replayed in simulation, and keypoints are projected onto the image plane using known camera intrinsics and extrinsics. The reprojection (computed as the Fuclidean distance
36

---

Published as a conference paper at ICLR 2026
Gripper Model
WSG-50
Robotiq 2F-85
Franka
AG-95
Nambee Keypoints
10
10
10
Visualization of Keypoints
Figure 18: Keypoint Annotation Visualization.
between projected and annotated 2D points) is minimized to estimate the optimal offset. The cali- brated offset is then integrated into the URDF, thereby aligning the simulated end-effector-griiper geometry with the configuration used during data collection.
Projection accuracy verification. We have defined set of corresponding 3D keypoints, and man- ually annotated a subset their 2D trajectories on the 2D image, which serve as the pseudo ground truth. Using the offset calibration, we project the 3D keypoints back onto the image plane and compute the per-pixel L2 distance to the human annotations. As seen in Table 21, across all con- figurations, the optimization procedure reduces the reprojection error by large margin (from 46.25 to 5.31 on efg.1. 1, and 108.21 I to 11 11.61 on cfg.4). This consistent improvement demonstrates that our calibration method significantly enhances 3D-2D geometric consistency.
Table 21: 2D reprojection errors across seven RH20T configurations. We report reprojection
errors (4) before and after optimization.
Config
cfg 1
cfg 2
cfg
cfg 4
clg 5
clg
cgg 7
Num of Val frames
1295
1357
1136
1094
1017
1195
1154
Before optimization
46.25
40.63
67.35
108.21
43.68
69.71
66.24
After optimization
5.31
5.69
10.80
11.61
5.47
6.31
6.95
A.6.5 QUALITY CONTROL PIPELINE
Semantic Segmentation Annotation quality control. For the quality control of semantic segmen- tation annotations, we implemented a multi-stage verification pipeline combining real-time self- inspection, cross-validation. and sampling-bass based acceptance tests. During annotation, point prompts for manipulated objects were asynchronously processed on the server by the Segment Anything returned for annotator review. Annotators
Model (SAM), and the generated masks quired to validate all masks, and any object failing three consecutive checks was marked as a hard sample and discarded, preventing low-quality data from entering the dataset. After each annota- tion round, cross-validation was conducted, with annotators reviewing and correcting each other's work reduce bias and enhance consistency. In the final acceptance stage, the dataset was di- vided into 100 subsets, with 50 samples from each subset randomly validated by new annotators. Subsets achieving less than 90% valid annotations were returned for re-annotation. This repair-and and- verification process was performed once, resulting in a final dataset with an overall high accuracy.
Language Annotation quality control. For language annotations, we established hybrid pipeline combining automated pre-annotation with multi-stage human verification. Initially, ChatGPT is used for generating draft annotations, which are then inspected and refined by human annotators. A cross-checking stage then followed, where annotators reviewed each other's work to mitigate bias and omissions. The dataset was subsequently divided into 100 subsets, each undergoing sampling- based validation (50 samples per subset) by new annotators. Subsets falling below 90% acceptance rate were returned for re- annotation. The repair-and -nd--anifiication cycle was iterated 4 times until all subsets met the acceptance criteria, resulting in final dataset with overall high accuracy.
37

---

Published as a conference paper at ICLR 2026
A.6.6 POSTPROCESSING
Semantic-Primitivv Alignment for Task Descriptions. In our annotation pipeline, semantic seg- mentation annotations and primitive skill annotations are defined at different granularities. Contact frames in semantic segmentation annotations are labeled at the progress level, whereas a single subtask may span multiple primitive skills. For instance, in the task "Pick up the green cup on the table, transfer above the blue cup, and place on top", only the initial pick action contains contact frame in semantic segmentation annotations, while the primitive-level annotation yields three clips: pick, transfer, and place. This mismatch necessitates alignment between subtask-level language descriptions and semantic segmentation annotations when constructing VQA pairs. To this end, we define clips derived from semantic segmentation annotations as semantic- based clips and those derived from primitive skill annotations primitive-based clips. For simple tasks con- taining only single primitive-base the task description. For more complex cases, where a semantic-based clip corresponds multiple primitive- based clips, we extract the start and end timestamps of the semantic-basee clip from the raw data, identify all primitive skills whose temporal overlap with this interval exceeds predefined threshold, and concatenate their descriptions to form the complete task-level language description.
clip, we directly adopt the video-level language description as
Spatial and Temporal Correction in DROID. The episode in DROID is individually calibrated, as its camera intrinsics and extrinsics are not complete and not fully reliable. We find there are misalignments between 2D trace timestamps and video timestamps, as well as positional misalign- ments. To address these two issues, we introduce complementary spatial position correction and temporal alignment, which together enhance data consistency and quality. The visualization il- lustrated in Figure [9 (1) Position Correction. Using annotated object masks, we compute object bounding boxes and centroids, and compare them with gripper bounding boxes derived from 2D keypoint projections at contact frame. If the nntersection-over-umioo exceeds a predefined thresh- old, no correction is applied. Otherwise, we assume that at the contact frame the TCP projection should approximately coincide with the object centroid. For eligible cases, calculate the 2D off- set between the TCP projection and the object centroid, and apply this offset uniformly to the entire trajectory and projected keypoints, thereby correcting systematic positional deviations. Objects box with extreme aspect ratios are excluded from correction. (2) Temporal Correction. For temporal alignment, we use CoTracker3 (Karaev al., 2024) to detect the onset of gripper motion in the video and match it with the corresponding onset frame from TCP trajectory data. The difference be- tween the two defines correction offset, which is applied to synchronize video and trajectory. After alignment, shorter trajectories are padded and longer ones truncated, ensuring precise frame-level correspondence.
38

---

Published as a conference paper at ICLR 2026
Figure 19: Visualization of correction results on the DROID dataset. For each episode, the first row shows the TCP 2D points before correction, while the second row presents the results after applying our correction method.
39

---

Published as a conference paper at ICLR 2026
A.7 DATASET OVERVIEW
A.7.1 DATASET STATISTICS
As shown in Table 22 our dataset contains a large number of videos across diverse embodied scenar- ios, which can be utilized as large-scale pretraining data. Our dataset provides extensive coverage at multiple levels: video-level annotations, such global language descriptions; frame-level anno- tations, including subtasks and keyframe counts; and local-region annotations, such as object masks and gripper keypoints. It also supports the full spectrum from generation to understanding, spanning both spatial and temporal domains.
Table 22: Statistics of the raw dataset for our annotation.
Dalasel
AVideas
#Objects
MTrames
Avg. Objects Frame
Avg. Length
Embodinent
Scene Type
Language Seg
TobleTop
47M
In the Wild
152k
53M
46M
1.16
302
All
234k
100M
86M
1.16
368
571
760k
A.7.2 VISUALIZATION FOR INTERMEDIATE REPRESENTATIONS
We visualize several examples of our annotated intermediate representations Figure 20) Figure 21 Figure 22 and Figure 23 These results show the high diversity and labeling fidelity of our dataset.
40

---

Published as a conference paper at ICLR 2026
<TAS0O press the growe dit towa back
<TARKs pres green turter yeloww nawa
<TAGIe press reth
w pawes
pome sockul
press
CTARAO list anulitiou Fur tunder bboc fram place blct Paltba
CIN the socond blick fom place
STAdOO up tind srell fomith seraler Bar)
tadle
the table
<TAcke pirk the amall brow on adde
-0B dlace Bu brown cuda Bue dowar
TWAd dass the browt cude Bue dravece
mach put pocie araal platoo racc
tacile put on spourd ane poode to rach
tack put on the secard amall dipcil
xIABO undid the showes
< unfiiitte showas
<lAgio uncii showes
<TASO firt blok be
<TAGKO
block
DB0 aidk bilied Llcck
Figure 20: Visualization of contact points (red dots) and contact boxes (white) in multi-task episodes. Each subtask is annotated below, with two images per subtask: a global scene (left) and zoomed-in in interaction view (right).
41

---

Published as a conference paper at ICLR 2026
Figure 21: Visualization of gripper bounding box (yellow) across consecutive frames.
42

---

Published as a conference paper at ICLR 2026
block de LAuc Ioand. Lltch un the Lound
Llack bluch bound, pluzu Mooduus Slock Loard!
Liock LRtck bound
<TUS 2wan ths paper box sown
-TACK> prea the poper bes down
<TASKs prene paper box down
.D pick lnth Forn arturpp cutting
-TASSON place lntle
poua
<DAGA SIAE button tockets
TTASIO g buttnn
<TAdko Rodiati the doan Baraable
<DMS pisk form cap
<TA8O slaao Bee laria
<TABK -TA-Krrp
<TABP handeolf the thewer Ber drawx
<ONdoo handle rinww the block hor the draver to the tabiits place the block lada
<BS0 uilk Viei cup the ta90e you ofa fhe oup fayn the kablle to abovo Rao rol Dx).
<TAGRO pitk the groen our on the tailt trande Be cup tablee abowo the Miu cop.
Figure 22: Visualization of manipulated object segmentation masks and object bounding boxes (magenta) multi-task episodes. Each subtask annotated below.
43

---

Published as a conference paper at ICLR 2026
brie to the Fower place the fiics the rixeer
de fan tdben to dhower
-TUO PLBBE
pull duear
Mile
<liSto pox tao roxwe traude The oijeet Front dower to above trttiigg
pauss the zutkon top posliicn
press midi Bon
preas bution
-IABK thopca bom condans
<TASK place
furt the addle to poollao abowe seroard block
fovn Ber uci la the Continn ubboce the Bisdt block
block burn line luble botta pou Bon abow du bloch
ATASO trmnder knds cuti na munizuita zutting
board allor
<itart and thea tuldk, turake the Mika from kaldo above rettioss
<TARRO pick caprr budte humo the graancaus foom tarie showe plax graun oau
Moo kgr
<TABK the Mack surlice
LUBB Mlock smeal block
Figure 23: Visualization of traces in multi-task episodes. The color gradient encodes temporal progress within each subtask: points closer to red indicate positions near the subtask start, while those closer blue denote positions approaching the subtask end. Each subtask is annotated below.
44

---

Published as a conference paper at ICLR 2026
A.7.3 DETAILS OF VQA DATASET
Statistics. Figure 24 presents an overview of the language characteristics of our dataset. The word cloud highlights the diversity of language instructions, while the accompanying histogram illustrates the object distribution. Figure 25 reports the length distribution of subtasks, providing insight into the temporal structure of trajectories. Finally, Figure 26 summarizes multiple aspects of the dataset, including the number distribution of subtasks, language token length distribution, skill distribution, and embodiment distribution, offering a comprehensive view of task complexity and multimodality.
g opper
Mable
Nptionphras
FOOD
bow
tab next tch
S Subtas up OOOL
boX
ce
4 a
Temporal
Spatial
Figure 24: Word cloud and object distribution.
CUPe
Figure 25; Transition step-length distribution.
Figure 26: More statistics. From left to right: the subtasks number distribution, language token length distribution, skill distribution, and embodiment distribution.
45

---

Published as a conference paper at ICLR 2026
Table 23: Statistics of the constructed VQA on visual intermediate representation. The choice and decision stand for the abilities of Understanding.
Name
Task Type
Class
Number
Trace
Temporal&Choice
Tabletop
3,610
mathe-Wild
8,245
Grounding
Sphoial&Choice
Tablctop
8,158
In-the-Wii Wild
Trace dir
Tempural&Choiie
Tabletop
3,729
Trace lang
Temporal&Chhoie
Tabletop
3,610
Wild
8,245
Contact
&peciel&Deeidee
Tabletop 15,060
In-the-Wild
18,184
Grounding
Spatial&Generation
Tabletop
In the Wild
149,671
Trace easy
Temporal&Geeerntion
Tabletop 33,803
the Wild
31,282
Trace hard
Cemmprra&GGnerrttion
Tabletop 33,803
In-the-Wild
31,282
Grasp Affordance (Box Mode)
Spatial&Generation
Tabletop
115,266
Wild
Grusp Affordance (Key point Mode)
Spatial&&Gemerationn
Tabletop
115,266
In-the-Wile Wild
Gripper Detection
Spatial&&Gnseation
Tabletop
120,747
In-the-Wild
84,777
Place Afferdance
Gpncral&Gnnoration
Tabletop
145,996
In-the-Wild
224,944
Grasp pose
Spatial
Tabletop
Table 24: Statistics of the constructed VQA on task planning. The choice and decision stand for the abilities Understanding.
Task Name
Task Type
Number
Planning Task
Temporal & Generation
82,939
Planning with SoathhhContttttic Task
Temporal & Generation
$8,439
Plansing Remaining SeeiaiinnnSeees_iie Steps.Task
Temperal & Generation
58,439
Generative Affordance Task
Temporal & Generation
61.804
Prediction
Temporal Generation
61,804
Success Success.Nogative.. Nogative_ Task
Temporal & Decide
79,929
Success Positive Tisk
Temporal Decide
57.413
Discriminative, Affiomlane Positive Task
Temporal & Decide
39,060
IBcciiiinatve_Affiidaaas Aftattaaeoooooneaaaaiieeeeeeee
Negatiive Task
Temporal & Decide
64,814
Pust_ Descrippiionnneee
Temperal &
Post Multi task Selection
Temporal & Choice
76,564
Future Mal.i. rask Selection
Temporal
58,439
Scene Drces__nndrsttnning
Spatial Choice
$7.413
Temperal Understanding
Temporal Choice
6,585
Past_ Primitive Selection
Temporal & Choice
76,564
Future_ Fature_Primitive SermitiveSSeecttin
Temporal & Choice
58,439
46

---

Published as a conference paper at ICLR 2026
Spatial VQA. We provide the annotation templates and visualize subset of the Spatial VQA sults, as shown below:
Grasp Pose Choice Task. Evaluates whether the model can identify the correct grasp configura- tion among candidate options. Images are annotated with orange fork-like gripper patterns that represent potential gripper poses. The model must select the one that corresponds the correct grasping pose. Visualizations are included in Figure 27
Scene Understanding. Requires selecting the correct scene (from images) that matches the long-horizon goal, assessing spatial grounding and environment recognition. Visualizations are included in Figure 37
Contact Decide Task. Assesses the model's judgment of physical contact between the gripper and the object. Given an image during a subtask, the model decides whether the gripper has already made contact, with binary choices [Yes, No]. Visualizations are included in Figure 27
Grounding Choice Task. Tests the model's ' ability to localize the correct object for interaction. Images include purple bounding boxes indicating possible target objects, and the model must choose the bounding box corresponding to the task-relevant object. Visualizations are included in Figure 28
Grasp Pose Choice Task
The mhot task <TASK>, We provide
images that feature orange fork: like gripper patterns: these patterns stand for the potential poses the gripper needs to adopt to pick up an object. Which of the images below contains the correct grasping posc? The available choice <CHOICE>, Please select one option only.
The robor task is <TASK>, Among the images offer, there are orange fork-like gripper partems-they represent the possible poses the gripper may use grab an object. Which of the following imagea has the correct grasping posc? The available choice is (CHOICK> Please select one option only.
The rhot task is <TASK>, We present some images with orange fork like gripper patters. These patterns comeesoond to potential poses the gripper requires to up object- which image below contains the correct grasping pose? The available options are <CHOICE> Please select one option only. <TASk>,
images we provide, there are orange lork-like gripper patterns: they represent the possible poses the gripper needs to use to pick an object. Which of the following images contains the correct grasping pose? The available choices are <CHOICES, Please select one option only.
The sobot
images annolated with orange lork-like gripper pullerns. These pallerns spond in the potential poses the gripper may adopi to grah an ohject so which image helow has the right grasping poxe? The <CHOTCE>, Please select
The robot task is <TASK>. We present
available choices
option only.
The robot task is <TASK>, We supply some images that include orange fork-like gripper patterns; these patterns represent the images below contains the correct grasinn pose!!
poses the gripper might eCHOICE>. Please select one option only.
lake pick up an object. Which
available choices
The robot task LASKS. Among the images we present, there are orange fork-like gripper patterns; they correspond to the grab object. Which image helow has the correct grasping pose? The available choices are <CHOTORs. Please select one option only.
potential poses the gripper needs
The robot is <TASK>, We offer some images featuring orange fork-like gripper patterns. These patterns stand for the poses the gripper might adopt to pick up an object-so which of the following images contains the correct grasping pose? The available chonces <CHOICE>. Please seleci ome option only.
The robot task is <TAsks, We provide the potential postures the gripper may use to grab an object. Which imuge below has the correcl grasping pose? The available chaices are <ChOTORs. Please select option only.
images that include orange fork-like gripper patterns; these patterns stand for
The robot task is <TASK> The images we supply contain orange fork-like gripper patterns: these pattemss represent the grab object. Which images below right grasping posc?
potential poses the gripper may soly available choices are <CHOLGE>. Please select one option only.
Answer: <CHOICE>
Scene Understanding
Given four images labeled B. D. which image hest corresponds the required for accomplishing (long-harizon)>
Please answer with single option (A/B/C/D).
Which images (A, C, D) best matches scene necessary achieve (long-horizon) Please answer a single
option (A/B/C/D).
LLooking at the four images A. B, C, D, which one best represents the scene for (long-borizon) Please answer with a single
option (A/B/C/D). Ammng the four images (A, B, C, D) provided, which scene correspands to hunn-harizon)? Plense answer with single option (A/B/C/D).
Which image (A, B, C, D) most accurately depicts the scene for (long borizon)? Please answer with single option
(A/BCID).
Answer: A/B/C/D
47

---

Published as a conference paper at ICLR 2026
Contact Decide Task
TTee is <TASK>, now you are performing subtask <SUSTASK> In image, you think the gripper has already made contact with the object? The available choices are [Yes, No]. Please select one option only.
The robot task is <TASK>, and now you are performing subtask <5U3TASK>, In this image, has the gripper made contact Yes, Nol. Please select one option only.
with the object? The available choices
The robot lask is <TASK>, and now you are performing sublask <EUSTASK>. In this image, do you think the gripper has already touched the object? The available choices are [Yes, Noj. Please select option only.
The robot task is <TAGX>, and now you are performing subtask <SUBTASK>, In this image, has the gripper touched the object? The available choices are [Yes, No]. Please select one option only.
The robot task is <TASK>, and now you are performing subtask <SUBTASK> K>. In this image, do you think the gripper has already reached the object? The available choices are Yess Noj. Please select one option only.
The robot task is <TASK>, and now you performing subiask сыUвΙASK>, In this image, has the gripper reached the object? The available choices are [Yes, No). Please select one option only.
The robot task is <TASK>, and you performing subaask <SUBTASK> In this image. you think the gripper has already appreached the object? The available choices are (Yss, Nol. Please scleet one option only.
image, has gripper approached the The robot task <TASK>, and now you are performing subtask <SUBTASK>, objoct? The available choices are [Yes, No]. Please select onc option only.
Answer: Yes / No
Grounding Choice Task
The robot task <TASK>, We offer some images containing purple boxes, and these boxes correspond the objects that maybe interact with robotic arm, which image describe right object bounding box? The available choice |<CHOL CE> Please select one option unly.
<TASK>, images we present contain purple bounding boxes, which for the ohjects the robotic amm maybe interacts with, which image depicts the correct object bounding box? The available choice is [<CHOICE=] Please select one option only.
The robot task is <TASK>, Among the provided images, purple boses present represent the objects that maybe gage with the robotic arm, which of the images shows the pruper hounding box for the object? The available options are
I<CHOICE>]
option only.
The rubot task <TASK>. We ofler some imuges where purple boxes are used to indicate the objects maybe interacting with robotic arm, which image accumtely descrihes the object's hounding hox? available choices are [sCHOTCRs]] Please select one option only.
The robot task is <TASK>, We present a set of images with purple boxes, and these boxes represent the objects maybe be involved in the robotic arm's interactions, which image contains the correct bounding box for the target object? The available choices are [sCHOTCE>]] Please select one option only.
provide several imuges featuring purple boxes, which signify objects that robotic arm mayhe interacts with, which image correctly illustrates the bounding box of the object? The available choices are [sCHOTCE>] Please select one option only.
The rubot lask <IASK>,
The robot task is <TASK>, The images provide include purple boxes that denote the objects maybe interacting with the robotic arm, which image accurately represents the bounding box of the object? The available choices are <ссооси>>]
Please select option only.
The robot task <TASK>, We present images containing purple baxes, which indicate the objects that the robotic arm mayhe interacts with, which image correilly shows the bounding box of the object? The available chuices are [<CHOICE>]] Please select one option only.
The robot task is <TASK>, provide images with parple boxes that represent the objects involved in the robotic arm's interactions, which image sccurately depicts the bounding box of the object! The available choices are |<CHOICE~|. Please select one option only.
The robot task is <TASK>, The images present feature purple baxes that signify the objects interacting with the robotic arm, which image correctly illustrates the bounding box of the object? The available choices are [sCHOICD>] Please select one option only.
Answer: <CHOICE>
48

---

Published as a conference paper at ICLR 2026
The robot task 'move the blue cup; and now you are performing subtask piick up the cup on table', you think the gripper has already touched the object? The available choices are [Yes, No]. Please select one option only.
image,
No
The robot task 'put green object in basket', now you are performing subtask pick up the green object the table'. In this image, do you think the gripper has already touched the object? The available choices are [Yes, No]. Please select one option only.
Yes
The robot task right, and now you performing subtask 'move the spout to the right'. this image, has the gripper reached the object? The available choices are [Yes, No]. Please select one option only.
the spout
The robot task 'take the of paper down from the shelf'. We supply some images that include orange fork-like gripper patterns; these patterns represent the poses the gripper might need to take to pick up an object. Which the images correct grasping pose? The available choices [A, B, DJ. Please select option only.
The robot task 'installing light bulb rotating Among the images offer, there orange fork-like gripper patterns-thh teenssttheyyee represent the possible poses the gripper may use to grab an object. Which of the following images has the correct grasping pose? The available choice [A, C, Dl.Please select one option only.
The robot task is 'installing the light bulb by rotating it'. Among the images we offer, there are orange fork-like gripper patterns- represent the possible the gripper may object. Which of the following images has the correct grasping pose? The available choice is [A, B, C, D).Plaase select one option only.
Figure 27: Contact decide task and grasp pose choice task.
49

---

Published as a conference paper at ICLR 2026
The robottase task is "push the faucet to the right,, The images we present contain purple bounding boxes, that the robotic maybe interacts with, which image depicts the correct object bounding box? The available choice is [[A, B. option only.
stand
objects
C, DJJ Pleese
The robot task is 'move the marker from silver bowl the table'. We provide several images featuring purple boxes, which signify the objects that the robotic arm maybe interacts with, which image correctly illustrates the bounding box of object? The available choices are [[A, B, DJ.. Please select option only.
A
The robot task is 'apen the upper cabinet door on the left. We offer some images containing purple boxes, and these boxes correspond
objects interact with the robotic arm, which image describe the right object bounding box? The available choice is [[A, C, DJJ. Please select one option only.
maybe
The robot task ppour some contents from the beaker into red bowl,, partorming suatask piic table'. We provide some imagee that featture gradient- colored trajectories: start a trajectory, while red marks its end. These trajectories regresent nt the possible paths the gripper needs image depicts the correct trajectory? The available choice is [A. B, C, Please select one option anly.
indicates the
finish the subtasik. Which
C
P
A
B
lopen top on right,, and now you are performing subiask 'pull the handle of drower. provide some images containing gradient-colored trajectories; green indicates the of trajectory, red indicates These trajectories are the possible pathe the gripper may take to complete the subtask. Which has right trajectory? available choices are [A, C,C, DJ, Please solect one option only.
on the
the
C
The robot task is lpour the things the cup into the bowl,, and up the cup from the table'. We provice some images conteining gradient colored trajactories: green indicates red indicates its end. These trajectories are the possible paths the gripper may take complete image the right trajestory? The available choicas are (A, B, C, DJ. Please select one option only.
pertorming subtask
trajectory,
Ed
A
Figure 28: Grounding choice task and trajectory choice task.
50

---

Published as a conference paper at ICLR 2026
Temporal VQA. We detailed the 18 types of temporal VQAs, aiming evaluate the models' quential understanding and decision becision-making abilities.
Trajectory Choice Task. Evaluates the model's understanding of motion planning. Images depict candidate trajectories using a green-to-red gradient, and the model must identify the cor- rect trajectory the gripper should follow to complete the subtask. Visualizations are included in Figure 28
Trajectory Direction Choice Task. Focuses on predicting the immediate motion direction of the gripper. Images display multiple colored arrows around the gripper, and the model must select which arrow correctly indicates the actual move direction. Visualizations are included in Figure 29
Trajectory Sub-goal choice Task. Requires the model to match a demonstrated trajectory with its corresponding subtask. Gradient-colored trajectories (green start, red end) are shown, and the model must decide which subtask best aligns with the trajectory among candidate options. Visualizations are included in Figure
Discriminative affordance negative task. Evaluates whether proposed task is not the correct next action, requiring the model to reject infeasible affordances with a strict "No". Visualizations are included in Figure
Discriminative affordance positive task. Tests the model's ability to identify the correct next action among possible options, confirming feasible affordances with strict "Yes". Visualiza- tions are included in Figure(3]]
Future multitask selection. Given a long-horizon goal, asks the model select the most ap- propriate next subtask from multiple choices, emphasizing sacision-making under alternatives. Visualizations are included in Figure 33
Future prediction task. Requires anticipating the outcome or immediate follow-up task after completing a given step, testing temporal foresight in task sequences. Visualizations are included in Figure 34]
Future primitive selection task. Focuses on predicting the next primitive skill apply (e.g., grasp, push, move) to achieve a long-horizon objective, ensuring fine- fine-grained planning ability. Visualizations are included in Figure
Generative affordance task. Prompts the model to generate the most feasible immediate action in the current state, highlighting short-horizon decision decision-making. Visualizations are included in Figure 32
Past description task. Asks the model recall the most recently completed task, evaluating memory of temporal progress. Visualizations are included in Figure 34
Past multitask selection. Provides multiple subtasks and requires the model to identify which one has already been completed, testing retrospective recognition of task history. Visualizations are included in Figure 33
Past primitive selection. Requires identifying the most recently executed primitive skill, rein- forcing lowelevel action traceability. Visualizations are included Figure 35
Planning remaining steps task. After partial progress, asks for the next five sequential steps toward the goal, testing structured long-horizon planning ability. Visualizations are included in Figure 35
Planning task. Directly queries the model for the immediate next step toward a long-horizon goal without additional context, serving as the simplest planning probe. Visualizations are in- cluded in Figure 36
Planning with context task. Builds on prior completed tasks to decide the most logical next step. testing consistency and coherence in long-horizon reasoning. Visualizations are included in Figure 36
Success negative task. Asks the model to verify that given task not successfully com- pleted, probing its ability to detect failure.
Success positive task. Asks the model to confirm that given task was successfully completed, probing its ability to detect success.
51

---

Published as a conference paper at ICLR 2026
Temporal understanding. Given multiple images, it requires identifying the one that corre- sponds to the execution state of a task, evaluating temporal grounding and process awareness. Visualizations are included in Figure 37
Trajectory Choice Task
The robot task <TASK> and you performing subtask <SUSTASK>. We provide images that feature gradient- colored trajectories: green indicates the start of a trajectory, while red marks its end. These trajectories represent the possible paths the gripper needs take finish the subtask. Which image depicts the correct trajectory? The available choice <CHOLCE>. Please select one option only.
The rubat lusk <IASK>, and now you are performing subtask <SUBTASK> We offer set images containing gradient- colored trajectories. Green indicates the start of each trajectory, and red indicates its end these trjjectories represent the potential routes for the gripper to finish the subtask. Which image depicts the right trajectory? The available choice
<CHOT cRs Please select option only.
The robor task is <TASK>, and now you are performing subtask <BUBTASK> The images we supply include gradiont-colored trajectories: green denotes the of a trajectory. and red denotes its end. These trajectories stand for the paths the gripper night need to follow to accomplish the subiask. Which image has the correct trajectory? ? The available options are SCHOICE>. Please select one option only.
The robot task is <TASR> and now you are performing subtask <SUBTASR> We offer set of images containing, gradient- colored trajectorios. Green indicates the start of each trajectory, and red indicates its these trajectorics represent the potential routes for the gripper finish the subtask. Which image depicts the right Inajectory? The available choices are <CHOICD>, Please select one option only.
images we provide, there are gradient- colored trajecturies: green represents the start point of injeectory, and red represents its end point. These irajectories cur- respond to the paths the gripper may need take to complete the subtask. Which image shows the correct trajectory? available choices are <CHOICE>, Please select one option only.
The sobot
<TASK>, and now you are performing subtask <SUPTASK>
The robot task is <IASK>, and now you are performing subtask <SU3DASK>, We supply some images that include gradient- trajectory, and red for its end. These irajectoris represent the potential paths the gripper needs to follow to finish the subtask. Which image has the right trajectory? The available choices are <cнo ICE>, Please select one option only.
colored Injjectories green stands for the beginning
The rohor task is <TASK>, and now you are performing subtask <SUSTASK>, We provide some images with gradient colored traectoories Green represents the start each trjjcctory, represents end, and these trajectories correspond to paths the gripper needs to take to accomplish the subtask. Which image shows the right trajectory? The available choices are SCHOLCE>. Please select one option only.
The robot task <TASK>, and now you performing subtask <SUETASK>, Among the images present, gradient- colored trajectories are included- -reen stands for a uraectory's 's start, and red for its end. These trajectories represent the potential paths required for gripper to finish subtask. Which image has the correct trajectory? The available chuices are sCHOTCF: CRS. Please select one option only.
The robot task <TASK> and now you are performing subtask <SUBTASK>, We provide some images containing gradient- colored trajectories: green indicates the start of each trjectoryy and red indicates its end. These trajectories are the possi- ble palhs the gripper may lake to complete the subtask. Which image has the right inaeector?? The available choices are <CHOTCE cES. Please select option only.
The robot task is <TAas>, and now you are performing subiask <SUBTASK>, We present a set of images with gradient- colored trajectories green marks where trajectory begins, and red marks where ends. These trajectories stand the potential paths the gripper requires to accomplish the subtask. Which image shows the correct trajectory? The available choices are <CHCICB> Please seleet one option only.
Answer: <CHOICE>
Trajectory Direction Choice Task
The robot task <TASK>, and now you are performing suhtask <SUBTASK> There some different color arrows around gripper the image, which color most likely represents the actual move direction of the robot gripper? Please choose only one option from <CHOICES.
The zubot lusk <TASK>, and you performing subtusk <BUBTASK>, There several arrows with different colors around the robat gripper on the image, which color describes the actual move direction of the rohoi yripper? Please choose only option from <CHOICE>,
The robot task is < <TASE>, and now you are performing subtask <BJDTASK>. On the image, different arrows surround the gripper-son some near its jaws, others hy the target part. Which color most likely shows the gripper's is actual move direction? Please choose only one option from <CHOICE.
The robot task is <TASK>, and now you are performing subtask <SUBTASK>, Several colored arrows are around the robot gripper in the image; same point loward the box, others away. Which color describes the gripper's real move direction? Please choose only ane option from <CHOTCRs.
The robot task <TASK>, and now you are performing subtask <SUBTACK>, Different arrows are around the gripper in the image-upward and downward ones. Which color most likely represents its actual movement direction? Please choose only one option from <CHOICE>.
The robot task is <TASK>, and now you are performing subtask <SUBTASK>, On the image, various arrews surround the gripper-sone point loward the box, others away. Which color most likely shows the gripper's is actual move direction!' Please choose only ane option from <CROICE>.
The robot task is < <TASK>, and now you are performing subtask <SUSTASK> Several colored arrows are around the robot gripper the image; some point up. others down. Which color deseribes its actual move direction? Please choose only one option from <CHOLCE>.
The robot task <TASK> and now you are performing subtask <SUBTASK> Several anows with different colors are around the gripper: some horizontal, others vertical. Which color describes its actual move direction? Please choose only one option from <CHOLCE>.
Answer: <CHOTCE>
52

---

Published as a conference paper at ICLR 2026
Trajectory Subchoice Task
You performing <TASK>, Now provide image that feature gradient colored trajectories: green indicates the of a trajectory, while red marks its end. The trajectory represent the path the gripper needs take to finish the subtask. Which trajectory among the following options: <CHCICE>? You are performing <TASX>. Below an imuge showing gradient-colored trajectories, where green signilies the starting point indicates endpoint. The trajectory illustrates path gripper should follow complete the subtask. Among the options <CHOTCE>, which subtosk does the trjecctory best correspond to? Please select one option only.
subtask the match
option only.
You are performing <IASK>, The image below displays gradient colored trajectories, with green representing the start and take accomplish the subtask. From the choices <CHOTCR>, which subtask aligns best with the trajectory? Please select one option only.
red the end. These tnjecttries depict the path the pripper
image provided shows gradient-colone trajectories. where green marks beginning and red the conclusion. These trajectories outlime the path the gripper needs to follow to complete the subiask. Which subtask from select one option only.
You performing <TASE>.
options sCHOTCE> does the trjecttory best represent?
You are perforning <TASK>, The image below fearures gradient-colored trajectories, with green indicating the start and red the end. These trajectories illustrate the path the gripper should take to accomplish the subtask. Among the choices <CHOICE>, which subtask does trajectory best correspond to? Please select option only.
You are performing <TA3K>, The image provided displays gradient-colored trajectorics, where green significs the starting marks the endpoint. These trajectories depici the path gripper must follow complete subiask. the optioos <CHOI cs>, which subrask aligns best with the trajectory? ? Please select one option only.
point
You are performing <TASK>, Relow
image showing gradient-colores tmjectories, with green representing
red the end. The trajectory illustrates the path the gripper needs to take to finish the subtask. Which subtask from the choices <CHOTCR>> does the tmjectory hest represent? Please select one option only. Answer: <BUBTASK>
Planning Tisk
short phrase. In order to fulfill (long-horizon), what is the next best step? Please answer in shoct phrase.
advance toward (long-horizon), what should done next? Please
should To continue progressing toward ( (long. horizon), ), what comes next? Please answer in short phrase.
Keeping (long horizon] } as the goal,
step move forward! Please answer short phrase.
Aimingg for (long-horizon) ). which step should you take now? Please answer in a short phrase.
What should be the immediste next
achieve (long-horizon) Please answer Considering [long-horizon] as the target, what task needs to be done next!' Please ans wer in a short phrase.
short phrase.
closer (long-horizon). J. what task should follow? Please answer
short phrase.
With flong-borizon in mind, what is the most appropriate next action? Please answer in a short phrase.
uchieving [long horizon), ). whats next actionable task? Please answer in short phrase.
Auswer: <task n>
Planning with Context Task
Having accomplished (past task ~- n-) what should he your next step achieve (long horizon) ) Please answer in short
phrase.
After completing (past task 1 ~J n-1 what action will best advance you toward (long-horizon ) Please answer short
phrase.
short phrase. Whats the next task to reach (long-horizon), given that youve finished (past task ~ n-I] Please answer in short phrase. best next move loward (long horizon] With progress far including (past task ~ n-1). what should you do next to achieve morizon ? Please answer in
With (past task n- done, what
to closer [long horizon) Please
Taking account [past lask n-l
answer
phrase.
short phrase.
Whhts the next step in pursuit of [long-borizon). after completing steps (past task o n-13? Please answer in short phrase. far, youve completed (past lask n-1). Whats the next logical slep reach [long horizon)? Please answer in short phrase.
Whats the next action towards (long-horizon), considering the completed tasks ([past task ~ n-13)? Please answer in short phruse. Given your progress ( ([past task ~ n-1 }],, whats the most logical next step for achieving (long-horizon Plense answer in short phrase.
Answer: <task n>
Gonerative Affordance Task
Whats the most practical action you can start immodiately? Please answer short phrase.
Considering the current state, what task can begin right away? Please answer in short phrase.
W Whats somethhnn you can work on right now? Please answer short phrase.
this point, what action you take immediately? Please
short phrase.
Considering whats possible now, what should you do first? Please answer in a short phrsse. in
Whats the task
immodiatcly? Please What action is most feusible to take at this moment? Please answer in a shoet phrase.
phrase.
Whats the next task you can focus right now? Please answer short phrase.
Whats the most available task handle present? Please answer in short phrase.
Given siuuttion, whats the best action to lake right now? Please answer in short phrase.
Answer: <task n>
53

---

Published as a conference paper at ICLR 2026
Planning Remaining Steps Task
(past task D-1) completed, what five actions {long-borizoo)? Please sequential
phrases.
steps you to achieve (long horizon), accomplishing (past n-1)? ) Please answer
sequential phrases.
Given that youve done {past lask ~ n- I. what are the next five things to do for (long horizon) ? Please answer in sequential
phrases.
Having completed [past task ~v n-1), what five tasks will best advance you toward (long-borizon)? Please answer
sequential phrases.
Considering the progress für ((past task ~ n-1)), what live sleps should follow to accomplish ( (long borizon Please
answer in sequential phrases.
Affer finishing (past task n-1), what are the next five actions take toward (long-horizon)) Please answer in sequential
phrases.
What
five tasks complete after completing (past task n-13 to achieve [long-horizon) Please
soquential phrases.
Considering llom-horrzoon)
logicul steps (past
n-1]?
sequential
phrases.
actionable steps to follow tpsst n-1) achieving (long-horizon] ) Please answer sequential phrases.
To move toward (loog-borizon), what are the next five key actions after {past task ~ n-13? Please answer sequential phrases.
Future Prediction Task
phrase. Once (task n-1) complete, what is expected happen next? Please answer in a short phrase.
After [task n-1). what development likely take place? Please answer short phrase.
What most likely to occur after (task n-1) fimishes? Pleuse unswer in a short phrase.
Given
might follow (task
Please
Following the completion of (Lask n-1 1. what is anticiputed to happen? Please answer in short phrase.
outcome expected
(task n-1] 1]? Please answer
phrase.
Whats the likely result after completing (task n-1 Please answer in short phrase.
Whats predicted to happen after (task n-1 22 Please answer in a short phrase.
After (lask n-1 }. whats the
you anticipate? Please
in short phrase.
Following (task n 1). what predicted to occur? Please unswer in a short phrase.
Past Description Task
What task was finished just now? Please answer in short phrase.
did you complete most recently? Please answer in short phrase.
What the last action carried out? Please
in short phrase.
What done immediately before this? Please answer short phruse.
Which lask did you
this? Please
short phrase.
Which task was concluded most recently? Please answer in short phrase.
previous action performed?
phrase.
What task did you complete last? Please answer in short phrase.
What was the final task before now? Please answer short phrase.
What action occurred just prior this? Please answer short phrase.
Answer: <task n>
Success Negative Task
Has (task n) boen finished as intended? Please answer strictly with
Is (task n) regarded as successfully completed? Please answer strictly with Yes or No.
Is il confirmed that [tukk n) was done properly! Please unswer strictly with Yes or No.
Has [tusk been accomplished
expected?? Please Can we confirm that (task n] was completed as required? Please answer strictly with Yes or No.
strictly with Yes No.
Did (task n. achieve the desired results? Please
strictly with Yes Was [task n) executed successfully? Please answer strictly with Yes or No.
Is (task n considerod success? Please answer strictly with or
Can [usk n) be marked complet?? Please answer strictly with Yes No.
Was [lask n) carried out as planned? Please unswer strictly with Yes or No.
Answer: No.
54

---

Published as a conference paper at ICLR 2026
Success Positive Task
Has (task a) been linished as intended"' Please answer stricily with Yes or No.
(task n) regurded succesfullly completed?? Pleuse unswer stricily
Is it confirmed that (task n] was done pmperly? Please answer strictly with Yes or No.
-Has {lask n) been accomplishedd as expected!' Pleuse unswer striclly with Yes or No. completed required? Please
Can confirm that (task
strictly with Yes No.
Did (task nj achieve the desired resalts? Please answer strictly with Yes or No. strictly
Was (task nJ executed succesfully? Please
No.
Is [task n considered success? Pleae answer strictly with Yes or No. (task n) marked complete?
answer strictly Was [task n carried our planned? Please answer strictly with Yes or No.
Answer: Yea
Discriminative Affordance Positive Task
Is (task n the task you should perform next? Please answer strictly with No.
Are you expected handle [task n) the step? Please strictly with Yes No.
Is (task n the focus of your upcoming efforts? Please answer strictly with Yes or No. addressed!'
Is (task n) the
answer strictly Yos No.
Are about work on ftask m neat? Pleae answer striclly with Yes or No.
(task n) possihle
answer strictly
Can [task n be executed in the next step? Please answer strictly with Yes or No.
Is it realistic begin (task n} in the following step? Please answer strictly Yos or No.
strictly with Yes No. Can you take up {task n} immodiately after the current task? Please answer strictly with Yes or No.
Is (task n] achievable the stage? Please
Discriminative Negalive Task,
Is (random task) the task you should perform next? Please answer strictly with Yes or No. frandom
step' Please answer strictly 1s (random lusk) the focus of your upcoming ellorts? Pleuse unswer strictly with Yes or No. be addressed?
you expected
(random task) the
strictly with
Are about work (random task) next? Please answer strietly with Yes or No.
Is {random task] possible start next? Please answer strictly with Yes No. step? Plense
strictly with Yes No. Is it realistic to begin frandom task) in the following step? Please answer strictly with Yes or No.
Can [randomm lusk] be executed
stage? Please answer strictly Can you take (random task) immediately after the current task? Please answer strictly with Yes or No.
(random
achievable
Answer: No
Pusl Mulii lask Selection
(long-horizon), which following
already completed?
Choices: [A. <Lask 2>, clask 25, clask 3>, D. <Lask 4>].
In the context of (long-horizon). which subtask has been finished so far?
Choices: [A. <task 1>, B. <teak 2>, <task 3>, <task 4>}
&Reganding (long- hongghorrzzn)... which subtask among the following has already been been accomplissed?
2>, To achieve (long- horizon), which suback has just heen completed?
Choices:
<task
Choices: [A. stask 12, stask 29, staak 30, <taak 4>].
Which of the listed subtasks has already boen exccuted for (long-borizoa]?
Choices: IA. <task 2>, B. <task 2>, C. <task 3>, <task 4>1.
Answer: A/B/C/D
appropriste Choices: [A. <task 1>, B. <taak 2>, <task 3>, D. <taak 4>].
achieving (long-horizon). which subtask
next?
order complete (long-horizon), which subtask should executed next?
Choices: [A. <task 1>, <tesk 2>, C. <task 3>. D. <task 1>].
-hhich sublask should the next step toward (long-horizon]
Choices: [A. <task 1>, B. steak
<taak 3>, <taak 4>}.
-Looking ahead to (long-horizon), which subtusk should be performed next?
1>, For proressing toward horig-horizon), which subtack is the most suitable next step?
Choices: [A.
cLask 3>,
Choices: [A. cLask 12, B. clask 2>, cLask 3>, <Lask 4>].
Answer: A/B/C/D
55

---

Published as a conference paper at ICLR 2026
Scene Understanding
which imago corresponds the required accomplishing (long-borizon Please unswer with single option (A/B/CID)
Given four images labeled
images (A, C, D) option (A/B/C/D).
scene necessary achieve (long horizon]?
answer single
-Looking al the four imuges A, B, C, which one best represents the scene (long harizon)? Please answer with a single
option (A/R/C/D). Among the four images (A. B, C, D) provided, which sceme corresponds to (long-borizon)? Please answer with single
option
Which image (A, B. C, D) most uccuraliely depicts the scene fur (long barizon)?? Pleuse unswer with single option (A/B/CAD). Answer: A/B/C/D
Temporal Understandding
Given four images laheled A, B, C, and D. which image corresponds to the state while executing (task n)? Please answer with single option (A/B/C/D).
Which of the four images (A, C, D) shows the state during the execution (task n)? Please with single (A/B/C/D).
-Lanking at the four images, which depicts when (lask n) heing carried out? Please answer with single which represents while performing (task n)?
(A/B/C/D).
AAmong images (A/B/CT).
answer single option
Which image (A, B, C, D) corresponds to state during exccution of {task n)? Please answer with single option (A/B/C/D). Answer: A/B/C/D
Past Primitive Selection
Which primitive skill has just been upplied achieve flong horizon)
Chaices: [A. <ak1 12, <akill 1 2>, <ak111 3>, D. <ski1) 1 4>].
In pursoit (long-borizon). which primitive skill applied recently?
Choices: [A. <skill 1s, Bs <skill 2>, C. <akill 3x, D. <skill 4>).
-Which was just executed for purpose
Choices: [A, <akt 1>, B. <ole 2>, C. <aki 3>, <ski11 4>].
achieving (long-horizon), which primitive was appliod last?
1akill 2>, <aki 11 3>, Soskkll 4>]. Which primitive skill had just been used in the process of (long-horizon)?
Choices: [A. <akill
Choices: [A. <skill <skill <akill 3>, <skill
Answer: A/8/C/D
Primitive Selection
Which primitive skill should applied achieve (long-horizon)?
Choices: [A. <akill 1s, B. <akill 2>, C. <akill 3>, D. <akill 4>}.
lorng-horzzon), which primitive
most
apply
Choices: [A. <skill 1>, <skill 2x, C. <akill 3>, D. <skill 4*].
In order to accomplish (long-horizom), which skill should applied next?
Choices: [A. <akill 1>, <ekill 2>, <akill 3>, <skill 4>].
Lonking ahead to (long horisonrr which primitive should he chosen next?
Choices: [A. <skill
<skill 25, <akill 3x, <ekill
Which of the following skills should be applied as the next step for (long borizon)?
Choices: [A. <skill 1>, <skill 2>, <akill 30, <ekill 4>].
Answer: A/B/C/D
56

---

Published as a conference paper at ICLR 2026
The robot task the cup forward', and now you are pertorming subtesk "pick up the cup
table'. Several different colors are around the gripper; some are horizantal, others vertical. Which color describes actual choose only one option from [A. Red, B. Green, Blue, Yellow].
direction? Please
robot "put towel', and now are performing subtask "pick up the item on the table'. There are several arrows with different colors around the robot gripper on the image, which color describes the actual move direction robot gripper? Plesse choose only one option from (A. Red, B. Green, Blue, Yellow).
paper
The robot task is 'pour the contents the green the plate, then the you pertorming cup on the table: Different arrows are around the image upward dowmward ones. Which color likely represents
'pick
movement direction? Please choose only one option [A.
Yellow)
C
finish setting up starting position of cheseboard that is almoet erranged;
performing subtask 'pick There are some difforent arrows around grippar, the
with image. color most likelly represents actuel movement direction? Please choose only option from [A. Red, Green, C. Blue, Yellow]
trajectory
The robot tack is 'hit the pool ball', and now you are performing subtask pick up the que the table are
around gripper, along with passed trajactory of gripper on the image. Which color most likely represents chocse
actual movemnnt direction? one option from (A. Red, Groen, C, Blue, D. Yellow].
B
g
The robot task is 'pick up the bottle table', and now you are performing subtask 'pick up the white bottle from the
Around gripper the image, there are multiple cistinct arrowss plus trajectory the gitpper has followed past. Which the actual movement direction of the robotic pripper? Please choose Green, Blue, Yellow))
likeyy
for
option from (A.
Start
B
Figure 29: Trajectory direction choice task and trajectory direction choice with trajectory task.
57

---

Published as a conference paper at ICLR 2026
The image displays gradient colored trajectores, with gi gue repres coveesninng ing the start and denoting the end. the trjectory nut mas the parh the g gropper must takne accomplish cort tasi. From the choices dAuioose the cuphosed door, Butrenaferthe the tall from the table the bowd, C-put the inside the theddsk),, one option only.
ento the trajectory?
the
slighs
the image, grad ent colo trajectorics are ilkstruted, with grean sig ilying the start and red indicating end. The trajectory outtines the route the gripper needs take to finish particular task. From the options (A (Alium light soitch, plaee the the button table), wligha best trayectory? Please option only.
Start
Start
The image features gadiert-cntond trajectories, with indicatingthe
showinngte
pipper clip toble, B.open Cuemove clothies on tacke, turn off the thind switch from the rightt. which one best matches the trajectory? Please select one aption
End
designet
storagle
performing
cup the imagle
dishwoaalher the corner below features aadient-cotored trajectories. with green indicating the start and red the end. These trajectories illustatt the pach the gripper shoulc tabe to accamplish the subtask. Among the choices (A. scrub the table brush, B. pick the white from the diahowishhr, C. transfer remote the table transfere trajectiry soereppond substekk
Stort
option
You performing press the buiton from botioni. New provide trajectories,
teature colored
trajectiory represent subtask, match trajectory among the following optiones prwss the button bettom, B. place the white plate on the counterioo, pick up the merkat from the red may, D. place the first cottee cup on the second cotfee cupf? Please select one option only.
while
gripper needs take subtask is
to
performing colored trajectories, green representing gripper
the displeye
trajector las depict paththo subtsskk choices (A. pick up the toy the bowl. pick up the mediam pipette bus the stand for medium pipaties, C. pick up the marcer from the bow,, D. pick up the id from the pot), which suotask west the trajectory? Plasse sellct one option only.
sccampisss
Figure 30: Trajectory sub-goal choice Task
58

---

Published as a conference paper at ICLR 2026
Are expected handle pick the lid from the table' as the next step? Please answer strictly with
Are you about to warkon upckup the green
Please answer strictly with Yes No.
No
"ickk up white cup the drawer' the focus of your upcoming effortss Please answer strictly with
No
'transfer pack cabinet the counter' the focus of your upcoming efforts? Please answer strictly
Yes
Is 'place the marker on the table' the ta you should perform next? Please strictly with Yes
Is 'transfer the marker from the table top cup' achievableas
next stage? Plea se answer strictly with Yes or No.
Yes
Figure 31: Discriminative tasks of task planning.
59

---

Published as a conference paper at ICLR 2026
Looking 'stacking blacks', which primitive should be chosen next?
Choices: A. pick B. pull C. tramsfer, D. place. Please one option only.
A
Looking ahead to stacking blacks' which primitive should be chosen next? Choices: A. pick B. pull trunsfer, D. place. Please select one option only.
To move toward a plece paper towel and it the left side of the sink', which primitive skill should be Choices: transfer, slide, twist D. Pull. Please select option only.
A
What action most feastble this moment? Please answer in short phrase.
Place the marker inside the cup.
At this point, what action can take immediately? answer phrase.
Place the cup on the tabletop
Given the situ atttot action to take right now? Please answer in a short phrase.
Place the green packing bag transparent bowl
Figure 32: Future primitive selection task and generative affordance prediction task.
60

---

Published as a conference paper at ICLR 2026
To more closerto 'ou hoode the bot/whii which sabrask ahould
bood.
pitkup the honcle from the soft,
D. frans the baodielionn centte so the box. Plesse eelect ore option only.
Porthe gast ppu the sprun un thgrttht side of the bond". which altesk is mout upprogriate as the rest action? Choces the right
traneke the spoon fremtta
bood
top towed tuble
pirkspphhe spoon the sahle, Ploae esseeteeeteoe one ontion only.
Por scchavinngg pick
and the bottum
thedesk', which vubrast is the most oppespiaate execute nost? cholcee: gide
pht,
push spea.
transl the soda
thetable
Pleame select only
firidhedisn fyr? Cholees
A chabe the red alagahice shove rabie
ii pidk up the red flus on the tuble,
plaen fleeze seot
D. place opeion
e
For objectve blcc which saltack alrasdy sucuendtall compleedd shotces:
Pheas the puzzi plece the puzdia.
andtoordhhe the desk, pes
select
of sateadichus almody been "put onange block froms bond os the table? Choteas rrvache the srange slock the bowi shoow the rable, pros the lover on toaster doon,
pick up orange
hawe,
place arage
Figure 33: Future multi-task selection task and past multi-task selection task.
61

---

Published as a conference paper at ICLR 2026
What task was the finished just now? Please answer in a short phrase
third power outlet
What final task before now?
Please anever in shart phrase.
Place the chocolate in the sink
What task did you complete last?
Please answer in short phrase.
Place the right slide sandal on the topmost shelf shoe
What outcome is expected to lllow ppick up the tool on the stand'? ? Please chort phrase.
Transfer the tool from the stand the cup
'pick up ground', what's the you anticipute? Please answer in shart phrase.
Fold the towel from the left right.
What's the likely result after completing 'pick up purple bag from the sink? Please answer short phrase
Transfer the purple from the sink to the countertop
Figure 34: Past description task and future prediction task.
62

---

Published as a conference paper at ICLR 2026
Which primitive skill jus used in the process of prress the button'? Choices: press, B. pull, C. place
Considering push the black object', which primitive skill bas just been performed? Choices: transfer, push C. insert, D. fold.
Which primitive skill had just been used in the process l'rrrange the small pillow on the sight-side of the sofa"? Choices: A. twist B, pick place transfer
B
After finishing '1. push the howl on the table right, pick up the spoon the table, transfer the spoon from the table the right side of the baw!,, what acti take toward' the spoon the right side of the bowl'? Please answer in sequential phrases.
place the spoon on the right side of the bowl
What live should you take achieve 'rip piece of paper towel and put left side the sink' accomplishing '1.
piece of paper towel from the roll, 2. transfer the tom paper towel from the right side of the sink to the left side of the sink? Please answer in sequential phrases.
place the tom paper towel on the left side
Figure 35: Past primitive skill selection task and planning remaining steps task.
63

---

Published as a conference paper at ICLR 2026
Toward achieving pick up the tool and in the cup', what's the nextt actionable task? Please answer in short phrase.
place the tool in the
To advance toward "close drawer', what should be done next? Please answer in a short phrase
push handle the drawer close drawer
What should bo immeddate next step to achieve 'press the water dispenser ser button"? '? Please answer in short phrase
Press the water dispenser button
With progress so far including" '1. pick the table, the glue stick from the table to the green cup',
up the glue
you put the glue stick in the green cup'? answer short phrase.
achieve
the glue stick the green cup
you've completed pick up black cloth on the table'. What's the next logical step
'wige cardboard box with the black eloth then take the power card answer in short phrase..
with the black cloth the box
With progress so far including '1. pick hoodie from back transfer the hoodie from the back the should you do next to achieve put the hoodie
chair?
answeri in short phrase.
place the hoodie seat the chair
Figure 36: Planning task and planning with context task.
64

---

Published as a conference paper at ICLR 2026
Which sourringohilee pick the lork the plate 7 Please unsweer with single option (A/R/C/D1
Which cornesparads the
of place
the ball with
While pick up the aleee of benadd froms the oven is being performed which of the images LA, B. C) illustratzs this state? Pleass answer with single optionn (A/B/C/0)
From A, D, which curreppndete the scmmefor scumpls lupen and then close the cover') Please unswer with single option (A/B/C/D1
D
Consideringthe which matchees the somnel Pleaee with single (A/R)C//))
ofject basket' best
Which wooden block holding down table' Plesseannwer with single option (A11)/1/1))
Figure 37: Scene understanding task and temporal understanding task.
65

---

Published as a conference paper at ICLR 2026
Are you expected to handle ppick up lid from table' step? Please wns striedy with Yes No.
No
Are you about work 'pick green block on the blue table' next? Please answer striedy with Yes or No.
Is piikk up the white cup from the drawer' of your upcoming efforts? Please answer strictly with Yes or No.
Na
Is 'transfer the foam pack from the cabinet counter' your upcoming efforts? Plea se answer strictly with No.
Yes
Is "place marker table' the task you should perform next? Please answerstrictly with Yes or No.
Is 'transfer the marker from the table to the top the cup' achievable next stage? Please answer strietly with Yes or No.
Figure 38: Discriminative tasks of task planning.
66

---

Published as a conference paper at ICLR 2026
A.7.4 CoT DATA FOR ROBOINTER- -LL
Flexible Chain-of-Thought Data. For training the ApplicityyConndtioonel E2E and Modular Planner-to-Executor settings, we construct Flexible Chain-of-Though (F-CoT) data. A representa- tive visualization provided below; practice, we serialize F-CoT as textual sequences as shown in Figure 39 and 40
Your tail Put mag Int plats Syerr current plane stap
tokloingg
10T To parn
Priveiv Terele
Blaadi on cucourtt acons plans ronaur das anglard pronds pu lolwing trommanie
tranater
Perrate:: verskr
Printiie Hamster
Portligg pla30
touk up the pluat bot hammthe and put cuurar then dhaeche
soulds iniveraiio
Subrosts pick the bear plustoo the
Suttall: tna taarafertee bear plashitp form drawer tablrop.
Subraoos piace the bear plushssg tata.
Sumarks pust the crnva nanda down
Sattakk tuthite drowe hancl down sa Stawer.
Figure 39: Visualization of the F-CoT.
Your taok ax Pick the beer plush toy from the thawer and put the counter, then close the drower Bused on the current acerss, please ression stop by stup and provid the follow inginformacoont Buboeskk
Primitivo Actions
Gripper Boxcs
Teeget Brows
Allardance Buses
Trajectory
Lut
Subtweke: pick
disww:
Primiton Astionee
Gripper Broee: 701, 77[).
Tangel Burese [[83) /0), 800)
Allordannce Boaas: [[90, 72), (SS,
Trajectory:
(67, 258, (90, (88,
Figure 40: Visualization of the textual chain-of-thougg reasoning.
67

---

Published as a conference paper at ICLR 2026
A.8 FURTHER APPLICATION, LIMITATIONS AND LLM USAGE STATEMENT
Further applications. Beyond training VLMs and VLAs, RoboInter-Date also supports broader research directions: (1) Expert generative models for individual intermediate representations. In contrast to datasets centered on a single annotation type, Robolnter- spans diverse embod- ied scenes while providing large-scale, high-quality annotation for each intermediate representation, enabling the pre-training of specialized generative models tailored to specific representations. (2) Human-robot interaction. Intermediate representations such traces, bounding boxes, and sub- task sequences naturally bridge high-level instructions and low-level control. Our dataset supports the development intuitive and precise shared-autonomy systems grounded in these representa- tions. (3) Embodied world model learning. All annotations in Robolnter-Data are temporally aligned with x40x360 raw manipulation videos. These intermediate representations can serve as structured control signals for controllable video generation, while the aligned information about the robot, objects, and environment provides strong supervision for learning structured embodied world models. (4) Video action model learning. The combination of action annotations, videos, and diverse intermediate representations makes Robolnter-Daa well suited for training video action models with rich, multi-level supervision.
Limitations. While the RoboInter suite offers broad coverage across tasks and scenes, current annotations are primarily collected fixed-base, single-arm robots, limiting embodiment diversity. Future extensions may incorporate more complex setups, such as mobile or dual-arm systems, to enhance cross-embodiment generalization. Additionally, while our planner exhibits strong reasoning capabilities, improving its robustness and generalization across unseen scenarios remains open challenge. The executor, though effective, relies on relatively heavy inference due to long-chain reasoning; future work may explore lightweight architectures and optimization strategies to improve deployment efficiency.
LLM Usage Statement. We employed large language models (LLMs) solely for grammar refine- ment and minor linguistic polishing. All LLM--asssseed edits were carefully reviewed and verified by the authors to ensure that fabricated content or unintended alterations to the original meaning were introduced. The research ideas, experimental design, data analysis, and conclusions presented in this work were entirely conceived and executed by the authors without LLM assistance.
68


Reviews:
[{"review_id": "pdCHnrVV0s", "reviewer": "Reviewer_ryjd", "summary": "The paper presents the RoboInter Manipulation Suite, which operationalizes a plan-then-execute paradigm for robotic manipulation. It first introduces RoboInter-Tool for semi-automatic, per-frame annotation, and uses it to build RoboInter-Data—a large-scale, scene-diverse dataset that includes object and gripper bounding boxes, contact frames, trajectories, affordances, atomic skills, and sub-tasks. Based on these annotations, the authors design RoboInter-VQA, which turns intermediate representations into concrete task types to train and evaluate VLMs’ embodied perception and planning. At the model level, the paper proposes RoboInter-VLA with three variants—implicitly conditioned end-to-end (IC-E2E), explicitly conditioned end-to-end (EC-E2E), and a modular Planner→Executor setup—and bridges planning and control with a composable intermediate “chain-of-thought” (F-CoT) in text or visual form. Experiments show that planners trained on the proposed data and VQA substantially outperform base and embodied VLMs on Where2Place, RoboRefIt, RoboVQA, and the RefCOCO family; in both open-loop (OLS) and real-world closed-loop (Franka, ID/OOD) evaluations, explicit intermediate supervision and modular designs yield faster convergence and stronger generalization.", "questions": "Technical Implementation Details**: Could you provide more detailed analysis of camera parameter estimation accuracy and 2D-3D projection errors? What is the quantitative inter-annotator agreement (e.g., Cohen's kappa) for different annotation types?\nWhat are the specific time costs and economic overhead for human-in-the-loop annotation? \nGiven that experiments primarily use Franka robots, how would you validate generalization to other robotic platforms ? Do you have plans for cross-embodiment evaluation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper makes important systematic contributions to the field of robotic manipulation. Originality: It represents the first large-scale systematic effort to address the data scarcity problem of intermediate representations in robotic manipulation, with RoboInter-Tool providing an innovative semi-automatic annotation solution. Quality: High-quality annotations for 200,000 videos are ensured through human verification and multi-round cross-validation, with comprehensive experimental design covering spatial/temporal understanding generation tasks as well as open-loop/closed-loop evaluation, achieving significant improvements across multiple benchmarks. Clarity: Experimental settings and evaluation metrics are clearly defined, and ablation studies help understand the contributions of different components. Significance: This work addresses a key bottleneck constraining the development of the \"plan-then-execute\" paradigm, provides valuable data resources and tools for the robotic learning community, and demonstrates good generalization capabilities in real-world experiments.", "weaknesses": "The accuracy verification of camera parameter estimation and 2D-3D projection is insufficient, lacking quantitative analysis of inter-annotator consistency and failure case studies. Meanwhile, current experiments are primarily based on the Franka robotic platform, lacking cross-platform validation and generalization capability evaluation in more complex scenarios, with limited scale and diversity in real-world experiments. The systematic ablation studies on F-CoT design and various intermediate representations are inadequate. The time costs and economic overhead of human-in-the-loop annotation have not been quantitatively analyzed and require more detailed evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8yKkNZHqD", "reviewer": "Reviewer_BJ8z", "summary": "This paper introduces the RoboInter manipulation suite, which includes RoboInter-Tool, a lightweigh tGUI for semi-automatic\n per-frame annotation of embodied videos,and RoboInter-Data,a human-verified dataset with over 200k episodes across 571diverse scenes. Based on them, This paper also introduces a RoboInter-VQA benchmark and there 'plan-then-execute' framework (IC-E2E / EC-E2E / Modular). Experiments show the pre-trained Planner improves open- and closed-loop generalization.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduced RoboInter-Data is large-scale, including 200k episodes, 91M frames, 571 scenes.\n\n2. This paper offers rich intermediate representation categories (sub-tasks, skills, affordances, boxes, traces, contact frames, etc.) during constructing RoboInter-VQA. These representations are critical for improving the abilities (like spatial intelligence) of embodied brain and explaination during bulding VLA systems.\n\n3. This paper also introduces a full toolchain (RoboInter-Tool): an open-source GUI integrating SAM-2 and GPT-4o for semi-automatic segmentation, tracking and language annotation, reducing labeling cost.\n\n4. Based on these rich intermediate representation, this paper also develop three flexible training framework (IC,EC,Module).\n\n5. Extensive are conducted in multimodal benchmarks, RoboInter-VQA, and real-world tasks.", "weaknesses": "1. The paper does not introduce architectural or training-paradigm innovations for vision-language-action (VLA) models. Its contributions mainly lie in demonstrating that the proposed intermediate representations can be effective within existing VLA and plan-then-execute frameworks.\n\n2. While the paper mentions prior works that employ explicit chain-of-thought (EC) representations (e.g., ECoT), it does not provide sufficient discussion or comparative analysis with these conceptually related methods. This omission makes it difficult to position RoboInter’s contributions relative to previous efforts.\n\n3. In most real-world experiments, the IC variant performs better than the EC one, raising uncertainty about the practical utility of explicit intermediates. A more systematic analysis of when and how different forms of intermediate representation benefit VLA learning would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9bZiHkbrKC", "reviewer": "Reviewer_v8Xb", "summary": "This paper presents RoboInter Manipulation Suite, a unified resource of data, benchmarking, and training vision-language-action (VLA) models. A key contribution of this paper is to propose a new dataset for improving embodied reasoning capabilities of VLA models. The dataset consists of fine-grained annotations of 200k video data, including language, key frame, target object, gripper coordinate annotations. The annotations were labeled in the RoboInter-Tool. Based on this data, the authors designed the visual question answering (VQA) tasks that require vision-language models to answer spatial or temporal questions. Finally, this paper also proposes a VLA model trained on the collected data. Experiments show that RoboInter-VLA improves reasoning capabilities on both third-party benchmarks (e.g., RoboVQA) and the proposed RoboInter-VQA tasks. RoboInter-VLA was also deployed to the real-world environment, and it showed improved performance, especially in out-of-distribution robot manipulation tasks.", "questions": "- [Q1] Will the RoboInter-VQA dataset be made publicly available?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- [S1] The paper proposes a new data, a benchmark, and a VLA model for embodied reasoning, which could be a great resource for the community. \n- [S2] Experiments validate the effectiveness of the collected data on both other benchmarks and the proposed benchmark.\n- [S3] The paper is well-written and easy to understand.", "weaknesses": "- [W1] While a plan-then-execute framework improves reasoning and interpretability by generating intermediate representations, it inherently hurts the latency of robotic systems. The paper does not provide any analysis regarding the inference speed of RoboInter-VLA compared with other VLA models. This could improve the understanding of the trade-off between reasoning capabilities and real-time performance in robotic systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiYk4GFKcj", "reviewer": "Reviewer_hDxo", "summary": "This paper focuses on the plan-then-execute paradigm for robotic manipulation and tries to fill gap of lacking fine-grained intermediate supervision with the proposed RoboInter Manipulation Suite. RoboInter Manipulation Suite contributes in three aspects: 1) RoboInter-Data, a human-veriffed dataset with over 200k episodes across 571 diverse scenes, 2) RoboInter-VQA, 8 spatial and 20 temporal embodied QA categories to benchmark and enhance the embodied capabilities of current large vision-language models, and 3) RoboInter-VLA, a ffexible plan-then-execute framework with modular and end-to-end variants that link planning to execution. Experiments indicate that RoboInter-Data significantly improves the model's reasoning capabilities for embodied tasks. In closed-loop evaluations, RoboInter-VLA is shown to effectively complete both in-distribution and out-of-distribution tasks when compared to Pi0 and OpenVLA.", "questions": "1. The RoboInter-Modular variant demonstrates superior performance in open-loop evaluations. However, it is not included in the closed-loop evaluation. What are the reasons for this exclusion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The dataset makes a significant contribution to the plan-then-execute paradigm and can also be utilized in other research paradigms.\n\n2. Comprehensive experiments validate the contributions of the dataset effectively.\n\n3. The insightful analysis of the variants of RoboInter-VLA provides valuable inspiration for other researchers in this field.", "weaknesses": "1. There is a lack of evaluation of other plan-then-execute methods trained on the proposed dataset, which would provide a more comprehensive validation of the dataset's contributions.\n\n2. There is insufficient analysis regarding which types of intermediate representations (subtask, primitive, target box, affordance) are most beneficial for robotic manipulation tasks. Are all the mentioned representations utilized as inputs to RoboInter-VLA during both open-loop and closed-loop evaluations?\n\n3. An analysis of inference time for RoboInter-VLA is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)