Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints
Authors: Jianshu Hu
Lidi Wang
Shujia Li
Yunpeng Jiang
Xiao Li
Paul Weng
Yutong Ban
Venue: ICLR 2026 Poster (2026)
Keywords: 3D manipulation
Imitation Learning
Coarse-to-fine Policy
Primary area: applications to robotics, autonomy, planning

Abstract: Hierarchical coarse-to-fine policy, where a coarse branch predicts a region of interest to guide a fine-grained action predictor, has demonstrated significant potential in robotic 3D manipulation tasks by especially enhancing sample efficiency and enabling more precise manipulation.
However, even augmented with pre-trained models, these hierarchical policies still suffer from generalization issues.
To enhance generalization to novel instructions and environment variations, we propose Coarse-to-fine Language-Aligned manipulation Policy (CLAP), a framework that integrates three key components: 1) task decomposition, 2) VLM fine-tuning for 3D keypoint prediction, and 3) 3D-aware representation.
Through comprehensive experiments in simulation and on a real robot, we demonstrate its superior generalization capability.
Specifically, on GemBench, a benchmark designed for evaluating generalization, our approach achieves a 12\% higher average success rate than the SOTA method while using only 1/5 of the training trajectories.
In real-world experiments, our policy, trained on only 10 demonstrations, successfully generalizes to novel instructions and environments.

Published as a conference paper at ICLR 2026
GENERALIZABLE COARSE-TO-FINE ROBOT MANIPU- LATION VIA LANGUAGE-ALIGNEI 3D KEYPOINTS
Jianshu Hu, Lidi Wang, Shujia Li, Yunpeng Jiang, & Yutong Ban
Global College
Shanghai Jiao Tong University {his1998,, lidiwang, sjtu7264970, jyp9961, ybanjesjtu. edu.s..uuud....... cn
Xiao Li
Paul Weng
School Mechanical Engineering Shanghai Jiao Tong University (sjtu_lixiao] Esju. edu.cnn
Digital Innovation Research Center
Duke Kunshan University
(paul hernl..wweggeuukkkuuushhhee eddu. aedu.cn cn
ABSTRACT
Hierarchical coarse-to-fine policy, where coarse branch predicts region of inter- est guide fine- gr-gggrined action predictor, has demonstrated significant potential in robotic 3D manipulation tasks by especially enhancing sample efficiency and enabling more precise manipulation. However, even augmented with pre-trained models, these hierarchical policies still suffer from generalization issues. To en- hance generalization novel instructions and environment variations, we propose Coarse-to-iie
Manguage-Alignee manipulation Policy (CLAP), framework that integrates three key components: 1) task decomposition, 2) VLM fine-tuning for 3D keypoint prediction, and 3) 3D-aware representation. Through comprehen- sive experiments in simulation and on a real robot, we demonstrate its superior generalization capability. Specifically,
GemBench, benchmark designed for evaluating generalization, our approach achieves a 12% higher average success than the SOTA method while using only 1/5 of the training trajectories. In real-world experiments, our policy, trained on only 10 demonstrations, success- fully generalizes novel instructions and environments.
INTRODUCTION
Robot learning, especially via imitation learning, has demonstrated promising success in enabling robots to solve complex 3D manipulation tasks (Intelligence et al., 2025; Liu et et., 2024). However, scaling these methods to a broader range of real-world applications (e.g.. industrial, service, or home robotics) requires enhancing both (GI) their generalization to environment variations, and (G2) their skill compositional generalization. Indeed, GI is necessary, because deployed robots need to be able to operate in settings (e.g., object or background variation), while G2 is highly desirable, so that trained robots can tackled new tasks by composing previously-eaareed skills. To achieve G1 and G2, the robot needs be endowed with combination of capabilities, such as scene understanding, reasoning or planning, and sample efficient techniques, since robotics data is costly collect.
manipulation, exploiting preferably
In this paper, we focus on one type of 3D manipulation policies, called coarse-to-fine policies (Gualtieri Platt, 2020: James al., 2022: Ling et al., 2024; Goyal et al., 2024: Gervet et al., 2023; Liu et al., 2025), because they achieve superior precision in manipulation tasks while enjoy- ing strong sample efficiency. These policies process 3D observations (or 3D scene representations) using a hierarchical architecture whose higher-level coarse branch identifies a region of interest for lower-level fine-grained branch to focus on and predict a final action. Typically, the coarse branch is trained to predict a 3D keypoint, which serves the center for cropping and zooming into the original 3D observations. To help with visual understanding and to some extent spatial
indicates corresponding author

---

Published as a conference paper at ICLR 2026
Prompt: Take cube from the drawer
Generalize
Align
Decompose
Novel Objects
Step1: Open the drawer
Roason
Stap2:
Novel Tasks
Stap3: Put the cube on the table
Long horbson compositions
Figure 1: Intuition of CLAP. Our method achieves strong generalization ability by decomposing tasks into step-wise language instructions, each aligned with 3D keypoint.
reasoning, recent work (Li ct al., 2025b; Fang et al., 2025) has extended this approach to exploit Vision-Language Models (VLMs) (Beyer et al., 2024) or visual foundation models (Ravi et al., 2025). However, the performance of these obtained methods is still limited in terms of generalization capability (Gl and especially G2), indicating that their scene understanding and reasoning capabilities are actually still rudimentary. Our experimental study suggests that this primarily due combination of various issues (depending on the method), such as domain shift between pre-training and robotic images, inadequacy of pre-trained models to predict 3D keypoint, poor adaptation object variations, or enplo-exploitation of the planning ability of VLMs.
pre-trained
To address these limitations and issues, we propose Coarse-to-fin Policy (CLAP), novel coarse-to-fine 3D manipulation policy. In contrast to previous coarse-to-fine policies, CLAP includes a novel architecture for the higher-level branch, which we name coarse task planner, and a novel implementation of the lower-level fine grained action predictor, both leveraging pre-trained models.
Sanguage-Alignee manipulation
The coarse task planner, implemented as a VLM, introduced to play the additional role of task planning. Before the usual 3D keypoint prediction, it decomposes task into step-wise language instructions, representing basic skills. This change allows both 3D keypoint and action predictions to depend step-wise instructions instead of the whole task description, which promotes skill compositional generalization (G2). The training of this coarse task planner consists of three parts to reinforce its scene understanding and reasoning capabilities. First, the pre-trate mr-itrained VLM fine- tuned on language plans of different tasks to directly improve compositional reasoning. Second, specialized for 3D keypoint prediction by fine-tuning it perform sequential reasoning process: first localizing task-related objects, then generating the step instruction, and finally predicting a corresponding 3D keypoint. Finally, to further boost its scene understanding capability, the VLM is further fine-tuned with an auxiliary task of 3D object detection, using an additional dataset of object positions. Together, these components form a comprehensive pipeline that significantly enhances the generalization ability of the coarse-to-fine policy to object variations (G1) and novel tasks (G2).
The fine- mne-grained action predictor takes as input both the step instruction and the multi-vier RGB-D images and outputs an action. It is implemented with specialized pre- teaiira models to improve sam- ple efficiency and increase its precision during manipulation. More specifically, step instruction and RGB images are processed using a pre-trained visual-language encoder, ensuring the two modalities are well-aligned. The depth information is processed by a dedicated encoder and augmented with 3D position embeddings beddings, which we call 3D-aware representation, are fused via a Multi- View Transformer (Goyal et al., 2023) to predict the final actions.
help better align 3D and 2D image information. All the obtained em-
To evaluate the performance of our method, we run experiments both simulation and real-world. For simulation, the generalization ability of multi-task language-conditioned policies across varying difficulty levels. Our approach outperforms the state-of-the-art method, achieving 12% higher average success rate with only 1/5 of the training trajectories. In real-world experiments, our method demonstrate strong generalization ability to novel tasks and object variations with only 10 demonstrations per task.
GemBench (Garcia et al., 2025), benchmark specifically designed assess
Contributions
1. We introduce a novel coarse-to-fine 3D manipulation policy, as shown in Figure 1, with two main innovations: (1) tasks are decomposed into step-wise language instructions to
2

---

Published as a conference paper at ICLR 2026
promote compositional generalization ability; (2) action inference is performed via rea- soning step to improve generalization to object variations.
2. We design a finetuning pipeline that effectively adapts a pre-trained VLM to 3D keypoint prediction and incorporate a 3D-aware representation in the fine- fine-grained action predictor, overcoming the issues observed in previous methods.
3. Empirical evaluations in simulation and on real robot demonstrate lfate-of-th-aut perfor- mance in both robustness to visual and object changes and generalization to unseen tasks.
RELATED WORK
In this section, we discuss the related works in the field, including viskon-kanguage--ctiion models, 3D manipulation policies, and coarse-to-fine policies.
Lision-Language-AAttion (VLA) models Training VLMs (OpenAI, 2024; Beyer et al., 2024; Bai et al., 2025; AI, 2024) on vast internet-scale image-text corpora has led to remarkable capabilities in image understanding, excelling at tasks like image classification, object detection, and visual question answering tasks. However, applying a similar training strategy directly to robotics presents a challenge due to the relatively scarce robot trajectory data. A prominent solution is to transfer the knowledge from pre-trained VLMs by fine-tuning them on robot data. This approach is the foundation for recent VLA models (Driess et al., 2023; Brohan et al., 2023; Kim et al.. 2024; Octo Model Team et al., 2024; Intelligence et al.. 2025; Wen al., 2025a; Shukor et al., 2025; Liu al., 2024; Wen et al., 2025b; Li et al., 2024; Cheang et al., 2025; Team et al., 2025; NVIDIA et al., 2025), which are fine-tuned on large diverse datasets of robot trajectories. Such extensive training strengthens generalization to novel objects, environments, and tasks. However, since they commonly use multi-view 2D images as visual input, learning to reason in 3D space from 2D images alone is data-intensive. This leads to sample inefficiency and low success rates on some tasks. Recent work has sought to more explicitly incorporate 3D information (Li et al., 2025a; Qu et al., 2025; Zhen et al., 2024) or introduce Chain of Thought (Mu et al., 2023; Zawalski et al., 2024; Zhao et al., 2025) to enhance the 3D) reasoning ability. However these directions remain relatively underexplored within the VLA paradigm. Our method, which fine-tunes a pretrained VLM coarse task planner and predicts the final action with a fine-grained action predictor, can also be viewed as a VLA model. In contrast to other VLA approaches, we propose specific training and inference techniques to better align pre-trained VLMs to 3D manipulation, further enhancing generalization (G1-G2) while retaining the sample efficiency inherent to hierarchical coarse-to-fin policies.
3D Manipulation Policy 3D) manipulation policies (Shridhar et al., 2022; Gervet et al., 2023; Jia et al., 2025; Zhu et al., 2025; Wang et al., 2024b; Ze et al., 2024; Wang et al., 2024a; Goyal et al., 2024; Ke al., 2024; Chen al., 2025a:b; Fang et al., 2025; Li al., 2025b; Garcia et al., 2025) directly work with 3D inputs and outputs. They generally include structured architectures that con- struct a 3D representation of the scene, leading higher sample efficiency and better generalization to new camera viewpoints. For example, PerAct (Shridhar et al., 2022) explicitly represents the scene with a voxel representation. Gervet al. (2023) and Ke et al. (2024) process RGB images with pre-trained image encoder and lift 2D features to 3D by aggregating with depth information. An alternative approach (Jia et al., 2025; Goyal et al., 2024) is to project point clouds into canonical virtual views and use the resulting multi-viww images as input. Explicitly exploiting 3D information allows these models to achieve high success rates with much less training data, which can be further reduced by enforcing hierarchical structure like in coarse-to-fine policies.
Coarse-to-fine Policies Gualtieri & Platt (2018; 2020) first propose this coarse-to-fing scheme for pick-and-place tasks. Subsequent work has considered more general tasks and explored various 3D representations, such as voxel observations (James et al., 2022; Liu et al., 2025), 3D feature fields (Gervet et al., 2023), and multi-view images (Goyal et al., 2024). Ling et al. (2024) apply the coarse-to-fine architecture to handle noisy point clouds. Among these, Robotic View Transformer 2 (RVT2) (Goyal et al., 2024) is an effective language-conditione conditioned multi-tasé policy, demonstrating strong performance in both training and inference efficiency by using multi-view images projected from canonical views. However, RVT2 is trained from scratch, which limits its generalization ability to visual perturbations and task variations. Subsequent efforts built upon this work have sought to
3

---

Published as a conference paper at ICLR 2026
overcome these limitations. Existing works (Li et al., 2025b; Fang et al., 2025) have attempted to enhance generalization through strategies such as: pre-training on object detection datasets (Yuan et al., 2024) or integrating encoders from powerful visual foundation models like Segment Anything Model (Ravi et al., 2025). In contrast, we achieve this by introducing a novel architecture, where tasks are decomposed into step-wise language instructions for skill compositional generalization and design specific training and inference pipeline to leverage pre-trained models in both coarse and fine-grained branches.
3 BACKGROUND
In this section, we first briefly recall the multi-task imitation learning set-up, introduce coarse-to-fin policy, and then present Robotic- Tra-TTranffomerr 2 (RVT2) (Goyal et al., 2024), state-of-the-an coarse-to-fime policy that serves as the foundation for our method.
In multi-task imitation learning, a dataset D [(T,, L.) I Nj is available for pairs of robot demonstrations T; and task description L, A robot demonstration is a trajectory т; = (oo, oo, 01, .1.... containing sequence of observations or and corresponding expert actions az- Observations include multi-view RGB-D images and gripper status indicating whether it is close or open. Actions denote the state of the end-effector, which contains the 3D position pt (Iz, yt, ze) of the gripper, the orientation of the gripper and gripper status.
A coarse-to-fine policy contains a coarse branch and a fine-grained branch, where the coarse branch predicts a 3D keypoint as the center to zoom in the 3D observation and the fine-grained branch uses the refined observation to predict the target action. Such policy trained according to the key-frame based imitation learning framework (Johns, 2021; Shridhar et al., 2022; Goyal et al., 2024). Specifically. key-frames identifies timesteps in a trajectory when an important action, like grasping or placing, occurs. In practice, they are usually heuristically defined for each trajectory. With these key-frames, a trajectory is segmented into K subsequences (oo, ..., 0t2 at ) ..., 111 0tx-1+1; ..., Otx atx), where the htth key-frame occurs at time step tg, from which we can extract sequence of key-frame actions (At,, atg). In this framework, the goal is to train policy π predict the key-frame action Oeg at the closest next key- ney-frame of timestep tx given an observation oe and a task description Lg:
observations and actions
π (oo, Li) + are for b St <t< lx
(1)
The predicted actions are executed by motion planner, which moves the robot to the desired state, generating thus the intermediate actions in a trajectories. In coarse-to-fie to-fine policies, the 3D position Pt, output by the coarse branch for the next key-frame is typically used as the 3D keypoint to zoom into the observation for the action predictor.
In RVT2, multi-view RGB-D images are first aggregated into point cloud, which is then projected into three canonical views: front, left and top. These three views are orthogonal to each other, which allows mapping between pixel positions in these views and a 3D position in the scene. Each pixel in the projected images contains 3-channel RGB values, 1-channel depth value and its corresponding 3D position in the global coordinate. In the coarse branch, the projected images tokenized using convolutional layers while the task description and robot states (e.g., gripper status) encoded by pre-trained language encoder and trainable Melti-Layyer-Peceptrree respectively. All these tokenized features are fused via Multi- View Transformer (Goyal et al., 2023). The image tokens in the output of the transformer are then processed by upsampling layers to predict heatmaps, from which 3D keypoint is extracted. The keypoint from the coarse branch is used to zoom in and crop the point cloud while the cropped region is again projected into the canonical views. The refindd observations along with the same task description are processed by the fine-grained branch, implemented as another multi-view transformer with different weights, to predict the final actions. While RVT2 achieves strong sample efficiency and enables precise manipulation via projections to canonical views and its coarse-to- to-fine architecture, it is trained from scratch and therefore does not leverage recent pretrained large models. In addition, its architecture design does not fully exploit common skills among tasks. As a result, suffers from deficient generalization to visual changes, object variations, and novel tasks.
4

---

Published as a conference paper at ICLR 2026
Course Task Planner
Tink Deseriptien: Stack and
Fine-grained Action Prodictor
Generalization
ZoonI
Language Instraction Decompestion
Indtaction:
The mustt
Langunge Insruction
Inguts
loven the
Vasal
on the red
Step The robot moves la gilaper above
Sup The prppar greqp....
Step
Algntd
Objjct Vartarions
RGB
Depth
SigLIP
DANOY2
Langragy
Emhedding
Lunguag-sllgnod Kespaint Predictor
Herken
Contitatioos
3D Key Prints
Real-world Experiments
Scenu Uledenttnding:
Reason
Stup Instructione: The Resson nbot lnwers the grocs upp on fh nal Lap.
The pripper is pustiarn yl. The syan
Rotate Open Closs
Heatrnsp
Figure 2: Overview CLAP. We propose novel coarse-to-fine 3D manipulation policy, compris- ing of a coarse task planner and fine-grained action predictor. The coarse task planner reasons about the task plans and the positions task-related objects to generate language-aligned 3D key- points. The Aine-grinnd action predictor fuses the corresponding step instruction with a 3D-aware visual representation from refined observations to predict the final action.
METHOD
Our hierarchical policy consists coarse task planner and a fine-grained action predictor, as shown in Figure 2. To promote compositional generalization, we first decompose tasks into step- wise language instructions, each describing the motion of the robot between two consecutive key- frames. This enables the coarse task planner to perform language-guide the fine-grained action predictor to learn and reuse common skills across tasks. Moreover, we adapt pre-trained VLM to 3D) keypoint prediction in the coarse task planner, by finetuning it with sequential reasoning procedure. The VLM predicts the pixel position each view and we follow the prior work (Goyal al., 2024) to map them 3D position. The pretrained VLM is trained to first reason about the positions of task-related objects, then generate step instruction and finally predict the corresponding 3D keypoints. To enable zero-shot generalization ability novel objects, we add an auxiliary task of 3D object detection by augmenting the training data with additional object positions dataset. Finally, in contrast to RVT2, the coarse and fined-grained branches are implemented with different architecture, since they play different roles. Thus, for the latter, we utilize specialized pre-trained visual foundation models construct 3D-aware representation. The design choices of both the coarse task planner and the fine-grained action predictor are detailed in the following sections.
planning while allowing
4.1 COARSE TASK PLANNER
Prior coarse-to-fine policies condition all actions within trajectory on a single high-level task de- scription, limiting compositional generalization. To address this, we leverage pre-trained VLM, denoted fo. to decompose a high-level task description L into hep-wise language instructions L (2, Lk which naturally align with the key-frame based trajectory segmentation. Each trajectory segment (ot, 1it--+1, 1+1, 141,.... Ot,, at.) and key-frame action ate have corre- sponding step instruction l describing the motion of the robot within the segment. For example, the task "open the top drawer" is decomposed into following step instructions: 4; "The robot arm lowers itself to align with the handle the top drawer". εz: "The robot arm grasps the top drawer's handle firmly". and fz: "The robot pulls the handle back, smoothly opening the top drawer".
In addition to task decomposition done at the beginning. at every execution timestep. the VLM fo is also exploited predict both the step instruction f, (used novel input of the action predictor) and its language-aligned 3D keypoint Pt, (for cropping a region of interest, as usually done in coarse-to-fine policies). We discuss next how this prediction can be realized effectively.

---

Published as a conference paper at ICLR 2026
Task decomposition enables reasoning about task plans before predicting actions. However, directly training the model fe to simultaneously generate a task plan L and predict step instruction L and 3D keypoint Pt. given inputs multi-view images obtained from observation ot, and high-level task description I., (i.e., fo(or._ L) + (L. P )), does not ensure generalization to novel instructions. Previous studies (Kim et al., 2024; Intelligence al., 2025; Zhao et al., 2025; Gao et al., 2025) indicate that VLAs exhibit strong bias towards visual inputs, due to the richer information embedded in the visual inputs. This reliance often causes failures to follow novel language instructions. Under our data-scarce training setting, this issue is intensified by the limited diversity of language instructions. To address this issue, we propose decoupling task planning from keypoint prediction via two-round inference protocol. First, a purely textual query generates language plan, the sequence of all step instructions L. Second, the visual inputs augmented with this plan are used to predict the corresponding step instruction and keypoint:
fo(L.) + C, fo C) - Ptc).
This approach not only mitigates visual bias but also enables training with an auxiliary dataset of language plans, which serves as a manual to enhance compositional task reasoning.
For plan L corresponding to a task description L, directly fine-tuning a VLM to predict both the step instruction 4 and its language-aligna aligned 3D keypoint Рt, reveals to be insufficient due to the inadequate alignment of the visual-textual embeddings of the VLM for this 3D keypoint predic- tion. Instead, inspired by Chain-of-Thourr et al., 2025) for robotics, we design a reasoning process by training our model to first reason about the pixel positions of task-related objects, then generate a step instruction and finally predict the corresponding keypoint, changing the second step of Equation (2) to:
reasoning (Mu al., 2023; Zawalski et al., 2024; Zhao
folon_,,, L) (Pobj- Pte),
(3)
where Poby are the 3D positions of the task-related objects.
We further observe that for long-horizon, especially repetitive tasks (e.g., "stack several blocks"), only providing the entire task plan whether as input or output can degrade the performance. This often causes the model to generate repetitive sequences until reaching the output length limit or to struggle in determining the next step from an excessively long plan. To mitigate this, we introduce two ideas. First, we provide to the VLM an additional input: the step instruction predicted in the last timestep, which serves as short-term memory cue to contextualize the current situation. Moreover,
we decompose the plan into sub-tasks, i.e., L = (ci = (2., (x),
(n+2,,.. .......
For instance. "stack the blue and yellow cup on the red cup" is decomposed into two sequential sub-tasks: L1 "stack the blue on the red cup" followed by L2 "stack the yellow cup on the red cup". The model is trained to generate only the task plan of current sub-task, preventing repetition and improving focus:
(4) where é is the step instruction predicted in the last timestep and L. is the language plan of the m sub-task. At the beginning of trajectory, where previous timestep exists, é is defined as "the robot is currently at the initial state" to indicate the initial state.
fo + C.. fo(o12-1 Cm. 2) + (Pobs, Lex. Pte),
To further enhance the generalization ability of the coarse task planner to object variations, include an auxiliary task of predicting the object positions. We randomly initialize diverse envi- ronments and record the RGB-D images of the scene along with the 3D positions of the objects. Following the same pre-processing, both the RGB-D images and the 3D positions of the objects are projected into canonical views. The projected multi -ieti images and pixel coordinates of the object positions in each view are used to construct an object position dataset. This dataset is then utilized co-train the VLM, reinforeing its spatial understanding and improving zero-sha shot generalization to object variations.
4.2 FINE-GRAINED ACTION PREDICTOR
The fine-grainn ine-grained action predictor uses a predicted step instruction E,p- instead of the original high- level task description L, enabling more precise and generalizable skill learning. Considering the significant domain shift between the images focused around predicted Pt,, from those used to pre- train standard VLMs, we decide to employ instead pre-trained specialized encoders process these

---

Published as a conference paper at ICLR 2026
inputs. Our feature encoding pipeline consists of three stages to construct unified 3D-aware and Lingugge-aligne representation. First, RGB images and step instructions are processed through vision- vision-language encoders to establish semantic alignment between visual and textual inputs. Second, depth images are encoded separately to extract explicit geometric structure. Finally, we generate a 31 position embedding from pixel-wise 3I) coordinates to incorporate spatial awareness. These components are combined to form a 3D-aware, language-aligned representation for downstream fine-grained action prediction, following the architecture of prior work (Goyal et al., 2024).
EXPERIMENTS
We now present the experimental settings and results simulation and real-world experiments.
5.1 SIMULATION RESULTS
Experimental Set-up Our method is evaluated on GemBench (Garcia et al., 2025), benchmark specifically designed for evaluating the generalization ability of policy, A dataset containing 100 demonstrations per task along with a task description per trajectory is prepared for training. This training set contains 16 tasks with 31 variations. Within trajectory, 4 cameras front, left shoulder, right shoulder and wrist to collect RGB-D images as the observations. The resolution of the original RGB-D images is 256x256 while the resolution of the projected images 224x224. Instead of evaluating on in-distribution tasks and variations, GemBench designs an evaluation containing 4 levels of tasks, where different elements are varied:
placed at the
Placements (L1: same 16 tasks (31 variations) as in training set, but with novel object placements. Rigid Objects (L2)): 15 novel tasks (28 variations) with newly-colored or -shaped rigid objects. Articulated Objects (L3): 18 novel tasks (21 variations) with appearance object variation.
Long-horizon Tasks (L4): novel long- horizon tasks (12 variations).
The specific configuration for tasks used for training and evaluation in GemBench are listed in Appendix A.I. We explain in detail how create the language plan dataset and object position dataset in Appendix A.2. Following the evaluation setting in GemBench (Garcia et al., 2025), all trained models are evaluated with 20 episodes per task variation per seed, and 5 different seeds are used.
In our method, we finetune Qwen 2.5 VL-3B (Bai et al., 2025) as the coarse task planner. It is LoRA fine-tuned (Hu et al., 2022) with the object keypoint dataset, language plans, and robot trajectories. We use SigLIP (Zhai al., 2023) extract features from the RGB images and step instructions, leveraging its language-aligned representations. For depth images, we use DINOv2 (Oquab et al., 2023), which excels capturing geometric structures like edges and contours. We further enhance these features by using the 3D coordinate of each pixel to construct 3D position embedding. The hyperparameters, such batch size and learning rate used in training are listed in Appendix A.3.
To construct the fine-tuning dataset from robot trajectories, we design sampling strategy to choose samples from the trajectories. Apart from key-frame pairs of observation and action (Otz. (te.r); ) RVT2 augments the training data by sampling observations every n frames (e.g., every 10 frames). However, this results in an uneven number of samples per trajectory segment, due to the vary- ing length of each segment. We initially attempted to sample observations within a window (ot.-m...., Otpy (..p around the time step t, of the Ath key-frame. However, observations at time steps before a key-frame and after key-frame are visually similar while corresponding to distinct actions. For example, 02 and ou++1 are similar while their corresponding keypoints are the gripper positions t and f respectively. Using all observations around the kramfrsmes to fine-tune the VLM risks confusing it. Finally, we choose to use observations (Oty) (0.00t..Ot +m) at the time steps immediately following each key-frame. We empirically choose m 5 in all experiments.
Main Results The evaluation results on GemBench are summarized in Table 1, reporting the av- erage success rate for tasks at each generalization level. The detailed success rate for each task are recorded in Appendix A.4. The experimental results demonstrate the strong generalization ability of our method to novel tasks and object variations, as indicated by the performance gain on Level- 2, Level-3, and Level-4 tasks. Our method achieves overall success rate 12% higher than prior

---

Published as a conference paper at ICLR 2026
Models
Avg. Success
L1
1.2
L33
L4
HiveFormer (Guhur et al., 2023)
30.4
60.3 = 1.5
26.1 = 1.4 35.1
1.7
0.0 0.0
PolarNet (Chen et al., 2023)
38.4
77.7= 0.9 37.1
1.4 38.5
0.1 0
3D Diffuser Actor (Ke al., 2024)
44.0
91.9=
43.4 37.0 +
0.0
RVT2 (Goyal et al., 2024)
44.0
89.1 51.0
2.3
=
0.0
3D-LOTUS (Garcia et al., 2025)
45.7
94.3
49.9 =22 111
0.3
3D-LOTUS++ (Garcia al., 2025)
48.0
68.7 0.6 64.5 0.9 41.5 1.8
17.4
BLi (Li et al., 2025b)
50.0
=1.1=11 65.0
= 1.3 43.8
0.0
0.0
CLAP (Proposed)
62.0
83.9 83.2 1.9 49.6
31.4
Table 1: Multi-Task Performance GemBench. Here the average success rates of 4 levels of evaluation tasks from Gembench, Except CLAP, we use the results reported in BridgeVLA
Exp ID
Language Plan Object Keypuints Last
Reason
Reason Pretrained
Level
Level
Level Level
Avg.
Data
Data
Step Plan
Objects
Encader
Succ.
X
X
X
X
K
V 86.9
68.2
36.4
0.4
48.0
2
X
X
X
X
/
81.8
74.8
39.0
0
48.9
X
X
V
X
4
83.4
66.1
41.9
2.0
48.3
/
/
/
84.8
81.4
43.8
30.4
60.1
/
X
82.4
79.1
25
57.8
83.8
83.2
49.6
31.4
62.0
Table 2: Ablation study of CLAP on GemBench. Here are the average success rates of 4 levels of evaluation tasks from Gembench under different training settings.
state-of-the-arr method (Li et al., 2025b). Notably, this improvement is obtained using only 20 tra- jectories per task variation for training, significantly fewer than the 100 trajectories used by other baselines. Furthermore, our design leads to substantial performance gain the most challenging Level-4 tasks, where several baselines methods fail consistently. We also include an experiments on the effects of using different number of trajectories in Appendix A.5.
Ablation We further experimentally validate the design choice for both coarse task planner and fine-grained action predictor on GemBench. The configurations are detailed below and correspond- ing results presented in Table 2. A specific ablation the inputs of the fine-grained action predictor is include in Appendix A.5.
1) Base the base version (corresponding to Exp ID 1), the coarse task planner trained with only the robot trajectories to predict step instruction and the corresponding keypoints. We use this version as a baseline to ablate our method.
2) Object Reasoning To adapt the pre-trained VLM for 3D keypoint prediction, we introduce structured reasoning procedure where the model first localizes task-relevant objects before predict- ing the step instruction and its corresponding keypoint. We evaluate the efficacy of this object po- sition reasoning in Exp ID 2. A comparison with the base model (Exp ID 1) reveals performance improvement Level-2 and Level-3 tasks, indicating enhanced generalization to object variations.
3) Language Plan Reasoning The proposed task decomposition enables a two-round conversation, where a language plan is first generated through textual reasoning, followed by keypoint prediction. This approach also permits the inclusion additional language plans during training to enhance compositional generalization. Compared to the base model (Exp ID 1), this version shows improve- ments on Level and Level tasks, demonstrating stronger generalization to novel task variations.
4) Include Previous Step Instruction Previous step instruction is included in the input as short- term memory to help contextualize the current status. This design yields performance gains across Levels 2, 3, and 4, with particularly notable improvements on long-horizon tasks in Level 4.
5) CLAP w/o Pre-trained Encoder An ablation study (comparing Exp ID and Exp ID 6) the coarse planner confirms that incorporating the 3D-aware representation contributes to performance gains all generalization levels.
6) CLAP Our method integrates components mentioned above. comparison between Exp ID
4 and ours can further validate the performance gain from adding the object position dataset.
8

---

Published as a conference paper at ICLR 2026
Training Setups
Evaluation Setups
Place shape in sorter
Put block in cup
Original
Distractions
Combinations
Open drawer
Put block drawer
Table Color Variation
Relighting
Novel Object Category
Overview of the tasks in real-world experiments. There are four training tasks: put shape in shape sorter, put block in cup, open drawer, put block in drawer. We evaluate the same tasks under different visual perturbations and novel tasks designed based on the training tasks.
Figure 3:
No Variation
Table Color
Distracted Objects
Light Strength Average Succ.
RVT2
CLAP
RVT2
CLAP
RVT2
CLAP
RVT2
CLAP
RVT2
CLAP
place shape in shape sorter
60%
60%
35%
50%
30%
40%
20%
50%
36.2%
50%
put block cup with same color
40%
100%
20%
70%
20%
80%
20%
%
25%
82.5%
open drawer
100%
100%
85%
95%
100%
100%
0%
100%
71.2%
98.7%
put block in open drawer
90%
25%
90%
20%
70%
0%
90% %
21.2%
85%
block in cup with different color
20%
100% %
15%
70%
0%
70%
10%
80 %
11.2%
80%
shape open drawer
30%
80%
20%
65%
10%
70%
0%
80 %
15%
73.7%
shape in cup block in close drawer
0%
80 %
0%
70%
0%
60%
0%
70 %
0%
70%
0%
90%
0%
75%
0%
70%
0%
80 %
0%
78.7%
average success rate
36.2%
87.5
25%
73.1%
22.5%
70%
6.2%
78.7% %
22.5% 77.3%
Table Real-world Performance. Here the average success rate under different generalization settings for real-world experiments.
5.2 REAL-WORLD EXPERIMENTS
Experimental Setting We keep the training settings the same as in the simulation and list key modifications here. In the real-world experiments, collect RGB-D images of size 640x360. 10 trajectories are collected per task to cover all variations of each task. The hyperparameters used for training the models are listed in Appendix A.3. The training tasks, illustrated in Figure 3, are listed below. 1) Place shape in shape sorter: insert objects into a box with variations the object shape. Put block in cup: put colored block in colored cup with 3 variations colors. 3) Open drawer: open drawer with 3 variations on the handles. 4) Put block in drawer: put a colored block in an open drawer with 3 variations on colors.
single camera (Intel RealSense D435i) to
We assess the generalization ability of CLAP along two key dimensions (see Appendix A.6 for an overview these evaluation tasks):
1) Visual Perturbations: The model is tested on the tasks same as the training tasks but with the following conditions: different table colors, introducing distracting objects and altered backgrounds. Generalization is evaluated through: i) Object substitution (e.g., plac- ing a "shape" object into a cup), and ii) Skill composition (e.g., combining "open drawer" and "place block" into single, sequential task).
2) Task/Object Variations:
Results The results of evaluating the trained models in real-world experiments are summarized in Table 3. Our method achieves strong generalization ability novel tasks and object variations, trained with only demonstrations per task. CLAP achieves 54.8% higher average success rates compared to RVT2 the evaluation tasks.

---

Published as a conference paper at ICLR 2026
6 CONCLUSIONS
We propose a novel coarse-to-fine 3D manipulation policy, where tasks are decomposed and pre- trained models are leveraged in the hierarchical architecture. Our method demonstrates strong gener- alization capabilities while maintaining the sample efficiency inherent to coarse-(o-fine approaches. Although leveraging pre-trained models for robotics tasks is common, their effective adaptation for generalizable and precise is still under-explored. We hope this work inspires further research into building highly generalizable and sample-efficiet 3D manipulation policies. Our method has two key limitations. First, key-frame based imitation learning is suitable for structured tasks that can be easily decomposed into discrete steps. Unstructured tasks, such as wiping a desk, where key- frames are difficult to define, present significant challenge. Moreover, the current framework lacks a robust error-correction mechanism. An incorrect action prediction at any step might lead to task failure. A promising future direction is integrate self-correction module to enhance robustness.
Acknowledgement This work has been supported in part by the program of National Natural Science Foundation of China (No. 62176154), the program of National Natural Science Foundation of China (No. 62503322), Shanghai Magnolia Funding Pujiang Program (No. 23PJ1404400), and the AI for Science Seed Program of Shanghai Jiao Tong University (project number 2025AI4S- QY06).
LLM Usage We used Deepseek (AI, 2024) and ChatGPT (OpenAI, 2024) for grammar check and related work retrieval. The authors have reviewed the content generated by the LLM.
Ethics Statement content.
We adhere to the ICLR Code of Ethics and take full responsibility for the final
Reproducibility Statement our method and experimental setup in the Section and the Section 5, document all hyperparam- eters in the appendix Table 6, and release our code at https: / ////tttttiit gighub......... com/ Jianssua-aaaaaaaaaeaa Hu/ Generalizabi ineeralizable-CLae
To ensure reproducibility, we provide comprehensive description of
REFERENCES
DeepSeek AI. Deepseek-v3. https://deepseede com, 2024. Model and technical details available at attps://gitte
com deppseekppeee......
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula. Fangyu Liu, Adam Gryener, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harm- sen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. URL https: //arxiv. http:/aa//22200770
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Her- zog. Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wa Wei Edward Lee, Sergey Levine, Yao Lu, Hen- ryk Michalewski. Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart,
10

---

Published as a conference paper at ICLR 2026
Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt- 2: Aision-language-acti -ccttin models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.
Chilam Cheang. Sijin Chen, Zhongren Cui, Yingdong Hu, Liqun Huang, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Xiao Ma, Hao Niu, Wenxuan Ou, Wanli Peng. Zeyu Ren, Haixin Shi, Jiawen Tian, Hongtao Wu, Xin Xiao, Yuyang Xiao, Jiafeng Xu, and Yichu Yang. Gr-3 technical report, 2025.
URL https::p////.mm pestps://arxee /arxivv..v...////i///aa////aaavv org/ abs/ 2200/25002222 15493.
Shizhe Chen, Ricardo Garcia Pinel, Cordelia Schmid, and Ivan Laptev. Polarnet: 3d point clouds for language-guided robotic manipulation. In Jie Tan, Mare Toussaint, and Kourosh Darvish (eds.). Proceedings of The 7th Conference on Robot Learning, volume 229 of Pro- ceedings of Machine Learning Research, pp. 1761-1781. PMLR, 06-09 Nov 2023. URL https: httts://proceedingg proceedings wwwpnngssm.....mm mallr.peess / v229//222//22 chen23b. html.
Yangtao Chen, Zixuan Chen, Junhui Yin, Jing Huo, Pinzhuo Tian, Jieqi Shi, and Yang Gao. Gravmad: Grounded spatial value maps guided action diffusion for general- ized 3d manipulation. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.). In- ternational Conference Representation Learning. volume 2025, pp. 102042-102074, 2025a. /tLhttps:/ //proceedriicoooooo ngs, iclr. cctpappr_filleeeeeeeeeeeeeeessee 6e6f9c852 Paper-
Zixuan Chen, Junhui Yin, Yangtao Chen, Jing Huo, Pinzhuo Tian, Jieqi Shi, Yiwen Hou, Yinchuan Li, and Yang Gao. Deco: Task decomposition and skill composition for zero-shot generalization in long-horizel 3d manipulation, 2025b. URL https: arxiv. /ttpa//wwwww..//////////000 22000/0002/
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. PMLR. 23-29 Jul 2023. URL https: /ttes://prooooo http//ppooeeennn..//.e mlr press/v2022 driess23a. html.
Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
Chongkai Gao, Zixuan Liu, Zhenghao Chi, Junshan Huang, Xin Fei, Yiwen Hou, Yuxuan Zhang, Yudi Lin, Zhirui Fang, Zeyu Jiang, and Lin Shao. Vla-os: Structuring and dissecting planning rep- resentations and paradigms in vision-language-action models. arXiv preprint (2di::2006.176..1 2025. URL https: htt.r/iw..rr/bbb//55.111111
Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. Towards generalizable vision-language robotic manipulation: A benchmark and Ilm-guided 3d policy. In International Conference Robotics and Automation (ICRA), 2025.
Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d fea- ture field transformers for multi-task robotie manipulation. In Jie Tan, Marc Toussaint, and Kourosh Darvish (eds.). Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings Machine Learning Research, pp. 3949-3965. PMLR, 06-09 Nov 2023. URL https: http://pooeeeeengggee mmlr www.mmmr...sss...... press w229//222//////// vttess//22///eeet22ee.e http:///ggeeettaaaanemmma
Ankit Goyal, Jie Xu. Yijie Guo. Valts Blukis, Yu- Wei Chao, and Dieter Fox. Ryt: Robotic view transformer for 3d object manipulation. In Jie Tan, Mare Toussaint, and Kourosh Darvish (eds.), Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp. 694-710. PMLR, 06-09 Nov 2023. URL https: httpt::tt/ppptteet /ttpa////pcccceineg... w22 http:///22g/g//l22larr aooall2aa....... htmmm.l.........m.mmmmmmmm.....m
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Ke-Wei Chao, and Dieter Fox. Ryt2: Learning precise manipulation from few demonstrations, RSS, 2024.

---

Published as a conference paper at ICLR 2026
Mareus Gualtieri and Robert Platt. Learning 6-dof grasping and pick-place using attention fo- cus. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.). Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 477-486. PMLR, 29-31 Oct 2018. URL https: petttp::/pprceeee ptwc//ooocecccooooooooooo ngs ml pil...rress/8//
gualtieri iecierriia.... html.
Mareus Gualtieri and Robert Platt. Learning manipulation skills via hierarchical spatial attention. JEEE Transactions on Robotics, 36(4): August 2020. ISSN 1941-0468. doi: 10.1 109/ tro.2020.2974093.. URL http://///// doi org/10. 1109/ / TRO 2020. 2974093.
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), Proceedings of The 6th Conference on Robot Learn- ing. volume 205 of Proceedings of Machine Learning Research, pp. 175- PMLR, 14-18 Dec 2023. URL https::tpp........p....tt proceed ngs.... . ml www...peessvvvvv gww.guuuuuuuuuu html.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: oppnneviiw. net forum?
id=nZeVKeeFYfF
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. To.5: a vision-language-action model with open-world generalization, 2025. URL https: //axiiv. org/abs/ 250../6044..... 16054.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In 2022 JEFE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 13738, 2022. doi: 10.11009CVPPS52688.2222222
Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilve Wang, Longzan Luo, Xiaoqi Li, Peng- wei Wang. Zhongyuan Wang, Renrui Zhang, and Shanghang Zhang. Lift3d policy: Lifting 2d foundation models for robust 3d robotic manipulation. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 17347-17358, June 2025.
Edward Johns. Coarse-to-fim fine imitation learning: Robot manipulation from a single demonstration. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations, 2024. URL https: /ttp:///aavvvviii org/ abs /20.2.222222 10885.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-actionn model. arXiv preprint 24X.::220000922646 2024.
Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, and Yichen Zhu, Pointvla: Injecting the 3d world into vision-language-actioo models, 2025a. URL aattps://arxe org/ /bs/ 2503.00 07511.
Peiyan Li, Yixiang Chen, Hongtao Wu, Xiao Ma, Xiangnan Wu, Yan Huang, Liang Wang, Tao Kong, and Tieniu Tan. Bridge Input-output alignment for efficient 3d manipulation learning with languaee models. In The Thirty-ninth Annual Conference on Neural Information Pro- cessing Systems, 2025b. URL https: wop///peevvveeeeeeeeee net. forum? i d= f fBF 6hYuQv.
Qixiu Li. Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei. Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint ar 2024.
12

---

Published as a conference paper at ICLR 2026
Suhan Ling, Yian Wang, Ruihai Wu, Shiguang Wu, Yuzheng Zhuang, Tianyi Xu, Yu Li, Chang Liu, and Hao Dong. Articulated object manipulation with coarse-to- toase-ttofiim- fine affordance for mitigating the effect of point cloud noise. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 10895-10901, 2024. doi: 10.1 R..109//CR55144...0 1061059..
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu, Rdt-Ib: a diffusion foundation model for bimanual manipulation. International Conference on Learning Representations (ICLR), 2024.
Yangjun Liu, Sheng Liu, Binghan Chen, Zhi-Xin Yang, and Sheng Xu. lusiionperreeption-to-oee totpttitiooottttionn transformer: Enhancing robotic manipulation with 3-d visual fusion attention and proprioception. doi: 10. 200.7.0...02.33339
IEEE Transactions Robotics, 41:1553-1567, 2025.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang. Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain of thought. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: https://ppeeeevieeee
net ftre/formmmiddi -d-InIIIIIIIaaaa
NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da. Runyu Ding. Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz. Kaushil Kundalia, Lawrence Lao, Zhiqi Li. Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang. Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr00t nl: An open foundation model for generalist humanoid robots, 2025. URL https: / aaaiiiiiii.v wtg/abs/////oooooooeo..op/...ara aaa///////// /2003.2533. 14734.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
OpenAl. Chatgpt-40 (may 13, 2024 version) [large language model]. com, 2024.
openai.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q Vo, Marc Szafranicc, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Ass- ran, Nicolas Ballas, Wojeiech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang- Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning ro- bust visual features without supervision. ArXiv, abs/2304.07193, 2023. URL httpss://pi.. semant wwwchcar......... or wwtp:orgg//oosseeee /Crrpporpsss 258170077
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. Spatialvla: Exploring spatial representations for visual-languuge-actioo model, 2025. URL https:
org. abs 2501. 15830.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Fe- ichtenhofer. SAM 2: Segment anything in images and videos. In The Thirteenth International Conference Learning Representations, 2025. https::: /ttpp/ppeeeee.eeeeeeee www.:/wweeeeooeeeeeeooeeeeeeeeoe formm?? id=Ha MRD.
Mohit Shridhar. Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf. and Remi Cadene. Smolvla: A vision-language-actioo model for affordable and efficient roboties, 2025. URL https: / aaaxiiiiiw wwwpi/o.rrgb/b/////ee/ 01844, httttpps//aaiiioooo/////////////
13

---

Published as a conference paper at ICLR 2026
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong. Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez. Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Oscar Chang, Jose En- rique Chen, Xi Chen, Hao-Tien Lewis Chiang. Krzysztof Choromanski, David D Ambrosio, Sudeep Dasari, Todor Davchev, Coline Devin, Norman Di Palo, Tianli Ding, Adil Dostmohamed, Danny Driess, Yilun Du, Debidatta Dwibedi, Michael Elabd, Claudio Fantacei, Cody Fong, Erik Frey, Chuyuan Fu, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Leonard Hasen- clever, Nicolas Heess, Brandon Hernaez, Alexander Herzog. Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Deepali Jain, Ryan Julian, Dmitry Kalashnikov, M. Emre Karago- zler, Stefani Karp, Chase Kew, Jerad Kirkland, Sean Kirmani, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Isabel Leal, Alex X. Lee, T'san-Wer Edward Lee, Jacky Liang, Yixin Lin, Sharath Maddineni, Anirudha Majumdar, Assaf Hurwitz Michacly, Robert Moreno, Michael Ne- unert, Francesco Nori, Carolina Parada, Emilio Parisotto, Peter Pastor, Acorn Pooley, Kanishka Rao, Krista Reymann, Dorsa Sadigh, Stefano Saliceti, Pannag Sanketi, Pierre Sermanet, Dhruv Shah, Mohit Sharma, Kathryn Shea, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Jost Tobias Springenberg, Rachel Sterneck, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Vin- cent Vanhoucke, Jake Varley, Grace Vesom, Giulia Vezzani, Oriol Vinyals, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Yuxiang Yang, Rui Yao, Sergey Yaroshenko, Wenhao Yu, Wentao Yuan, Jingwei Zhang, Tingnan Zhang, Allan Zhou, and Yuxiang Zhou. Gemini robotics: Bringing ai into the physical world, 2025. URL https: //taaaaavvv//////////eee.eawavv. wwtp://ggg/bb/a///e/..
abs/ 2503//2553.. 20020..
Chenxi Wang, Hongjie Fang, Haoshu Fang, and Cewu Lu. Rise: 3d perception makes real-world robot imitation simple and effective. 2024 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2870-2877, 2024a. URL https://api.
wwwpcchhlaaa.r.oooooooooooooo htap://orpusssi 269214333.
Yixuan Wang, Guang Yin, Binghao Huang, Tarik Kelestemur, Jiuguang Wang, and Yunzhu Li. Gendp: 3d semantic fields for category-level generalizable diffusion policy. In 8th Annual Con- ference Robot Learning, 2024b.
Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, et al. Tinyvla: Towards fast, data-efficient ssiionlanguaag-actinnn models for robotic manipulation. In JEEE Robotics and Automation Letters (RA-L), 2025a.
Junjie Wen, Yichen Zhu, Zhibing Tang, Jinming Li, Yaxin Peng, Chaomin Shen, and Feifei Feng- Dexvla: Vision languaee model with plug-in diffusion expert for visuomotor policy learning. Conference on Robot Learning (CoRL), 2025b.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction in robotics. In 8th Annual Conference on Robot Learning, 2024. URL https:
/ /oopezeview
jp2ohU.
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic reasoning. In 8th Annual Conference on Robot Learning, control via embodied wptpreveeeeeeeeee/eeeeeee. net forum? waterrummii--7- d=S7000 2024. URL https:
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In Proceedings of Robotics: Science and Systems (RSS), 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11941-11952, 2023. URL nttps://agg semanticschon orgee.or//ooaa herg/CorpussD. 257767223
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Ming- Yu Liu, Donglai Xiang, Gor- don Wetzstein, and Tsung-Yi Yi Lin. Cot-vla: Visual chain-of of thoughtt reasoning for vision- language-action models. 2025 IEEF/CVF Conference on Computer Vision and Pattern Recog-
14

---

Published as a conference paper at ICLR 2026
nition (CVPR), pp. 1702-1713, 2025. URL httpss//ppp..eemntttcccccooo....
CorpusID:277455005.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d iision-lanuagge-cction generative world model. In Proceedings the 41st International Conference on Machine Learning, ICML' 24. JMLR.org, 2024.
Haoyi Zhu, Honghui Yang, Yating Wang, Jiange Yang, Limin Wang, and Tong He. Spa: 3d spatial- awareness enables effective embodied representation. In Y. Yue, Garg, N. Peng. F. Sha, and R. Yu (eds.), International Conference Representation Learning, volume 2025, pp. 26361- 26391, 2025. URL htt.tttss/prrroedng...oooo iel.. sttprrrrcecpppeeeleeee///eerrrre file/ / 4aA.0000c33E11155544440---areeee
15

---

Published as a conference paper at ICLR 2026
Table 4: Number of samples. We record the number of samples in training set and validation set for different datasets used in the simulation experiments for GemBench.
Language Plan
Object Position
Robot Trajectory Data
Train Validation 6103
30515
39777
169665
16650
17400
A APPENDIX
A.1 GEMBENCH TASK SPECIFICATION
The detailed tasks and variations in GemBench (Garcia et al., 2025) used for training and evaluation listed in Table 5.
A.2 TRAINING DATA SPECIFICATION
We have three datasets in total for finetuning the pre-trained VLM: language plan dataset, a object position dataset and robot trajectory dataset.
For language plan dataset, we label each task with language plan according to the decom- position of the task. Note that one task may contain different variations. Each variation of the task is assigned with different language plan. We augment the data by asking LLM to provide some variations on the language descriptions of the plan.
For object position dataset, we randomly initialize the scene the tasks and apply transfor- mations (translation and rotation) on the scene. We record the RGB-D images along with the 3D positions of the objects. The 3D positions of the objects are projected into canonical views to obtain the pixel positions in each view.
For robot trajectory dataset, we explain in the Section 5.1 on how we sample transitions from the demonstrations build observation-action pair. Transformations (translation and rotation) are also applied on the transitions to augment the data.
In practice, the number of training samples for each dataset used for GemBench is listed in Table 4.
A.3 EXPERIMENTAL DETAILS
All experiments are conducted on 4 NVIDIA RTX 4090 GPU. The hyperparameters and training time are listed in Table 6.
A.4 GEMBENCH SUCCESS RATE PER TASK
The results for each task across different generalization levels in Gembench are listed in Table 7, Table 8, Table 9, Table 10.
A.5 ADDITIONAL ABLATION STUDY
We first show the performance of our method on GemBench trained with different numbers of robot trajectories (10, 20, 50, 100) in Table 1. Further increasing the number of robot trajectory improves on the in-domain performance (LI) while does not help the average success rate. We conclude that the data efficiency is attributed to the following points.
Task decomposition allows the agent to learn common skills among different tasks and allows compositional generalization.
Bridging VLM and 3D manipulation with projected images. separating task planning from keypoint prediction, and Chain-of-Thougth reasoning lead to an effective adaptation of pre-trained VI.M to the coarse task planner.
Data efficiency with respect to the robot trajectories also benefits from leveraging language plans and object position data. By co-training on these additional sources, framework
16

---

Published as a conference paper at ICLR 2026
Table Training and evaluation tasks & variations in GemBench. The evaluation tasks contain four levels of generalization, where Level evaluates the generalization to novel placements, Level 2 novel rigid objects, Level 3 novel articulated objects, and Level 4 novel long-horizon tasks.
Task
Variation
Shape
Instance
Category
Action- Part
Long horizon
maroon button
azure button
2 buttons
Press
Push bution
button
button
buttons
yellow
buttion
red block
teal block
red cylinder
Pick and lift
lime block
violet block
red star
Pick
cyan black
black block
red
magenta cup
gray cup
Pick up cup
silver cup
olive cup
red toy
oninge
purple
green target
target
Push
bluc targel
yellow larget
Reach and drag
scal target
eyan target
black target
novyy target
jur
Screw
Screw balb
rose bull
violet,
lime bull
groen jar
white bulb
bull
fridge
fridgo
fridge2 laptop lid2 microwaves2
Close
Close laptop lid
laptop
grill
box
Close macrowave
microwave
drawer
doce
door
deer2 drawerl, drawer3
fridge laptop
Take shoes
Open
Open
Open drower
bottom drawer
drawer
long drawer w/ 4 levels
middle drawer
microwave
Pur items
in drawer
gray
orange blocks
blacks
2 purple blucks
mugeenia blocks
Slack cups
Pat/ Stack
gmoeeiie
crackers box.
soup
mustard bottle
sugar box
Put all groceries
Put money
middle shelf
shelf
cube bottom shelf
top shelf
Table 6: Training time and hyperparameters used in different experiments. Here we list the training time and hyperparameters used for training the model with GemBench and real-world data.
GemBench
Real-world
Coarse task planner
Fine- grainer Action Predictor
Coarse task planner
Fine- cene-grained Action Predictor
Training time
hours
Learning rata
3e-4
0.0024
lc-4
0.0024
Batch Size
64
192
64
192
Epochs
5
Lora Rank
Lora Alpha
32
32
Freeze
False
False
Froeze Aligner
True
True
Freeze LLM
False
False
equips the model with task mask-planning and boject-groundingg abilities, leaving learn only how to act given sub-task instructions and known object locations. In contrast, prior ap- proaches required the model learn task planning, object detection, and action prediction solely from robot trajectories.
We then show the results of variations on inputs to the fine-grained action predictor in Table 12. Removing some components in the fine-grained action predictor leads performance drop some extend.
A.6 REAL-WORLD EXPERIMENTS
An overview of the tasks used in real-world experiments are shown in Figure 4.
17

---

Published as a conference paper at ICLR 2026
Marhol
Frigge0
Close
1aels
Close
1e-16
Closs
Choetaptop
Lade0
MicnsravosO
Close
LightBul
Ine-17
Lightbub
In-19
Buse)
Daareo
Denserso
Ojen
Hooeemerr (latter et 2011)
Mhess
Mass
Vless
Mlacs
17eaa
1hast
4x47
15a32
Wist
Wusa
16.65
Difliee Actor ct. 2004)
RVTS (Gou et al, 2034)
17am
07:45
18.01
100
93.11
13157
1Ds LOLUU (Camii et al., 20155
94
100....
100...
01:44
M4n4
31-L (Cants 2005)
6377
05.24
160
35amu
CLAP
Mochod
12
Oyen
Li0
Paks
Lik12
Pukk
L.81
Pasty
Cupt8
Raup
Cap19
Cupil
PiMii
Belcen
Push
Beloei3
Pusk
Betoon
Push
Cuphoard
Pullc
Hivebomer (Faher atal. 2013)
87ara
34
Priuree (Chen mtll 1007))
90211
M7.11
K7ar
77
108
106
M433
57a7
10 Diliter Actor (ee ctsl.. 2004)
97.45
99.23
1912
97.45
98.44
98.57
00.13
08.07
85
(Good al, 2024)
42
06127
100..0
99.13
1001e0
100 c0
180.04
LO-LOTUS
20255
10-1 LITT (Cants 2008)
BriigeV A stal. 30046
19
100..
Mazh
94437
108
Waa.
74
CLAP
03227
Plas
100
Wiarr
Method
Capboagds3
IsSale10
InSafes1
D4sg-14
15
Bock-0
Block1
Bleckas30
Blocka436
Bleckas39
HiveFommer (Galex at 3023)
14
85.15
88:37
57.57
32an
98.31
655s
7:41
6243
10 biffiee Acter 2003)
07:43
M5253
09233
Wasa
10
0gna
34:31
16
2004)
WV (Gagg es 2034)
50.01
15.181
04:01
100.14
1D LOTUS (Ganta et al.. 2015)
04.17
99.23
Wu2a
100.co
100.2
100.1
Maas
Plana
90.55
Sis (Gussis al, 2025)
2325
10159
55.15
IndggVV
Table 7: Per-task Success Rate on GemBench Level 1.
Method
Aug.
Pinh Buttone33
Pish Buttone15
Pish Button 17
Preik 1ih-14
Pickkk Lifel8
Priik Lih-18
Cup-10
PickUp
Cap+12
PickUp
Cup+13
PhokUip
HiveFwmerr (Gisuur al.,, 20133)
16
17.27
B
9:45
30gt
22
26
PolaNer 3D Diffeer Actur (Ke al. 2024)
2023)
37.
100
100
Ownn
46
43.4
60
9.15
14
Own
60
M2
(Gryale e al, 2004)
100
100
100
47
Sast
3D-1 OTUS (Gmeis
499.22
99420
100...
3,20
ID-LOTUS+: (Grssis al.,
98:20
94
BridgeVLA (Li 20256)
650.1.
74
Quee
CLAP
100.0.
100.00
Mi21
100.04
Mia7
43.45
27_21
98.17
Method
Blockse24
Blocks-37
Blockse33
Block=2
Blacke3
Jee43
Lightbuib
Lightliuhh
Lamp
Hiveoomeer (Gihue al. 2023)
PolaNet (Chen 2023)
0uee
3D Diffeer (Ke 2024)
0.00
RVT2 (Goval
2024)
0.50
1u23
7.36
30 LOTLSS (Garcia 2025)
24.49
30: LO-UO (Garcia 2025)
2ssa
IrndgeVLL
20258)
CLAAP
95
Methal
ReadiA
Druge5
ReadiA
Drage?
InSales0
PacCuè
Cylindere0
Stare0
Pikk8iin
Mouse0
PLLaLIn
Toys0)
Cuphourd77
Pullo
Cupboande8
Pull
Hover (Gahure et 2013)
Jaza
Ozen
4.2a
T3
73at
$1.s7
$7
Daso
0.30
PotarNet (C hen al. 2023)
6lass
10a11
40:11:
21
18
97.07
50
Dano
0250
3n Difiner Actor (Ke et al. 2024)
0150
6415
$117
99 99.1
451779
91.94
1015.
01en
$.45
RVT2 (Grgal et al 2004)
9liza
6158
96
9814
94.11
01so
0.an
an-4 OTUS (Cincis al., 2015)
95.45
18.104
25.15
6/att
M0.K4
9657
01so
0.50
3D-LOTUS44 (Careis et al., 2025)
94.20
64.144
37411
91a20
94
29:04
71ane
Lazo
0.5n
BrndgeVLA (La et ol. 20256)
Ma11
98.35
93
0seo
0.50
CL.PP
98.
61g
27
100.01
54
50
Table 8: Per-task Success Rate on GemBench Level 2.
Examples of Real-world Tasks
Sert duppe
Place block mno
Open disver
Put Vooc in dence
Place daps is cap Race bleck is cup
Place blokk in duver Place alaps trewe:
Original
Table Color
Distractors
24
Light Strength
Figure 4: Overview of the evaluation tasks in real-world experiments. We evaluate the all these eight tasks acroos different variations and record the success rate.
18

---

Published as a conference paper at ICLR 2026
Method
Avg.
Door4D
Close
Box40
Close
Fridge--0
Close
CloseLaplop
Lid2.0
Microwmec240
Clase
Don220
Open
Bor2-0
Open
Hineoomerr (Guhhur 2023)
34.n
32am.
PolarNe:
2023)
Sa1.7
Ozoe
78anT
26.s2
Masa
23
20124)
27.0
Dat.
O1oe
97.t7
23
88476
86474
67
RVT2 (Guya] et al,, 2024)
Laza
2107
72467
7lann
Suca
3D-LOTUS (Gancia et al, 2025)
38. 11111
020.
S8
36nn
S4g10.
85
42s0s
l1gt
3DJ OOTUS++ (Cinciia 2025)
41.5218
29.8
93.85
50.e5
99,00
12
16
Bridge VLA 2025b)
43.8
1as0
95.55
S1aana
68 68.124
49.6.1.1
3
92
36a0
Method
Dmwer220
Open
Drwer3-0
Open
OpenDmmerr
Long-0
OpenDmoerr
Longe1
OpenlXower
Long+3
OpenDraver
Long+3
SeatUp40
Fridge+0
Opea
HiveFormer (Gihor et al, 2023)
30.11
30
7844
82445
40-1
57.
6442
0200
PolarNet (Chen all.. 2023)
29
84ana
SSa5t
63
37a76
21a7
3D Dillserr Actor (Kk 2024)
19an2
laza
15a50
35
26.ss
79.12
Oano
7asr
RVT2
2024)
Mlatta
7a57
0203
LOTUS (Gancia al.,
90.a2
22
$6,114
33tta
17
75
O4ne
4,54
3D-LOTUS++ (Gandi el 2023)
70
4l243
72sa0
$2
78ah
0-00
Bridge VLA VLA (Li 2025b)
87 27
59380
34
18gi0
85
0as
7
CLAP
682x4
87
44aac
94
142s
76:15.4
724s
3:45
Method
Opesi
LideO
Microwave+O
Open
PurMonay
InSafe+2
Drawer+1
Open
Drawer+
Close
Gail+O
Close
Hoveerrmer (Guhor (Chen al., 2023)
s00.
Ouc.o
Dzae
Ozno
83a
44.30.
100
4gt
42.11.5
3D Dilluer Actor (he et al., 2024)
100.0.
0.c.
2145
Osno
66.14
68.11.7
RVT2 (Goya] et al,, 2024)
Osuu
6422
78
9:42
LD-LOIUS (Garcia et al,, 2025)
100.00
0.00
Os00
87.81
2930
3D LOTUS++ (Gurcia
2025)
19:13.9
Bridge
2025b)
2z25
Ozno
S8azza
$5.11.3
78
Dgee
76355
96265
84.82
40gr1
Table 9; Per-task Success Rate on GemBench Level 3.
Push Buttons+e]
Push
Push Buttens4+3
TakeSShoss OutOlBox4O
Puiliems InDeawer40
Puiliems
Avg.
Buttons++2
InDrawer+2
HiveFormer (Guhur et al., 2023)
Ogoo
Ogne
110.0
O_o.
0_oo
Ogoo
PolarNat (Chen
2023)
0.1022
1a22
020.0
0.0.0
0=00
Oz0o
3D Diffuser Actor (Ke et al., 2024)
O20.
Ogo0
Osnc
Oa0.0
020.a
0_00
Ogo0
RVT2 (Gnyal et al., 2024)
Dec.
0200
Osnc
040.0
020.0
0.00
0200
3D-LOTUS (Garcia
0.3gna
3:40
Oanc
0an.0
0zu.
0=no
Ogno
3D-LOTUS*+ (Garcia et al., 2025)
17.4-d
76g74
49.8.
8718.1
O2n.
0_00
01o0
BridgeVLA (Li al., 20256)
Ogon
O,ou
Oanc
010.
0.u.a
31.4a6
87.45
92g5.7
0.0.0
0_oe
0.00
Method
InDrawer:4
Pudliems
Tower4-1
Tower443
Cups10
Stack
Caps+3
Stack
PutAllGroceries
InCaphourdd
HiveHormer (Guhar al., 2023)
Oscn
Osoo
Oanc
Da0.0
Ozu.
0=00
PolaNNet (Chen et aal,, 2023)
Dzen
O.eo
Oins
110.0
0ina
0_00
Diffuser Actor (Ke
2024)
Ogoo
020.0
RVT2 (Goyal et 2024)
Oeci
Oepo
Oanc
0s0.0
020.a
0=00
3D-LOTUS (Garzia et al., 2025)
Ogg.
0200
Osnc
040.0
020.0
0_00
3D-LOTUS++ (Garcia
2025)
Oseu
17a1a
30s:54
040.0
Oz0.0
Bri (Li et el., 2025b)
Droun
0.o0
0ine
01n.
0_n.a
0_o0
CLAP
Dac.o
30
70.0.1
020.0
0.0.0
0.00
Table 10: Per-task Success Rate on GemBench Level 4.
Ablation on number of robot demonstrations used for training. We compare the results of training our method with different numbers of demonstrations. The results are recorded with three runs of different random seeds.
Table 11:
Number of Demos
LI
L2
L3
L4
Average
10
84.5 0.8
81.5 + 0.6
43.3 + 1.9
30.5 2.1
60.0
20
83.9 + 0.3
83.2 1.9
49.6 2.1
31.4 0.6
62.0 0.5
50
87.9 0.3
8.1
17.0 1.5
61.1 0.9 28.3
100
86.9 1.5
0.4
47.8 2.6
27.5 1.2
61.0 + 0.7
Table 12: Ablation on inputs Fine-grained Action Predictor. We compare the results of re- moving some inputs our fine-grained action predictor. The results are recorded with three runs of different random seeds.
Inputs RGB
L1
L22
L.3
1.4
Average
83.9 + 1.4
79.7 23
1.3
31.8 1.3
60.9 + 0.6
RGB+Depth
87.2 = 2.3
8.
18
24.
60.8 = 0.6
RGB-Depth+3D positional embedding (Ours)
83.9 = 0.3
13
49.6
31.4
62.0 0.5
19


Reviews:
[{"review_id": "20uvCdM7l3", "reviewer": "Reviewer_EPq1", "summary": "To address the generalization issues in the hierarchical coarse-to-fine policy, the authors propose to extend a prior work, RVT2, with an VLM for task decomposition and 3D keypoint prediction. Specifically, to enhance such capabilities on the VLM, several ideas for fine-tuning the VLM have been introduced, namely 1) task decomposition, 2) VLM fine-tuning for 3D keypoint prediction, and 3) 3D-aware representation. Experimentally, the superior performance and training sample efficiency have been validated in both simulation and real-world along with comparison of a wide range of strong baselines.", "questions": "- Run-time efficiency: have the authors tried to use the same amount of training data during training as the other baselines for further improvement?\n\n- Redundant spatial information as input for the fine-grained action predictor: the inputs include multi-view images, depth and the 3D position embedding from pixel-wise 3D coordinates. It seems that the spatial information is quite redundant from these inputs, have the authors ablate this desgin choice?\n\n- Generalization concern for spatial awareness of the VLM after being fine-tuned on a spefic dataset: it's a popular idea to fine-tune an VLM for better in-context generalization, e.g., task compositionality and spatial awareness. I am wondering whether the fine-tuning causes performance degradation when testing on OOD data outside the training distribution.\n\n- Another question regarding the robustness mentioned above is about the hallucination of the VLM after being fine-tuned for greater spatial awareness. In the entropy column of table 3 [1], it is shown that spatial reasoning also helps on predictive performance on failure detection task but not necessarily on uncertainty estimation. It would be interesting and inspiring to know if this also happens when using the VLM as the task planner.\n\n[1] Zheng, Z., Feng, Q., Li, H., Knoll, A., & Feng, J. (2024). Evaluating uncertainty-based failure detection for closed-loop llm planners. arXiv preprint arXiv:2406.00430.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation is easy-to-follow with elaborative description of the approach.\n\n- An effective extension on top of RVT2: a sensible idea to add an VLM with enhanced generalizability of task compostionality and spatial awareness; To achieve this, the authors also propose of a set of pratical techniques, i.e., adding spatial reasoning, including previous sub-plans as inputs, 3D key-points prediction, etc.. \n\n- Superior training sample efficiency has been shown. Its performance surprisingly surpasses the SOTA methods while using only 1/5 of the training trajectories.\n\n- Comprehensive experimental study in simulation and real-world, demonstrating significant performance gain against a wide range of baselines.", "weaknesses": "- Presentation seems too wordy. The writing style of the current draft reads like a technical report. I would suggest the authors to shorten the method part and make it more concise so that the readability can be further improved.\n\n- Limited novelty. Though the performance is encouraging, the technical novelty remains limited in terms of the desgin choice and tricks applied for fine-tuning the VLM. More understanding and analysis can be conducted in order to improve the technical quality of this work, e.g., what exactly is the data efficiency coming from? Is it from the greater task compositionality or the enhanced spatial awareness?\n\n- Lack of a explicit discussion about related works on system1&2 policies. The topic of this submission seems quite relevant to the idea of system1&2 policies, a discussion about the difference and connection would be highly recommended.\n\n- Following the issue above, the discussion of run-time efficiency is also missing. How is it compred to the original RVT2? To which extend does adding the VLM module sacrifice the run-time efficiency? \n\n- The fine-grained action predictor in Fig.2 is simplified, mismatched to the description in the main texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yj4mMYviEf", "reviewer": "Reviewer_x1Wc", "summary": "This paper introduces CLAP, a hierarchical coarse to fine policy for 3D robotic manipulation that aims to generalize to novel instructions and environment changes. The coarse task planner is a fine tuned VLM that first decomposes a task into step wise language instructions, then reasons about object locations and predicts a language aligned 3D keypoint used to crop observations. The fine grained branch fuses step instruction with a 3D aware visual representation from multi view RGB D and depth encoders to predict actions. In simulation on GemBench, CLAP reports an average success rate that is 12 percent higher than the previous best method while using only one fifth of the training trajectories, and in real robot tests it generalizes to new tasks and perturbations with only ten demonstrations per task.", "questions": "- What fraction of your training set comes from language plans versus robot trajectories, and how were plans obtained and validated. Please report sensitivity to the amount and quality of plan data.\n- How robust is the planner to plan errors or mis localized objects. Do you have a recovery strategy or re planning mechanism during execution.\n- Can you provide equal trajectory comparisons to RVT2 or BridgeVLA and a study controlling for the number of demonstrations per variation.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- CLAP augments the coarse branch into a task planner that performs step wise decomposition, object localization, and language aligned 3D keypoint prediction, then conditions the fine branch on the current step rather than the entire task. The two round inference protocol and the auxiliary 3D object detection task are thoughtful adaptations that better align VLMs to manipulation.\n- Strong empirical evidence spans GemBench with detailed multi level results and an ablation table that isolates the impact of language plans, last step memory, object reasoning, and pre trained encoders. Real robot experiments further support the claims.", "weaknesses": "- The idea of using coarse-to-fine planning is not new, such as ReKep [1] , MakeADonut [2] and VoxPoser [3]. The technical novelty is limited and the contribution is unclear either.\n- The paper lacks sufficient details about the method, e.g., how are keypoints predicted; what are their distributions; how to convert from keypoints to robot actions. The finegrained action predictor in Figure 2 is never described in detail, making it hard to understand the proposed method. Therefore, the paper seems to be unpolished and not ready to be accepted.\n\n[1] Huang, Wenlong, et al. \"Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.\" arXiv preprint arXiv:2409.01652 (2024).\n\n[2] You, Yang, et al. \"Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation with Tools.\" IEEE Robotics and Automation Letters (2025).\n\n[3] Huang, Wenlong, et al. \"Voxposer: Composable 3d value maps for robotic manipulation with language models.\" arXiv preprint arXiv:2307.05973 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6kiDuv4cZb", "reviewer": "Reviewer_uaCQ", "summary": "This paper proposes CLAP, a hierarchical 3D manipulation policy to address the poor generalization of existing coarse-to-fine methods, especially for novel environments and skill compositions. CLAP features two innovations: 1. Coarse Task Planner: A fine-tuned VLM (Qwen 2.5 VL-3B) acts as the coarse branch. It decomposes tasks into step-wise language instructions $l_k$ and predicts a corresponding 3D keypoint $p{t_k}$ for each step, guided by a sequential reasoning process and an auxiliary object detection task. 2. Fine-grained Action Predictor: A separate module receives the specific step instruction $l_k$ and a \"3D-aware representation\" (built from SigLIP and DINOv2) centered at $p{t_k}$ to predict the final action. Experiments on GemBench show CLAP outperforms SOTA by 12% using only 1/5 of the data (20 demonstrations). Real-world tests also confirm strong generalization from only 10 demonstrations.", "questions": "1. Please clarify the source and scale of the auxiliary data (the \"language plans\" and \"object position dataset\") required for VLM fine-tuning. Is this data auto-generated or does it require manual annotation? If manual, how does the effort (e.g., in person-hours) to create these datasets compare to collecting the 10-20 demonstrations?\n\n2. Can you elaborate on the justification for this complex architecture? Why is it necessary to use separate SigLIP and DINOv2 encoders for the fine-grained branch, rather than (for example) re-using the VLM's (Qwen's) own visual encoder? Were simpler, encoder-sharing setups tested?\n\n3. This framework is a cascade. How does the system perform if the coarse task planner makes an error in the first stage (e.g., predicting a keypoint $p_k$ with a 10cm offset, or generating the wrong step instruction $l_k$)? Does the fine-grained action predictor have any robustness to correct for moderate errors from the coarse planner, or does the system fail catastrophically?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The VLM is innovatively fine-tuned to be the coarse branch, directly outputting 3D keypoints. The \"sequential reasoning\" fine-tuning pipeline is a clever method for aligning language, 3D space, and planning.\n2. The method achieves strong results on GemBench (62.0% avg. success) with only 1/5 of the data (20 demos) and generalizes in the real world from only 10 demos, which is highly practical.", "weaknesses": "1. **Extreme architectural complexity:** This paper relies on RVT2's canonical views, uses one VLM (Qwen) as the coarse planner, and uses two other powerful pre-trained models (SigLIP and DINOv2) for the fine-grained branch. This complexity makes reproduction difficult, and it's unclear if the performance gain comes from the novel architecture or simply from stacking 3+ powerful (and separate) pre-trained encoders.\n2. **Dependency on and lack of transparency about auxiliary data:** The method relies on fine-tuning on an \"auxiliary 3D object detection dataset\" and a \"language plan dataset\". The paper does not detail the source, scale, or creation cost of these datasets. If these decomposed steps require significant manual annotation, it would be a major, hidden data cost that undermines the claimed sample efficiency of 10-20 demonstrations.\n3. **Dependency on keyframes:** As the authors admit in the conclusion, the entire framework is tied to keyframe-based imitation learning. This makes it unsuitable for unstructured tasks where keyframes are difficult to define (e.g., wiping, stirring), limiting its generality.\n4. **Cascading errors:** This is a hierarchical system. If the coarse task planner (VLM) makes an error in the first stage—e.g., (a) generating the wrong step instruction $l_k$, or (b) predicting an inaccurate 3D keypoint $p_k$—the system will likely fail catastrophically. The paper does not discuss or evaluate the robustness to such cascading errors.\n5. **Insufficient comparation with highly related work:** The \"Related Work\" section fails to discuss (or differentiate from) several key recent works exploring an extremely similar problem space. Specifically, this paper needs a clear comparison with GravMAD (arxiv: 2409.20154) and DeCo (arxiv: 2505.00527). All three papers (including CLAP) are trying to solve the same core challenge: how to use foundation models and task decomposition to enable imitation learning models to zero-shot generalize to novel, long-horizon, compositional 3D manipulation tasks. This lack of comparison makes it difficult for reviewers to assess CLAP's unique contribution in this increasingly crowded and important research direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzScOHUJSX", "reviewer": "Reviewer_zNT5", "summary": "This paper proposes CLAP, a hierarchical coarse-to-fine policy for robotic 3D manipulation that integrates a pre-trained VLM as a coarse task planner and a fine-grained action predictor. The method introduces task decomposition into step-wise language instructions and employs a sequential reasoning process for 3D keypoint prediction. Evaluated on the GemBench benchmark and real-world tasks, CLAP demonstrates superior generalization to novel instructions and object variations while using significantly fewer training trajectories than prior state-of-the-art methods.", "questions": "1. What is the inference latency of CLAP under real-world operational conditions, and how does it compare to baseline methods like RVT2?\n2. In the sequential reasoning process where the coarse task planner first localizes task-related objects before predicting step instructions and keypoints, how does the method perform when object localization fails? \n3. Can CLAP handle tasks that lack clear step-wise descriptions or require fine-grained motions not easily captured by language?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The approach effectively addresses compositional generalization by decomposing tasks into reusable step-wise instructions, enabling skill recombination across tasks.\n2. The integration of pre-trained models with a structured reasoning pipeline enhances sample efficiency and spatial understanding, as validated through rigorous ablation studies.\n3. The method achieves notable performance gains in both simulation and real-world experiments, particularly on long-horizon and articulated object tasks.", "weaknesses": "1. The computational overhead of the two-stage architecture and sequential VLM reasoning remains unquantified, raising concerns about real-time applicability.\n2. The reliance on a chain of reasoning steps introduces fragility; errors in object localization or instruction generation may propagate and compromise task execution.\n3. The method assumes tasks can be linearly decomposed into language instructions, which may not hold for tasks requiring non-sequential or implicit motor skills.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)