GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation
Authors: Yangtao Chen
Zixuan Chen
Junhui Yin
Jing Huo
Pinzhuo Tian
Jieqi Shi
Yang Gao
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Manipulation
imitation learning
foundation models
sub-goals
diffusion models
Primary area: applications to robotics, autonomy, planning

Abstract: Robots' ability to follow language instructions and execute diverse 3D manipulation tasks is vital in robot learning. Traditional imitation learning-based methods perform well on seen tasks but struggle with novel, unseen ones due to variability. Recent approaches leverage large foundation models to assist in understanding novel tasks, thereby mitigating this issue. However, these methods lack a task-specific learning process, which is essential for an accurate understanding of 3D environments, often leading to execution failures. In this paper, we introduce GravMAD, a sub-goal-driven, language-conditioned action diffusion framework that combines the strengths of imitation learning and foundation models. Our approach breaks tasks into sub-goals based on language instructions, allowing auxiliary guidance during both training and inference. During training, we introduce Sub-goal Keypose Discovery to identify key sub-goals from demonstrations. Inference differs from training, as there are no demonstrations available, so we use pre-trained foundation models to bridge the gap and identify sub-goals for the current task. In both phases, GravMaps are generated from sub-goals, providing GravMAD with more flexible 3D spatial guidance compared to fixed 3D positions. Empirical evaluations on RLBench show that GravMAD significantly outperforms state-of-the-art methods, with a 28.63\% improvement on novel tasks and a 13.36\% gain on tasks encountered during training. Evaluations on real-world robotic tasks further show that GravMAD can reason about real-world tasks, associate them with relevant visual information, and generalize to novel tasks. These results demonstrate GravMAD's strong multi-task learning and generalization in 3D manipulation. Video demonstrations are available at: https://gravmad.github.io.

Published as a conference paper at ICLR 2025
GRAVMAD: GROUNDED SPATIAL VALUE MAPS GUIDED ACTION DIFFUSION FOR GENERALIZED 3D MANIPULATION
.Jing Huo Pinzzuu Tian", Jieqi Shi?, Yang Gao 1.2 Sttaee Key Laboratory for Novel Software Technology. Nanjing University, Nanjing, China
Yangtao Chen" Zixuan Chen" Junhui Yin', Junhuii
yangtacchen&@saem. smali wttp://mmmmsmaaa.....ueu.... .nju. cr, ichenzx, huo jing, gaoy) (ejju.edu cn,
yinjunhui@smmain nju. edu. cn
Schooo of Intelligence Science and Technology, Nanjing University (Suzhou Campus), Suzhou, China
jaycees ja com
School of Computer Engineering and Science, Shanghai University, Shanghai, China
Sanzhuo@shu.edii
ABSTRACT
Robots' ability to follow language instructions and execute diverse 3D manipula- tion tasks is vital in robot learning. Traditional imitation tearning-basee methods perform well on seen tasks but struggle with novel, unseen ones due to variability. Recent approaches leverage large foundation models to assist in understanding novel tasks, thereby mitigating this issue. However, these methods lack a task- specific learning process, which is essential for an accurate understanding of 3D environments, often leading to execution failures. In this paper, we introduce Grav- MAD, -oblgoal-driven. combines the strengths of imitation learning and foundation models. Our approach breaks tasks into sub- -oolo based on language instructions, allowing auxiliary guid- ance during both training and inference. During training, we introduce Sub-goal Keypose Discovery to identify key sub-goals from demonstrations. Inference dif- fers from training, there are demonstrations available, so we use pre-trained foundation models to bridge the gap and identify sub-goals for the current task. both phases, GravMaps generated from sub-goals, providing GravMAD with more flexible 31) spatial guidance compared to fixed 3D positions. Empirical evalu- ations on RIBBench show that GravMAD significantly outperforms state-of-the-art methods, with a 28. 38.63% improvement on novel tasks and 13.36% gain on tasks encountered during training. Evaluations on real-world robotic tasks further show that GravMAD can reason about real-world tasks, associate them with relevant visual information, and generalize to novel tasks. These results demonstrate Grav- MAD's 's strong multi-task ttaki learning and generalization in 3D manipulation. Video demonstrations are available at: https: / /ra vmad. . gi thub. io
sanguugg--onnitionne
action diffusion framework that
INTRODUCTION
One of the ultimate goals of general general-purpose purpose robot manipulation learning is to enable robots to perform a wide range of tasks in real-world 3D environments based on natural language instructions (Hu et al. 2023a). To achieve this, robots must understand task language instructions and align them with the spatial properties of relevant objects in the scene. Additionally, robots must effectively generalize across different tasks and environments; otherwise, their practical application will be limited (Zhou al. (202).. For example, if robot has learned the policy for the task "Take the chicken off the grill", it should also be able to perform the task "Put the chicken on the grill". Without this generalization ability, its utility will be greatly reduced. Recent research in robot learning for 3D manipulation tasks has focused on two mainstream approaches: imitation learning-baa based methods and pre- treitrained foundation model-based methods. Imitation learning-based methods learn end-to-end policies from
"These authors contributed equally.
1 "orresponning author.

---

Published as a conference paper at ICLR 2025
Imitation Learning
Keypose
Model Trained
Trsjectory
68.32 Difilsst
3D Visual Obs
Demonetrraton
detor
(a) Pipeline Imitation Learning- Based Methods
Plan
Heuristios- Resed
Foundation
Instrcttonn
Mirion Planser
Based
AP Calls
3D Visual Oba
Language
Conditioned Policy
(b) Pipeline for Foundation Model-Basse Methods
Foundation Model
Instruction
Guidance
geal
Model Trained on Demonstration
Keypose
Learning
Instr action
Trajectory
Visual Obs
3D Visual Obe
Sub-gual
(c) Pipeline for GravMAD (ours)
Seen Instruction: "Take the chickon off the grlll"
Unveen Instrctiion "Put chicken on grill"
generalize
3D diffusion
lock geserability
Vexposer precision
GravMAD (ours)
Inil!
grasp
graap chicken
w wrong cion?
ecese!
(d) Comparison of Generalization Capabilities of the Three Types Methods
Figure 1: Comparison of Pipelines. (a) Imitation learning- based methods learn end-to and-to-end policies that map language and 3D observations actions for precise manipulation. (b) Foundation models- based methods use L.MMs/VLMs to process inputs, generate plans, and execute actions with predefined primitives for task generalization. (c)(d) GravMAD combines both, using sub-goal guidance to leverage the language understanding of foundation models and the policy learning of imitation learning for precise and generalized manipulation.
expert demonstrations in attempt to address 3D manipulation tasks (Walke al.) 2023 Padalkar et al. 2024 Argall et al.) 2009 Chen et al. (024a). By designing various learning frameworks, such as incorporating different 3D representations (Shridhar et al. 2023] Chen et al. 2023a Goyal et al.. 2023), policy representations (Ze et al., 2024 Ke et al.) 2024 Yan et al. 2024), and multi-stage architectures (Gervet et al. 2023) Goyal et al. 2024). imitation learning based policies can map perceptual information and language instructions to actions that complete complex 3D manipulation tasks. However, these policies often overfit to specific tasks (Xie et al. 2024 ZZhag et al. 2024). leading to significant performance degradation those encountered during training (Brohan et al. 2023a Zitkovich et al. (023).
failure when applied tasks that differ from
Another line of cutting-edge research seeks to leverage foundation models trained on internet- scale data (OpenAl 2023] Yang et al. 2023b) to enhance policy generalization across a variety of tasks (Brohan al. 2023b Hu et al. [223bb] Huang
2023). Unlike traditional imitation learning- based methods, approaches using tre-trained foundation models typically decouple perception, reasoning, and control during manipulation (Sharan al., 2024). However, this decoupling often leads to limited understanding of scenes and manipulation tasks (Huang et al. 2024), allowing robots to conceptually grasp tasks but failing accurately complete tasks 3D environments, resulting in failures. This underscores a key challenge: both imitation learning- based and foundation model- based approaches struggle to balance precision and generalization when adapting to novel 3D manipulation tasks. Such challenge raises crucial question: Can the strengths of both approaches be combined to achieve precise yet generalized 3D manipulation?
To this end, inspired by the approach of introducing task sub- to achieve efficient execution in
robotic manipulation (Black et al.) 2024 Kang et al.) [2023] [Xian et al.) (2023: Ma et al. (024). we propose discovering key sub-goals for 31D manipulation tasks
bridge between foundation models
2

---

Published as a conference paper at ICLR 2025
and learned policies, leading to the development of Grounded Spatial Value Maps-guided Action Diffusion (GravMAD), a novel sub-goald uub-goals-driven, iinguage-condttoonl action diffusion framework. Specifically,
data distillation method called Sub-goal Keypose Discovery is introduced during the training phase. This method identifies the key sub-goals required for each sub-task stage from the demonstrations. In the inference phase, pre-trained foundation models are leveraged interpret the robot's s 3D visual observations and task language instructions, directly identifying task sub-goals Once the task sub-goals are obtained, the voxel value maps introduced in Voxposer (Huang et al. 2023) are used to generate the corresponding Grounded Spatial Value Maps (GravMaps). These maps reflect both the cost associated with each sub-goal and the ideal gripper openness. The closer to the sub-goal, the lower (cooler) the cost; the farther away, the higher (warmer) the cost, while also indicating the gripper's state within the sub-goal range. Thus, they serve as intuitive tools for grounding language instructions into 3D robotic workspaces. Finally, the generated Grav Maps are integrated with the policy diffusion architecture proposed in 3D diffuser actor (Ke et al., 2024), forming the GravMAD framework. This enables the robot to utilize 3D visual observations, task language instructions, and GravMaps guidance to denoise random noise into precise end-effector poses. As shown in Fig. [ GravMAD effectively combines the precise manipulation capabilities of imitation learning-basee methods with the reasoning and generalization abilities of foundation model-based approaches. We extensively evaluate GravMAD on RLBench (James et al. 2020), a representative benchmark for instruction-followiin that GravMAD not only performs well on tasks encountered during training but also significantly outperforms state-of- of-the-art baseline methods in terms of generalization to novel tasks. Additionally, we validate these findings through 10 real-world robotic manipulation tasks.
3D manipulation tasks. The results show
In summary, our contributions are: 1) We propose leveraging key sub-goals in 3D manipulation tasks to bridge the gap between foundation models and learned policies. In the training phase, we introduce a data distillation method, Sub-goal Keypose Discovery, to identify task sub-goals. In the inference phase, foundation models are used for this purpose. 2) We generate GravMaps from these sub-goals, translating task language instructions into 3D spatial sub-goals and reflecting spatial relationships in the environment. 3) We propose a new action diffusion framework, GravMAD, guided by Grav It is sub-goal-driver and innguage-conditionned combining the precision of imitation learning with the generalization capabilities of foundation models. 4) The simulation experiments are conducted on 20 tasks in RLBench, comprising two types: 12 base tasks directly selected from RLBench, and 8 novel tasks created by modifying scene configurations or task instructions. GravMAD achieves at least 13.36 % higher success rates than state-of-the-a baselines on the 12 base tasks encountered during training, and surpasses them by 28.63% % on the 8 novel tasks, highlighting its strong generalization capabilities. Experiments on 10 real-world robotic tasks further validate GravMAD's 's effectiveness.
2 RELATED WORKS
Learning 3D Manipulation Policies from Demonstrations. Recent works have employed various
perception methods to learn 3D manipulation policies from demonstrations to tackle the complexity
of reasoning in 3D space. These methods include using 2D images (Chen et al. 2024b] (Brohar et al.)
2023a Zitkovich et al. 2023: Jang et al. 2022), voxels (Shridhar et al. [023] James al. (022))
point clouds (Chen et al.
[203a]] Yuan et al. 2023), multi-view virtual images (Chen et al. 2023b]
Goyal et al. | / 2024). and feature fields (Gervet al. 2023). To support policy learning, some
studies (Ke et al. 2024 Xian et al. 2023 Yan et al. 2024
Ze et al. (024) have integrated 3D scene
representations with diffusion models (Ho al. 2020p. These approaches attempt handle the
multi-modality of actions, in contrast to behavior cloning methods that train deterministic policies. By leveraging 3D representation learning, these policies can accurately complete tasks by accounting for the spatial properties of objects, such as orientation and position. This is especially effective for tasks that closely resemble those encountered during training /e et al. (2023.. However, these policies often lack the language understanding and generalization abilitics of foundation models. Our method builds upon the diffusion architecture (Ke et al.) 2024), enhancing its ability to utilize demonstration data through imitation learning, while integrating foundation models to improve generalization, combining the strengths of both approaches.
Foundation Models for 3D Manipulation. Recent foundation models trained on interne-scale data have shown strong zero-shot and few-shot generalization, offering new opportunities for complex 3D manipulation tasks (Hu et al. 2023a Zhou et al. 2023). While some approaches fine-tune
3

---

Published as a conference paper at ICLR 2025
Training phase
Keypose
Inference phase
GurrMay Geneentor
Detail of Doiser
Noaik doaper Stangee
Context "checkon:14
Spurial Maps (Grashtap)
Derector
Steak 12.
Jorrest
Sabageal List
V-
Lyrop chiccho
teverr
chicken
Composer
Code
S
Pore Cloud Frooder
cortsodine learring
contest
(a) GravMap Synthesis
Grreshiip token
Enstroment
Nowy
Openness goal Oponnes
Junilayy
RS Nokey Dutatoon
Self-Ante Artention
Pociion Need
denilay
Sub-gcal Pasitom
Attention
Openners
Grigper Openren
Esselei
Pusissx Rood
42
chickion
Rossivo Hoxl
Dinooing
Dencise Rouaton
Preeroceptio
Denoköng ary
FiLM conditioning
(b) GravMaps Guided Action Diffusion
During training, we use Sub-goal Keypose Discovery obtain sub-goals gos and go During inference, Detector, Planner, and Composer pipeline interprets visual observations and language instructions to derive ges and go which are processed into a GravMap and encoded as GravMap token. (b) GravMaps Guided Action Diffusion: The policy network perceives the scene and denoises noisy actions guided by the GravMap token. After denoising steps, the clean actions are executed by the robot.
Figure 2: GrayMAD Overview. (a) GravMap Synthesis:
vision-language models with embodied data (Driess et al., 2023 Li et al. 2024), this increases computational costs due the large data requirements. Alternatively. foundational vision models can generate visual representations for 3D manipulation tasks (Zhang et al. 2024 (202)), but they often lack the reasoning capabilities needed for complex tasks. To address these challenges, some studies leverages large language models (LLMs) high-level planners (Brohan et al. 2023b Hu ctt al. 2023b; Huang al., 2022), generating language-
plans executed by lower-level policies. Others utilize LLMs' code-writing abilities to control robots via API calls or create value maps for planning robot trajectories (Liang et al. 2023 Huang et al. 2023). However, these methods often sacrifice precision due rough understanding of complex 3D scenes. Recent works have combined the reasoning capabilities of foundation models with fine- tne-grained control in 3D manipulation to overcome this limitation (Huang et al. 2024 Sharan et al.) 2024). For example, [Huang et al.) (2024) uses pre-trained lssion-language models (VLMs) to provide spatial constraints and a nonlinear solver generate precise grasp poses. Our method combines the learning power of diffusion architectures with the generalization of VLMs. VLMs generate spatial value maps that guide action diffusion, enabling precise control and multi-task generalization in 3D manipulation tasks.
METHOD
In this section, we introduce GravMAD. multi-task, sub- -obl--oalddiven, language-condee conditionn diffu- sion framework for 3D manipulation, as shown in Fig. [ We divide GravMAD's design into three parts: Section 3.I defines the problem setting, Section 3.2 explains the definition and generation of GravMaps, and Section 3.3 details how GravMaps guide action diffusion in 3D manipulation.
3.1 PROBLEM FORMULATION
We consider a problem setting where expert demonstrations consist of robot trajectory (01, 01 02, .2) and natural language instruction é f. that deseribes the task goal, Each
4

---

Published as a conference paper at ICLR 2025
"take off grill"
Trajectory
sub-goal keypose
sub-poal keypose
Trajectory
Sab-task
+
Stuge
+
Keypose
Subayea Kespine
sub-task stage
sub-task stage 2
"pursh maroon button, pursh green button, then push the cyou button"
sub-goal keypuse
sub-goal keypose
sub-goal keypose
Task Instruction: "take the chicken off the grill"
subbfask stage
sub-lask stage
sub-task stage
Figure 3: Visualization of sub-goal keyposes and sub-task stages. The left sub-figure shows image- based sub-goal keyposes and sub-task stages for "take the chicken off the grill" and "push the button" tasks. The right shows the sub-goal key poses and sub-task stages in the trajectory for the "take the chicken off the grill" task.
observation o, O includes RGB-D images from more viewpoints. Each action de A contains the 3D position of the robot's end-effector a E R a 6D rotation a² E IR. and a binary gripper state a open {0, 1}. To address potential discontinuities from quaternion constraints and ensure smooth optimization, we utilize the 6D rotation representation (Ke et al. 2024). In this setting, assume that robotic task is composed of multiple sub-tasks, with each sub-task completed when the robot reaches sub-gos openness green {0, 1}. and the 6D rotation 9rr E R. Based on this, we construct new dataset D = (S1, .2....) from expert demonstrations. Each demonstration consists of trajectories with sub-goals ((01, 91; a1), , (02, 92, 02) ....} and the corresponding language instruction f. Our goal to learn policy (0, C. G) + A, which maps observations 0N, sub-goals gt, and instructions to actions az To facilitate sub-task segmentation and efficiently learn the policy, we frame the robot's 3D manipulation learning problem as a keypose prediction problem following prior works (James) Davison 2022 James et al.) 2022 Goyal et al. 2023) Shridhar et al. 2023). Our model progres- sively predicts the next keypose based current observations and a sampling baasiiggggsdddddd based motion planner (Klemm et al. 2015) to plan the trajectory between two keyposes. In the existing keypose discovery method, pose is identified as keypose when the robot's joint velocities
E G, which specifies the 3D position gmT RR3, the gripper
near zero the gripper state changes (James & Davison 2022). Our work filters the task's 's sub-goals based on these keyposes facilitate sub-task segmentation and ensure efficient completion of the overall task.
3.2 GRAVMAP: GROUNDED SPATIAL VALUE MAPS
To tackle generalization challenges in 3D manipulation tasks, we introduce the spatial value maps (GravMap), an adaptation of the voxel value maps proposed Huang al.) denoted as m. GravMaps adaptively synthesized based task variations, translating language instructions into 3D spatial sub-goals and reflecting the spatial relationships within the environment. This provides precise guidance for robotic action diffusion. Each GravMap m contains two voxel maps: (1) a spatial cost map me; with lower values near the sub-goal and higher costs further away, and (2) gripper openness map ma. indicating where the gripper should open or close. As shown in Fig. [[a), GravMaps are generated differently for training and inference. In training, they are identified from expert demonstrations using the sub-goal keypose discovery method. During inference, pre-trained models generate them from language instructions and observed images.
GrayMap Synthesis with Sub-goal keypose Discovery during Training, We define each sub-task stage in 3D manipulation as: (1) the process where the robotic end-effector transitions from not touching an object to making contact, or (2) the interaction between the end-effector or tool and a new object, where series of operations performed before disengaging. To efficiently segment these
5

---

Published as a conference paper at ICLR 2025
sub-task stages and find sub -oobbg we build upon the existing keypose discovery method (James & Davison 2022) and propose a novel data distillation method called sub-goal keypost discovery.
The sub-goal keypose discovery process iterates over each keypose K {Kp)i' where N₂ is the number of keyposes a task. For each keypose, the corresponding observation-action pair (ox;: (xy) is passed to the function Sk, which outputs a Boolean value to determine whether the given keypose sub-goal keypose. The decision is made based whether the keypose satisfies the discovery constraint: Sx Sx((ox; (x;))
should be discovered
1, if discovery constraints are met The
O, otherwise
function SK can incorporate multiple constraints. In our paper, we define two constraints for Sk. depending on the type of manipulation task, as shown in Fig. 3 (1) For grasping tasks, such as "take the chicken off the grill", sub-goal keyposes are discovered based on the following constraints: a change in the gripper's open/close state and a significant change in touch force. (2) For contact-based tasks, such as "push the button", ", sub-goal keyposes are discovered solely based on significant changes in touch force. For more details sub-goal keypose discovery, please refer to Appendix [A.2]
After discovering the sub-goal keyposes, the sub-task stages can be quickly segmented, and the corresponding sub-goals can be identified. The end-effector position g and gripper openness go at these sub-goals are then input to the Gravmap generator to generate the GravMaps m for training. The process of the GravMap generator is illustrated in Algorithm T in Appendix [A.] [ adapted from Huang et al. (2023).
GravMap Synthesis with Foundation Model during Inference. During the inference phase, we use pre-trained foundation models to synthesize GravMaps. First, to enable the robot to tie the task-related words with their manifestation in the 3D environment, we introduce a -etkof-Mark (SoM) (Yang eet (22322222a)assss Detector. This Detector uses Semantic-SAM (Li et al. 2023) to perform semantic segmentation on the observed RGB images and assigns numerical tags the segmented regions. Next, the Detector uses GPT-4o to select task-relevant objects and their corresponding tags from the labeled images as contextual information C. Based on the task instructions and the context provided by the Detector, we apply the LLM-based Planner proposed by Huang et al. to infer series of text-based sub-goals. Then, an LLM-based Composer (Huang et al. 2023) recursively generates code parse each sub-goal. During execution, the code uses the context C to obtain the end-effector positions gm and gripper openness states green corresponding to each sub-goal. Finally, 9 and gopen are fed into the GravMap generator shown in Algorithm [ skipping the data augmentation process to generate the GravMaps. Details of this process can be found in Appendix AA3.2
We synthesize the GravMaps via sub-goal keypose discovery during training or foundation models during inference. GravMaps m are then downsampled using farthest point sampling (FPS) and encoded into token fm with the DP3 (Ze et al. 2024) encoder, lightweight MLP network.
3.3 GRAVMAPS GUIDED ACTION DIFFUSION
After obtaining the GravMaps, they can be used to guide the action diffusion process, as shown in Fig. 2 b). Before the diffusion process begins, the robot should first perceive the 3D environment.
3D Scene Perception. Building on previous works (Gervet et al. 2023 Ke et al. 2024). we use 3D scene encoder transform language instructions and mieti-view RGB-D images into scene tokens, enhancing the robot's 3D scene perception. RGB images are encoded using pre-trained CLIP ResNet50 backbone (Radford al. 202T and feature pyramid network. These features are lifted into 3D feature clouds using 3D positions derived from depth images and camera intrinsics. Simultaneously, the CLIP language encoder converts task instructions into language tokens. These tokens interact with the 3D feature cloud to generate scene tokens (L.), enabling the robot to capture 3D environmental information.
GravMaps Guided Action Diffusion. GravMAD builds upon the 3D trajectory diffusion architecture introduced by 3D Diffuser Actor (Ke et al. 2024) and further integrates GravMap tokens tm to guide the action diffusion process. Specifically, MrD models policy learning as the reconstruction of the robot's ss end-effector pose using diffusion probabilistic models (DDPMs) (Ho et al. 2020). The end-effector pose is represented the denoising networks pos and eot perform K iterative steps progressively reconstruct the clean
(a d Starting with Gaussian noise eg (a .aa)

---

Published as a conference paper at ICLR 2025
pose ey (ao pas )g")):
k 1 a (ak c - -166 pos p, (m) + N (0. o² rf))
(I)
put k- (ak - "Eo (ek, k. (2) + (0.o21))
where a, y, and are functions of the iteration step k, determined by the noise schedule. / (0, rT) is Gaussian noise. Here, p represents proprioceptive information (a short action history). The denoising networks use 3D relative position attention layers (Gervet et al. 2023 Xian al. 2023 Ke iet al. 2024), with FiLM (Perez et al. 2018 conditioning applied to each layer based on proprioception p and denoising step k. As shown Fig. b), after passing through linear layers, dg and a are concatenated and attend to the 3D scene tokens to via cross-attention. A self-attention layer then refines this representation produce end-effector contextual features. These features are processed by five prediction heads: the Position Head, Rotation Head, Openness Head, Auxiliary Openness Head, and Auxiliary Position Head. all but the rotation head, contextual features undergo cross-attentie with GravMap tokens, followed by an MLP to predict the target values. See Appendix A.4 for details.
The first two prediction heads predict the noise added to the original pose using the L1 norm, with the losses defined as:
Spss ||c 600 (ck, k dus tim) ||
(2)
f.r || cB (ek, ki p,t,) ll,
where iteration & is randomly selected, and eu and ex are randomly sampled as the ground truth noise.
The third prediction head is used to predict the gripper's open/close state, and we use binary cross- entropy (BCE) loss for supervision:
Copen = BCE (fe (ex. hi, p. t.. tm) , a
(3)
The last prediction heads enable GravMAD better focus on the ideal end-effector pose at sub-goals, with the loss functions defined as follows:
Laux_pos pos = l1g joux-pos (ck, k, tm))ll:
Laux open = BCE (foth open (ex, k P, ta tm).
where fe represents the pose prediction network in GravMAD, while goes and goeen denote the ground truth sub-goal positions and gripper openness, respectively.
In addition to the losses related to robot actions mentioned above, a contrastive learning loss is applied to enhance feature representations from GravMaps. Positive pairs are features from the same GravMap, while negative pairs come from different GravMaps. In each forward pass, one GravMap is extracted from the dataset, and N 1 different GravMaps are randomly generated. The loss maximizes similarity between positive pairs and minimizes it between negative pairs:
N
log exp(Sa Sat (T)
Loon
(5)
Σ exp(fo: fog/T)
1=1
where T is the temperature parameter, fa; represents the feature of the i-th sample, and fat represents the positive feature of the i-th sample.
At this stage, the training objective of GravMAD can be formulated by combining the losses from Eq 3 4 and as follows:
LGIEMAD = Lopen + w) Lpos + W2 Lrot + W3 Laux, _ioss_pos__ + Laux open + w4 Leon;
(6)
where WT, W/ W/ are adjustable hyperparameters. For more detailed implementation of GravMap and GravMAD, please refer to Appendix A
4 EXPERIMENTS
We aim to answer the following questions: (i) Can GravMAD achieve superior generalization in novel
3D manipulation tasks compared to SOTA models? (See Sec. 4.2) (ii) Is GravMAD's performance competitive on the 3D manipulation tasks encountered during training? (See Sec. 4.3) (fii) What key design elements contribute significantly to MrD's overall performance? (See Sec. 4.4
7

---

Published as a conference paper at ICLR 2025
5D Difffseer Actor
Vieposee
GravMAD (ours)
3D Diffleser Actor
Yoxpaser
GravMAD (ours)
distrackor
white light
The prediction off muodel is dinttr boo
The direelonn correct. but the granping Eails
Generattre scene with distractors
mothil can prtt
The beitton is kot pressed precselly
Generulize to povel instructiio
reason currectly
"gruspnng the lid, upt from the tible and use it seal the cyan jac."
"push the button with the some cofor as the light (ikkie))
Avg.
Avg.
Close Drawer
Closs Jar
Close lar
Condition Block
Meal On
Open Drawer Stack cups Push Buttons
Models
Suceoss
Kank
Banana
Distractor
Grill
Small
blocks
Light
Noxposer (Huneg et
2025
34,29
2.6
9600 1am
17.33 11975 22.67 2m 25,00 2o8.s 38.67 2.2.2
6.62 12:1
0.00 1am
68.00 21835
Av13D Gervel RAA
17.83
3.5
66
66.67.924
29.35
41,35
0.00amm
1.33
2.67
0.00gam
1.33
Dimiser
2024
29.38
33
48.000
42.67+4022
27.000-1115
000g0m
2.67.410
2.6722
20.67.1225
Gra (VLM)
62.92
1.0
97.33 un
24.00:10100
$6.6722.11
74.00
4533.107
21.33.11M
18.67
76.00
Performance
+28.63
+1.33
+36.00
+44.00
+47.00
+6.66
+14.66
+16.00
+8.00
Table I: Generalization to 8 novel RLBench tasks. Evaluations on 8 novel tasks are conducted using 3 seeds, with 25 test episodes per task, utilizing the final checkpoints from training on 12 base tasks. Performance gains are compared to the best-performing baselines, indicated by underlines.
4.1 ENVIRONMENTAL SETUP
To thoroughly investigate these questions, we conduct our experiments on a representative instruction- following 3D manipulation benchmark, RLBench (James al., 2020). Simulation experiments are conducted on two types of tasks to provide comprehensive evaluation of GravMAD. 1) Base tasks. To evaluate GravMAD's performance across 3D manipulation tasks encountered during training. we select 12 base tasks from RLBench's 's 100 language-conditione variations in instructions, such as handling objects of different colors or quantities. For each base task, collect 20 demonstrations for training and evaluate the final checkpoints using 3 random seeds over 25 episodes. Detailed descriptions of these tasks are provided in Appendix [B.]| 2) Novel modify the scene configurations or
tasks, each featuring 2 to 60
tasks. To further test GravMAD's ' generalization capabilities, task instructions of several base tasks to create 8 novel tasks across 3 novelty categories as illustrated in fig. 10 These modifications introduce significant challenges for the robot regarding instruction comprehension, environmental perception, and policy generalization, as described in Appendix [B.2] For each novel task, we evaluate the final checkpoints trained on the 12 base tasks. We use 3 random seeds over 25 episodes for each novel task. For all tasks, we use a front-view 256 x 256 RGB-D camera and Franka Panda robot with parallel grippers. Additionally, we further validate GravMAD on 10 real-world robotic tasks, with details provided in Appendix D.6
Baselines. We compare GravMAD against various baselines, covering both foundation model- based and imitation learning-basee methods. For the foundation model-based approach, we use VoxPoser (Huang et al., (023) as the baseline. VoxPoser leverages GPT-4 to generate code for constructing value maps, which are then used by heuristic-based motion planner to synthesize robotic arm trajectories. We reproduce this baseline in our tasks using prompt templates from Huang et al. and our SoM-based Detector, with five camera viewpoints in RLBench. For the imitation learning-basee baselines, we select: (1) 3D Diffuser Actor (Ke et al.] 2024). which combines 3D scene representations with diffusion policy for robotic manipulation tasks. To highlight instruction- following tasks, we use the enhanced language-conditioned version provided by Ke et al] and (2) Act3D (Gervet et al. 2023), which uses a 3D feature field within policy transformer to represent the robot's workspace. Differences between GravMAD and these baselines are detailed in Appendix A.5
Training and Evaluation Details. GravMAD runs in multi-task setting during both the training and testing phases. All models complete 600k training iterations on an NVIDIA RTX4090 GPU, with the final checkpoint selected using three random seeds for evaluation. During testing, except for the novel task "push buttons light", which must be completed in 3 time steps, all other tasks must be completed in 25 time steps; otherwise, they are considered failures. Evaluation metrics include the average success rate and rank. The success rate measures the proportion of tasks completed according to language instructions. Meanwhile, the average rank calculates the average of the rankings of each model in all tasks, reflecting the overall performance of the model in the tasks. Two settings are used to generate context C during testing: Manual and VLM. In the manual setting, we manually provide the Detector with the precise 3D coordinates of task-related objects in the simulation to generate accurate context. In the VLM setting, we use a Detector implemented with SoM and GPT-4o to locate task-related objects and generate context.
8

---

Extensive ablation studies are conducted to analyze the role of each key design element in GravMAD, with the results shown in Fig 4 The following findings are revealed: 1) Impact of replacing GravMaps with specific sub-goal position and openness: Replacing GravMaps with sub-goal: get and ren (w/ GravMap) results in a significant performance drop. Without GravMaps, the policy lacks regional context, becoming overly sensitive precise positions and unable to generalize to Published as a conference paper at ICLR 2025
Models
Avg. Success
Avg. Rank
Open Drawer
Mcat off Grill
Slide Block
in Drawer
Vosposer (Hanng et
2026
15.11
4.5
12.00
10.67
15.33
0.00...m
1.000
Act3D (Grrer et 8016)
34.11
4.3
61 61.334460
41 41.33
60.0.4.0
78.67-3al
49.33-11077
Difiuxer Actor cf
2024
55.81
66367
88.00
84.00
94.67.em1
Gra (Manual)
69.17
1.3
100.00,n inco
76.67
89,33
93.33 2:31
78.67
Performance
+3.36
+33,33
-13.33
+1.33
+9.33
-16.00
(V.MM)
56.72
100.00...10
58.67
70.672281
80.00-300
61.33
Performance gain
+0.91
+33.33
-2.333
-17.33
-4.00
-3.34
Models
Push Buttons
Stack Blocks
Place Cups
Place Wine
Screw Bulb
Insert Peg
Stuck Cups
Vosposer (Huang al
2023
80.00
16:00
6.67as.1
a.33
4.00
z.00
1333
Actil> (Gerve 2155)
66 =2.a
0.00
g.00
6.672251
0.00
0.00
3D Dafluser Actor (Ke al. 2024
94.67-1a1
13.67:3a1
5.33
82.674am1
29.33 at
2.67-1m
20.00.a.o
GraMMAD (Manual)
98.67 .::11
56.67.142
5.33,am
27.33
66.67.611
32.00
57.33
Performance gain
+4.00
+40.67
-1.34
-3.34
+37.34
+29.33
+37.33
GravMAD (VLM)
97.33_2.1
51.33
5.33
33.33
54.67
18.67
49,33
Performance gain
+2.66
+35.33
-1.34
.4..4
+25.34
16.00
+29.33
Table 2: Multi-task test results on 12 base tasks. All models are trained on 12 base tasks with 20 demonstrations each. Final checkpoints are evaluated across seeds with 25 test episodes per task. Performance gains are compared to the best-performing baselines.
4.2 GENERALIZATION PERFORMANCE OF GRAVMAD TO NOVEL TASKS
In Table II we present the generalization performance of models trained on 12 base tasks when tested on novel tasks, along with visualized trajectories from two of these tasks. The results show that changes in task scenarios and instructions negatively impact the test performance of all pre-trained models to some extent. However, GravMAD exhibits superior generalization across all novel tasks compared to the baseline models. In terms of average success rate, GravMAD outperforms VoxPoser, Act3D, and 3D Diffuser Actor by 28.63% %, 45.09 %, and 33.54 %, respectively. VoxPoser leverages large models to achieve a certain level of performance motion planner fails to grasp object properties and task interaction conditions, leading to poor results on tasks requiring fine manipulation, as shown in the trajectory visualizations. Similarly, 3D Diffuser Actor and Act3D struggle to transfer skills from training to novel tasks, primarily due to overfitting training-speciffee tasks, which hampers generalization. In contrast, GravMAD uses VL.M-generated GravMaps to guide action diffusion, enabling effective object interaction and strong performance on novel tasks. These results clearly demonstrate GravMAD's superior generalization.
novel tasks, but its heuristic
4.3 TEST PERFORMANCE OF GRAVMAD ON BASE TASKS
Table[[comparc.. the performance of all models 12 base tasks. GravMAD (Manual) outperforms Act3D and Voxposer across all tasks and exceeds the best baseline, 3D Diffuser Actor, in 9 out of 12 tasks, with average success rate improvement of 13.36% %o Despite the Detector's coarse SoM positioning affecting GravMAD (VLM)'s performance, it still outperforms Act3D and Voxposer on all tasks, with a 0.91% % higher average success rate than 3D Diffuser Actor. These results clearly show that GravMAD remains highly competitive even on previously seen tasks. As long as task-related object positions are accurate, the generated GravMap effectively reflects sub-goals and guides action diffusion, enabling precise execution by GravMAD. GravMAD (Manual) underperforms 3D Diffuser Actor in the "open drawer"" "pul in drawer". and "place wine" " tasks due to slight deviations between the manually provided object positions and the sub-goals. In high-precision tasks, even small deviations can impact performance. For example, the "open drawer" task, the robot needs to grasp the center of the small handle for optimal performance. After manually adjusting the sub-goal to better align with the handle, performance improved. GravMAD (VLM) also struggles in tasks like "Place Wine" due to inaccuracies in the object positions provided by the Detector, especially when Semantic SAM fails to provide precise locations or the camera doesn't 't capture the full scene. For further analysis of failure cases, please refer to Appendix B.3
4.4 ABLATIONS

---

Published as a conference paper at ICLR 2025
Tayles
35-72
56.42 S4.8 3r.g.
56.18
5258
54.3
20,22
GravMAD
50
wio. GravMta
W/s. Cost map
Grip. map
wio. Contra, lozs
W/o. Aux. loss
Guided Rott
DP3 Encoder woe PointNet
DP3 Encoder wa PointNet++
Figure 4: Ablation Studies. We evaluate the impact of key design elements by reporting the average success rates across 12 base tasks and 8 novel tasks. In the results, "-" denotes replacement, "w/o" indicates "without", and "w." signifies "with".
slight spatial variations. 2) Importance of both cost map and gripper map in GravMaps: The combination of the cost map and gripper map within GravMaps is essential for guiding the model's attention to sub-goal locations and ensuring effective gripper usage. The absence of the gripper map causes moderate decline in performance (w/o. Grip. map). In contrast, omitting the cost map causes rero-gradient issues during training, leading to incorrect predictions and task failure. This occurs because the encoder cannot process such input. Additional experiments for this ablation, detailed in Appendix D.5] highlight the cost map's impact on performance. (w/o. Cost map) 3) Significance of contrastive learning loss and auxiliary losses: Removing the contrastive learning loss feon results in highly similar features from the point cloud encoder, diminishing their effectiveness in action denoising and leading to a decline in model performance (w/o. Contra. loss). Similarly, the absence of auxiliary losses Laux Laux_pos pos and Laux open weakens the model's focus on sub-goals, leading to a noticeable drop in performance (w/o Aux. loss). 4) Effect GravMap tokens on guiding Conditioning rotation actions with GravMap tokens in the action diffusion process results in a performance drop, likely due to the inherent nature of rotation actions, which makes them difficult to be guided explicitly through value maps (w. Guided Rot.). 5) Impact of different point cloud encoders on GrayMap performance. Replacing the DP3 encoder in GravMAD with PointNet (Qi et al. 201 7a) (DP3 Encoder PointNet) or PointNet++ (Qi al.) 2017b) (DP3 Encoder + PointNet+++) leads to a performance decline. We suspect that lightweight encoders help prevent overfitting training data details, enhancing GravMAD's generalization ability across different tasks or unseen data.
rotation actions:
CONCLUSION AND DISCUSSION
In this paper, introduce GravMAD, novel action diffusion framework that facilitates generalized 3D manipulation using sub-goals. GravMAD grounds language instructions into spatial subgoals within the 3D workspace through grounded spatial value maps (GravMaps). During training, these GravMaps are generated from demonstrations by Sub-goal Keyposes Discovery. In the inference phase, Grav are constructed by leveraging foundational models directly predict sub-goals Consequently, Grav MAD seamlessly integrates the precision of imitation learning with the strong generalization capabilities of foundational models, leading to superior performance across a variety of manipulation tasks. Extensive experiments on the RLBench benchmark and real-robot tasks show that GravMAD achieves competitive performance on training tasks. It also generalizes well to novel tasks, demonstrating its potential for practical use across diverse 3D environments. Despite its promising results, GravMAD has some limitations. First, its effectiveness highly dependent on prompt engineering, which can be challenging for inexperienced users. Additionally, visual-language models (VLMs) have limited detection capabilities and are sensitive to changes in camera perspective, affecting performance, and preventing optimal efficiency and accuracy. Future work will address these issues to enhance the performance of the model, expand its applicability. and validate its use on more complex and long-horizon real-robot tasks.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62276128, Grant 62 69197783, Grant 62206 16;; in part by the Jiangsu Science and Technology Major Project BG2024031; in part by the Natural Science Foundation of Jiangsu Province under Grant BK20243051; in part by the Shanghai Sailing Program under Grant No.23YF141 in part by the Postgraduate Research & Practice Innovation Program of Jiangsu Province under Grant KYCX24 LCCX2400663; in part by the Fundamental Research Funds for the Central Universities (14380128); in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization.
REFERENCES
Brenna D. Argall, Sonia Chernova, Manuela M. Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics Auton. Syst., 57(5):469-483, 2009.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In International Conference on Learning Representations (ICIRR, 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Decksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Proceedings of Science and Systems (RSS), 2023a.
Robotics:
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as say: Grounding language in robotic affordances. In Conference on robot learning (CoRL), pp. 287-318. PMLR, 2023b.
Shizhe Chen, Ricardo Garcia Pinel, Cordelia Schmid, and Ivan Laptev. Polamet: 3d point clouds for Linguage-guided robotic manipulation. In Conference on Robot Learning (CoRL), pp. 1761-1781. PMLR, 2023a.
Zixuan Chen, Wenbin Li, Yang Gao, and Yiyu Chen. Tild: Third-person imitation learning by estimating domain cognitive differences of visual demonstrations. In AAMAS, pp. 2421-2423, 20236.
Zixuan Chen, Ze Ji, Jing Huo, and Yang Gao. Scar: Refining skill chaining for long- horizhorizon robotic manipulation via dual regularization. Advances in Neural Information Processing Systems (NeurlPS), 37:111679 111::11669--1111111 111714, 2024a.
Zixuan Chen, Ze Ji, Shuyang Liu, Jing Huo, Yiyu Chen, and Yang Gao. Cognizing and imitating robotic skills via a dual cognition- action architecture. In AAMAS, pp. 2204-2206 2024b.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning (ICML), volume 202, pp. 8469-8488. PMLR, 2023.
Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. Towards generalizable rision-languee robotic manipulation: A benchmark and Ilm-guided 3d policy. arXiv preprint 15.11.2410.01345. 2024.
11

---

Published as a conference paper at ICLR 2025
Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In Conference on Robot Learning (CoRL), pp. 1949-3965. PMLR, 2023.
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Ryt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning (CoRL), pp. 694-710. PMLR, 2023.
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. Rv12: Learning precise manipulation from few demonstrations. Proceedings of Robotics: Science and Systems (RSS), 2024.
Jianye Hao, Tianpei Yang. Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain.
IEEE Transactions Neural Networks and Learning Systems, 35(7):8762 8782, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Zhibo Zhao, et al. Toward general-purpose robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.0878: 2023a.
Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao, Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17844 2023b.
Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint
arXiv:2403.08244, 2024.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning (ICML), pp. 9118-9147. PMLR, 2022.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning (CoRL), pp. 540-562. PMLR, 2023.
Stephen James and Andrew J Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2): 2022.
Stephen James, Zicong Ma. David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew Davison. Coarse-to-fin q-attention: Ef- ficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13739-13748, 2022.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In
Conference on Robot Learning (CoRL), pp. 991-1002. PMLR, 2022.
Xuhui Kang, Wenqian Ye, and Yen-Ling Kuo. Imagined subgoals for hierarchical goal-conditioned policies. In CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023.
Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. In Conference on Robot Learning (CoRL). PMLR, 2024.
Sebastian Klemm, Jan Oberländer, Andreas Hermann, Ame Roennau, Thomas Schamm, Marius Zollner, and Rüdiger Dillmann. Rrt* connectnet:: Faster, asymptotically optimal motion planning. In 2015 IEEE international conference on robotics and biomimetics (ROBIO), pp. 1670-1677. IEEE, 2015.
12

---

Published as a conference paper at ICLR 2025
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic- sam: Segment and recognize anything at any granularity. arXiv preprint ar TrXY:22377.04777 2023.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. In International Conference on Learning Representations (ICLR), 2024.
Jacky Liang Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy 7eng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Xiao Ma, Sumit Patidar, lain Haughton. and Stephen James. Hierarchical diffusion policy for kinematics- aware multi-task robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18081-18090, 2024.
OpenAl. GPT-4 technical report. CoRR, http/2333.087744 2023.
Abhishek Padalkar, Ajinkya Jain, Alex Bewley, Alexander Herzog, Alex Irpan, Alexander Khazatsky, Anant Raj, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limeries Beomjoon Kim, Bernhard Schölkopf. Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov. Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Gregory Kahn, Hao Su, Haoshu Fang. Haochen Shi, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, and et al. Open x eembodiment Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6892 6903. IEEE, 2024.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence (AAAI), 2018
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5099-5108, 2017b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML), pp. 8748-8766 8763. PMLR, 2021.
SP Sharan, Ruihan Zhao, Zhangyang Wang, Sandeep P Chinchali, et al. Plan diffuser: Grounding 11m planners with diffusion models for robotic manipulation. In Bridging the Gap between Cognitive Science and Robot Learning in the Real World: Progresses and New Directions, 2024.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning (CoRL), pp. 785-799 PMLR, 2023.
Mohit Shridhar, Yat Long Lo, and Stephen James. Generative image action models. In 8th Annual Conference on Robol Learning (CoRL), 2024.
Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata V2: A dataset for robot learning scale. In Jie Tan, Mare Toussaint, and Kourosh Darvish (eds.), Conference on Robot Learning (CoRL), volume 229, pp. 1723-1736. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
Zhou Xian, Nikolaos Gkanatsios, Theophile Gervet, Tsung-Wa Wei Ke, and Katerina Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In Conference Robot Learning (CoRL), pp. 2323-2339, PMLR, 2023.
Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. In IEEE International Conference on Robolics and Automation (ICRA). pp. 3153-3160. IEEE, 2024.
Ge Yan, Yueh-Hua Wu, and Xiaolong Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint arXiv:2403.04115 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. CoRR, abs/23 11411, 2023a.
Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Soung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision). CoRR, (tss/2309.1442)) 2023b.
Yida Yin, Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roei Herzig. In-context learning enables robot action prediction in llms. arXiv preprint arXiv:2410. 12782, 2024.
Wentao Yuan, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. M212: Multi-task masked transformer for object-centric pick and place. In Conference Robot Learning (CoRL), pp. 3619-3630. PMLR, 2023.
Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Jie Tan, Mare Toussaint, and Kourosh Darvish (eds.), Conference on Robot Learning (CoRL., volume 229, pp. 284-301. PMLR, 2023.
Yanjie Ze. Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In Proceedings of Robotics: Science and Systems (RSS), 2024.
Junjie Zhang, Chenjia Bai, Haoran He, Zhigang Wang, Bin Zhao, Xiu Li, and Xuelong Li. Sam-e: Leveraging visual foundation model with sequence imitation for embodied manipulation. In International Conference on Machine Learning (ICML), 2024.
Tong Zhang, Yingdong Hu, Hanchen Cui, Hang Zhao, and Yang Gao. A universal semantic- semantic-geometric representation for robotic manipulation. In Conference on Robot Learning (CoRL), pp. 3342-3363. PMLR, 2023.
Hongkuan Zhou, Xiangtong Yao, Yuan Meng, Siming Sun, Zhenshan Blng, Kai Huang, and Alois Knoll. iraguage-conditionen learning for robotic manipulation: A survey. arXiv preprins arXiv:2312. 2082. 2023.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wes Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez. Arenas, and Kehang Han. models transfer web knowledge to robotic control. In Conference on
RT-2: saiionlangguge-aan Robot Learning (CoRL), volume 229, pp. 2165-2183, PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 GRAVMAP GENERATION PROCESS
Algorithm I: GravMap Generation Process
Input: End-effector position gmt initial gripper openness M target gripper openness g map size Sm. offset range 3o. radius S,, downsample ratio Sd, number of sampled points Np, inference mode flag in. Jerence
open
Output: GravMap m
begin
Initialize cost map mer gripper map and evcidance map ma with size 9 Initialize me. Mg. and ma with shape Sm X Sm Sm. setting m.(u, w) = 1, mg(u U, w) = Pinit and ma (u, U, w) = 0 for all voxels (u, U, w);
/1 Convert 9rd Irom world coordinates (a, v) to voxel coordinates (6,),))
Extract (x, 2) g Convert (z, y, 2) to voxel coordinates (i. k);
// Determine voxel coordinates ($',5",') based on mode
if not inference then
/ // Apply random offseta 6, (f,j,k) for data augmentation
Sample ōj, 6g ~ Uniform( Bo);
Update voxel coordinates: (i', k) (i + 8, + أj, 8n);
else
// Use criginal vexel coordinates inCerence mode (K'J",k)= = -
// Compute Euclidesn distance from (K,3,K)
(a.v,)) w)
For each voxel (u, U, w), compute D(u, 01 w) = (u--u 17)2 (w 8& (w k2)
// conatruct avo dance map m.
Set ma (u, U, w) = 1 for all occupied voxels in the scene; / Update ma by excluding vexels near target ((')",,')
Set ma (u, U, w) = 0 for voxels where D(u, U, w) < 0.15 Smi Smooth ma with Gaussian filter (a 10);
// Compute and normallze the cost map
Set me(U, v, w) = D(u,,w)) for all voxels (u, v, w);
max 12
// Combine me and ma into the final coat map
Update mu,u, w) = 22me(u, U. w) + (u, w) for all voxels (u, w,w): Normalize m. to the range [0;
/ / Set my within redius 9, of (K')',k))
Set My(U, U, w) = go for voxels where D(u, u, w) S Br: // Dewnsample both
and my
Downsample m, and my by Ba;
Select N, points {up} from the downsampled m. and my using Farthest Point Sampling; // Construst GravMap using sampled points
Form =[(yg, me(1))
return m,
A.2 HEURISTIC' FOR SUB-GOAL KEYPOSE DISCOVERY
Building on keypose discovery (James & Davison 2022), we propose the Sub-goal Keypose Discovery method identify sub-goal keyposes from demonstrations, focusing changes in the gripper's 'ss state and touch forces. This is particularly relevant for object manipulation tasks, where the robot's interactions with objects can be segmented into discrete sub-goals.
The implementation of the Sub-goal Keypose Discovery algorithm starts with a set cre-comptted keyposes, which are frames selected from the demonstration sequence through initial keypose discovery process. We introduce two functions: touch touch_channgee change, shown in Algorithm [ and hilpppppeaaggggeepee change, shown in Algorithm 3 to evaluate whether keypose qualifies sub-goal.
15

---

Published as a conference paper at ICLR 2025
The first function checks for significant changes in the gripper's touch forces, while the second evaluates changes in the gripper's open/close state. The pseudocode Algorithm 4 outlines the heuristic steps for identifying sub-goal keyposes.
One current limitation the Sub-goal Keypose Discovery method is its inability to effectively handle tasks involving tool use, which we plan to address in future research.
Algorithm 2: touch caacc__hangeaa Function
Input:
Demonstration sequence demo, Keypose index k, Threshold touch thecchhhd Tolerance delta
Output:
Boolean indicating significant touch force change
begin
Set start max(0, k touch threshold); for each index j from start to k-l do if Touch forces at j differ from Touch forces at k within tolerance delta then return True;
return False;
Algorithm 3: gripper_ gripper_change change Function
Input: Demonstration sequence demo, Keypose index k, Threshold fitipper_trrsholle Output: Boolean indicating gripper state change begin
Set start to max(0, k gripper gripper_threshoie threshold): for each index j from start to k-1 do if Gripper state j differs from Gripper state at k then return True;
return False;
Algorithm 4: Heuristies for Sub-goal Keypose Discovery
Input: Demonstration sequence demo, Task type task str, Threshold parameters touch_thrent thheshoreeo
delta
Output: List of sub-goal keyposes sub gub_ keypose begin
empty list; Identify keyposes from demo using keypose discovery method; for each keypose k in keyposes do
Initialize sub_goal_ keyposes
task str str is a task involving touch without grasping then touch macchihreshold. delta) then
if touch totch_cchangee change(demo, Append & to sub sub_goal_ keyposes, else
if gripper-ehheee change(demo, gripper_threshold) touch touch_cchageeee change(demo, k. touch cirechhhhrrhhddde delta) then
Append & sub goaa_ooaaaaa keyposes;
Append the last keypose to sub_goal Lub__goll_eepppnlla keyposes; return sub goa_ goallkeyposese
16

---

Published as a conference paper at ICLR 2025
Algorithm 5: GravMap Generation
Input: Instruction É, Observed RGB Image
Prompt: Prompt for Detector Paet. Prompt for Planner Pplan. Prompt for Composer Poom.
Prompt::rompp
Few-shot task specified prompt Plask = (P P P Poom) ) Cost Map Prompt Posst
plan
Gripper Map Prompt P gripper
Output: GravMap m
begin
O' Semantic-SAM(()) // Label objects with numerical tags Detector(E, O', Pax. Pa Select relevant ob and get corresponding 3D positions as context
ST + Planner((,, C, Pplan P Plan): / Infer sub-tasks ST (st1, st2....,, // Generate API calls and their parameters for generating go0d and gopen wwwccoststtsoooooeeeee map(Function calls, parameters, Poos);; g + gge__cost,
Function calls, parameters & Composer(ST,C.. Pomm P Poom)
9 open + get_gripper_maar Funccion manunnctton calls, parameters, P Pripperr;; < GravMap (eatratto(cc(gP ))opn));
return m:;
Propricccpiion
Demoising stop
GrayMap
oedane
Mrp taken
2D Visual
Cross-X flestion
Cross Allintiion
Self-Atcation
Auailiary
Encoder
Openness Head
Auxihary
Language Fnooder
Position Head
Openness Head
Similer Asx Open Hexd
JUL
Position Head
Sunilar to Aux Pos Head
Scene
"Take the chicken off the grill"
Figure 5: Detailed description of the modules in GrayMAD, including the 3D Scene Encoder and the prediction heads
A.3 DETAILS OF GRAVMAP SYNTHESIS
A.3.1 TRAINING PHASE
To facilitate GravMap synthesis, we assign goal action to each keypose by linking it to the action performed at the nearest future sub- sub-goal. This association enables us determine the relevant cost and gripper state for different regions of the GravMap. In the first map, m. E Ruxxidd the cost lower near the positions of the robotic end-effector at these sub-goal keyposes and higher as the distance increases. In the second map, m, gexhxd, areas the end-effector's position at the sub-goal keyposes reflect the gripper state at the sub-goal, while other areas reflect the gripper state at the current frame.
A.3.2 INFERENCE PHASE
In this section, we introduce the complete pipeline for GravMap generation, as outlined in Algorithm This includes Algorithm I which details the process of generating GravMap from language instruction l and an observed RGB image O. The GravMap generation pipeline integrates VLMs to interpret instructions, ground them in the visual context, and translate them into coarse 3D voxel representations, i.e., the GravMap.
Our pipeline consists of the following three components:
17

---

Published as a conference paper at ICLR 2025
Detector. Starting with instruction é and observed RGB image O, the RGB image is passed through the Semantic- segmentation model, which labels each object with a numerical tag. producing labeled image O'. The GPT4o-based Detector uses the prompts Pae and Pet (adapted from Huang et al. 2024)) to select relevant objects and obtain their 3D positions. The output is set of selected objects, context C, which includes the objects' identities and their spatial coordinates in the 3D environment. In the VLM setting, the Detector accesses initial RGB images from four views: wrist, left shoulder, right shoulder, and front camera. In the manual setting, precise 3D object attributes are provided from the simulation.
Planner. The GPT4o-based Planner takes the instruction E, context C, and planner-specific prompts Ppln and plan (adapted from Huang et al. (2023)) infer sequence of sub-tasks action interaction needed to fulfiil the
(st1, st21,t2,.... st....., st,). Each sub-task describes instruction L. Progress is tracked based on the robot's gripper state (open/closed) and whether it is holding object. The current sub-task is then passed to the Composer for further processing.
Composer. Following Huang et al. (2023), the GPT4o-based Composer parses each inferred sub-task st, using corresponding prompts Poom and P Poom- The Com- poser generates the sub-goal position 9TTH and sub-goal openness g by recursively map,
generating code. This includes calls get_cost cost map and which are triggered by cost map prompt Pana and gripper map prompt P Pripperr- For example, for a sub-task like "push close the topmost drawer," the Com- poser might generate: get_cost map ('a point 30cm into the topmost drawer handle') ) and get_gripper__ueeeeeeeeeee_eeeeer map (' close everywhere') ). Natural language parameters are parsed by GPT to generate code that assigns values g and g The final GravMap generator in Algorithm then processes g and go to generate the GravMap m.
The prompts mentioned above can be found on the wehsite: htttpp::/graavad..iiitt
A.4 DETAIL OF MODEL ARCHITECTURE AND HYPER-PARAMETERS FOR GRAVMAD
The detailed hyperparameters of GravMAD are listed in Table 3 Additionally, Fig. 5 provides a detailed overview of GravMAD's modules, including the 3D Scene Encoder and the prediction heads.
(a) The 3D Scene Encoder processes visual and language information separately, merging them via a coss--attention mechanism, with proprioception integrated through FiLM. This allows the model to understand tasks like "Take the chicken off the grill" a 3D environment. First, the visual input is processed by 2D Visual Encoder, transforming image data into feature representations. These 2D features are then passed through a 3D lifting module, converting them into 3D representations using depth information. Simultaneously, the language input, such as the instruction "Take the chicken off the grill", is encoded into language tokens by the Language Encoder. Finally, the 3D visual features and language tokens are combined through cross-attention, producing 3D Scene tokens.
(b) Each prediction head consists of Attention layers and an MLP. The Auxiliary Position Head receives tokens from the previous layer, which first go through cross-attention with GravMap tokens, followed by self-attention to refine the features. The tokens are then passed through an MLP to output the sub-goal end-effector position. Similarly, the Auxiliary Openness Head takes tokens from the self-attention layer of the Auxiliary Position Head and uses an MLP to predict the sub-goal gripper openness. The Position Head follows the same process as the Auxiliary Position Head, while the Openness Head mirrors the Auxiliary Openness Head. The Rotation Head processes tokens with self-attention and an MLP to predict rotation error.
A.5 COMPARISON BETWEEN GRAVMAD AND OTHER BASELINE MODELS
We compare GravMAD with Voxposer (Huang et al. 2023 and 3D Diffuser Actor (e et al. 2024) in Fig. 6
(a) Voxposer. We describe our reproduction of Voxposer on RLBench. Voxposer uses our SOM- driven Detector to process the input observation and instruction, generating context information. The
18

---

Published as a conference paper at ICLR 2025
Values
Sub-goal Keypose Discovery
touch_thin thch_threshold Tolerance: delta thoper_trresood
0.005
GravMap
map_sie sizes S.
100
offset_range: 8%
radius: 1,
downsample ratio: 3g
number of sampled points: Np
1024
Model
image size token dim
diffusion_timestep timestep
diffusion
noise scheduler: position
scaled linear
noise schecceuuele:: rotation
squaredeos
action space
absolute pose
Train
batch_size
optimizer
Adam
train_iters iters
600K
learning rate
le
weight
Semd
loss weight:
30
loss weight:
10
loss weight: Wy
30
loss weight: Wa
10
Evaluation
maximal step except push_ button button_light maximal step of mush_button__iie
Table 3: Hyper-parameters for GravMAD, including Sub-goal Keypose Discovery, GravMap, model configuration, training, and evaluation.
Planner then receives this context and outputs sub-goal, representing intermediate step necessary for the overall motion plan. The Composer processes this sub-goal, producing three maps: Cost Map, Rotation Map, and Gripper Map. These maps guide the robot's movement toward the target in the environment. Note that Voxposer's testing process involves a different number of steps compared to 3D Diffuser Actor and GravMAD, completing only after all LLM inferences executed.
(b) 3D Diffuser Actor, We use the language-enhanced version of 3D Diffuser Actor as a baseline. 3D Diffuser employs 3D Scene Encoder to transform visual and language inputs into 3D Scene tokens, providing an understanding of the 3D environment. An MLP encodes noisy estimates of position and rotation into corresponding tokens, which are then fed, along with the 3D Scene tokens, into denoising network for action diffusion. This network, conditioned on proprioception and the denoising step, includes attention layers, Openness Head, Position Head, and Rotation Head. During diffusion, noisy position/rotation tokens attend to 3D Scene tokens, and cross-attention with instruction tokens enhances language understanding. These instruction tokens are also used in the prediction processes of the Openness, Position, and Rotation heads.
(c) GravMAD (ours), GravMAD shares components with Voxposer, such as the Detector, Planner, and Composer, but incorporates task-specific prompt engineering. Unlike Voxposer, which uses maps for planning, GravMAD encodes these maps into tokens using point cloud encoder, which are then employed in the action diffusion process. Compared to 3D Diffuser Actor, the key difference is that GrayMAD uses GravMap tokens instead of language tokens, improving generalization. Additionally, GrD introduces two auxiliary tasks predict sub-goals, enhancing representation learning.
19

---

Published as a conference paper at ICLR 2025
Detectox
instr
Motion Planning
AP
Linent
Cross-- Aieetion tion
Cross- Attention
Self-A Atention
Grapper Opennoo
Jennion
Cird Map
Pontlue Hival
Planner
Retation Map
Jatrpal
Gripper Map
Anaiie Roud!
Composse
(4) Yaxposer using our Detector
(b) aD Diffiser Actor with enhanced language conditioning
Austllary Productios for Training
abs instr
Deiectar
Qenoers
Aunilen
Sea- goal Opennas
Jonneet
Ausiler
Planner
Cool Map
Opanmnn Hoal
Graper
Compeser
Puniiix Hosd
Dewentin
GravMAD
Figure 6: Comparison of GravMAD with Voxposer and 3D Diffuser Actor. Unlike Voxposer, which uses planning, GravMAD leverages GravMaps for learning. Compared to 3D Diffuser Actor, GravMAD employs GravMap tokens guide the action diffusion process and introduces auxiliary position and openness heads to improve representation learning.
B
ADDITIONAL EXPERIMENTAL DETAILS
B. BASE TASK
Eight Action Primitives in Robotic Manipulation Tasks
Press
Pick
Push
Screw
Close
Open
Insert
Stack/Place
eg.
cp. Mov pmi
eg n6
Cherser
cg Ujer diveer
6s huntyo
Place
Figure 7: Eight action primitives in robotic manipulation tasks.
For the selection of base tasks, our primary criterion to ensure they comprehensively cover the fundamental action primitives in robotic manipulation tasks. Therefore, we follow [aarcia et al. (2024) and further summarize the eight essential action primitives required for robotic manipulation. as shown in Fig. 7 In line with this criterion, we select 12 base tasks from RLBench (James et al. 2020), as illustrated in Fig. These 12 tasks also include short-term tasks (close jar, open drawer, meat off grill, slide block, push buttons, place wine), long-horizon tasks (put item in drawer, stack blocks, stack cups), and tasks that require high-precisiio manipulation (screw bulb, insert peg, place cups). Each base task contains 2 to 60 variants in the instructions, covering differences in color, placement, category, and count. addition to instruction variations, the objects, distractors, and their positions and scenes randomly initialized in the environment. The templates representing task goals in the instructions are also modified while maintaining their semantic meaning. A summary of the 12 tasks is provided in Table
We provide a detailed description of each task below and explain modifications from RLBench origin codebase.
20

---

Published as a conference paper at ICLR 2025
Close
Open Drawer
Screw Bulb
Meat off Grill
Slide Block
Put in Drawer
Push Buttons
Stack Blocks
Insert Pog
Stack Cups
Place Cups
Place Wine Rack
Figure 8: Visualization of 12 base tasks.
Table The base tasks selected from RLBench James et 2020
Task
Variation Type
#f Variations
Avg. Keyposes
Language Template
close
color
20
6.0
"close the jar"
open drawer
placement
3.0
"open the drawer"
screw bulb
color
20
7.0
"screw in the light bulb"
meat off grill
category
5.0
"take the off the grill" target"
slide block
color
4.7
"slide the block
put in drawer
placement
12.0
"put the item the drawer"
push buttons
color
50
3.8
"push the button, [then the button]"
stack blocks
color,
60
14.6
"stack blocks"
insert peg
color
20
5.0
"put the ring the spoke"
stack cups
color
20
10.0
"stack the other cups top of the cup"
place cups
11.5
"place cups the cup holder"
place wine
5.0
"stack the wine bottle to the of the rack"
B.1.1 CLOSE JAR
Task: Close the jar by placing the lid on the jar.
filename: close_jar.py
Modified: The modified success condition registers single DetectedCondition to check the jar lid correctly placed on the jar using proximity sensor, discarding the previous condition of checking nothing is grasped by the gripper.
Success Metric:
The jar lid is successfully placed the jar as detected by the proximity sensor.
1.1 OPEN DRAWER
Task: Open the drawer by gripping the handle and pulling it open.
filename: open_ _drawerrryppyy
Modified: The camm_over_shooe Ioouudeer der_ left camera's position and orientation were modified
to better observe the drawer. The camera was repositioned to [0.2, 0.90, 1.10] and reoriented to
m0.5* pi, O, O].
Success Metrie: The drawer is successfully opened to the desired position detected by the joint condition on the drawer's joint.
1.1.3 SCREW BULB
Task: Screw in the light bulb by picking it up from the holder and placing into the lamp. filename: light_ bulb bulp_in.py
21

---

Published as a conference paper at ICLR 2025
Modified: No.
Success Metric: The light bulb is successfully screwed into the lamp and detected by the proximity sensor.
B.1.4
MEAT OFF GRILL
Task: Take the specified meat off the grill and place it next to the grill.
filename: meat_off__gt catt_ofgrrillppyyy
Modified: The cam_over_shoul der_right camera's position and orientation were modified to better observe the drawer. The camera was repositioned to [0.20,-0.36.1.85] and reoriented to m-0.85%math.pp 0, math.pi].
Success Metrie: The specified meat successfully removed from the grill and detected by the proximity sensor.
B.1.5 SLIDE BLOCK
Task: Slide the block to the target of specified color.
filename: slide boock__looott__oe cat__too_oooottrreettoeo
Modified: No.
Success Metric: The block is successfully detected on top the target color as indicated by the proximity sensor.
1.1.6 PUT DRAWER
Task: Put the item in the specified drawer.
filename: put_item_in_ iiee_jnn_drawer.py
Modified: The cam_over cam_over_shoollert shholddr_lees camera's position and orientation were modified
to better observe the drawer. The camera was repositioned to [0.2, 0.90, 1.15) and reoriented to
m0.5*math.pi. pi, 0, 0].
Success Metric: The item is successfully placed in the drawer as detected by the proximity sensor.
B.1.7 PUSH BUTTONS
Task: Press the buttons of the specified color in order filename: _estoosut
Modified: No.
Success Metrie: The buttons are successfully pushed in order.
B. STACK BLOCKS
Task: Stack specified number of blocks of the same color in a vertical stack.
filename: stack_ fack_blocks.p
Modified: No.
Success Metric: The blocks are successfully stacked according the specified color and number.
B. INSERT PEG
Task: Insert square ring onto the spoke with the specified color.
filename: insert_ontt_squure__ees
Modified: No.
Success Metric: The square ring is successfully placed onto the correctly colored spoke.
B.1.10 STACK CUPS
Task: Stack two cups on top the cup with the specified color. filename: stack _ypsspy
Modified: No.
Success Metric: The cups successfully stacked with the correct cup the base.
22

---

Published as a conference paper at ICLR 2025
Meat Grill
Stack Cups Blocks
Close Jar Banana
Close Distractor
Close Drawer
Open Small Drawer
Condition Block
Push Button Light
Figure 9: Visualization of novel tasks.
Robustness to Distractors Shape Variations
Action Understanding Meat on Grill
Understanding Language Reasoning
Stock Cups Dlocks
Close Jar Banana
Close Distractor
Close Drawer
Condition Black
Push Button Light
Open Drawer
Long
Figure 10: Three Novelty Categories for the Novel Tasks.
B.1.11
PLACE CUPS
Task: Place a specified number of cups onto a cup holder.
filename: place_cups,
Modified: No.
Success Metric: The cups successfully placed onto the holder according the task instructions.
1.122 PLACE WINE AT RACK LOCATION
Task: Place the wine bottle onto the specified location on the wine rack.
filename: place_ wine at lar_rac__ooeee locc_oocation.py Modified: No.
Success Metric: The wine bottle is successfully placed at the correct rack location and released from the gripper.
B.2 NOVEL TASK
As shown in Fig. 9 we create novel tasks that differ from the original training tasks to test policy generalization. These tasks feature scenes and objects similar to those the training tasks. We further define the novelty categories of the 8 novel tasks in our experiments to better explain the generalization improvements brought by GravMAD. As shown in Fig. [0] the designed novel tasks introduce three types of challenges to the model: Action Understanding (meat grill, close drawer), Visual Understanding & Language Reasoning (stack cups blocks, push buttons light, close jar banana. condition block)-includee including two morizoo tasks (stack cups blocks and condition block), and Robustness to Distractors or Shape Variations (stack cups blocks, push buttons light, close jar banana, close jar distractor, open small drawer, condition block).
23

---

Published as a conference paper at ICLR 2025
Table The novel tasks changed based on base tasks.
Task
Voriation Type
of Variations
Avg. Keyposes
Language Template
drower
placement
2.0
drower"
close jur bamara
placement
6.0
"close the jar closer to the banana"
close jar distrector
color
20
6.0
"close the jar"
condition meal on grill open small driwer stack cups blocks
soler, calegory
"build tall lower of cubes, add black block
5.0
"pul Uhe
the grill"
placement
3.0
"open the drawer"
color
20
10.0
"Identify the most
color the block and stack the other cups
cup motchee color"
push button light
color
20
2.0
"push the button with the same color as the light"
Specifically, Action Understanding refers to tasks involving changes in interaction actions with objects; Visual Understanding & Language Reasoning involve introducing entirely new operational rules or conditions compared to known tasks; and Robustness to Distractors or Shape Variations includes tasks that require interaction based on fixed object attributes (such as color, size, distance, or distractors). A summary of the seven tasks is provided in Table 5 We provide detailed description of each novel task below and explain the modifications from the base tasks.
B.2.1 MEAT GRILL
Task: Place either chicken or a steak on the grill depending on the variation.
filename: meat_ _o__oonnggloooooo grill.pl
Base task: meat off grill.
Modified: The task requires placing meat onto the grill, whereas the base task involves removing it. The cam Lovvrvveeeeeoooo aederreriirrr right camera's position and orientation were modified to better observe the drawer. The camera was repositioned to [0.20,-0.38 -..3,,...6..855 and reoriented to mathhi. mith.pi].
Success Metrie: The selected meat (chicken steak) is successfully placed the grill and released from the gripper.
B.2.2 STACK CUPS BLOCKS
Task: Identify the most common color in the block pile, and stack the other cups on the cup that matches that color.
filename: stack_cups sups_blocks.py
Base task: Stack cups.
Modified: The task involves identifying the cup that matches the most common color among the distractor blocks, then stacking the other two cups top. The base task is simply stacking the cups without considering block colors.
Success Metric: Success is measured when the correct cup is stacked with the other cups based the color identification and all cups are within the target area defined by the proximity sensor.
B.2.3 CLOSE JAR BANANA
Task: Close the jar that is closer to the banana by screwing on its lid.
filename: close_jar_ ja_jjarraaaaaapppppaaa
Base task: close jar.
Modified: The task involves identifying the jar closer to the banana and screwing its lid on, while the
base task only requires closing jar without proximity consideration.
Success Metric: The lid is successfully placed on the jar closest to the banana, confirmed by the proximity sensor.
B.2.4 CLOSE JAR DISTRACTOR
Task: Close the jar by screwing on the lid, while distractor objects are present.
filename: close_jar_dee jaaj_ddstrrttorrppppppp
Base task: close jar.
Modified: The task includes distractor objects, such button and block, which colored and placed near the jars. These objects have been encountered during training, adding complexity compared to the base task.
24

---

Published as a conference paper at ICLR 2025
Success Metrie: sensor.
The jar lid is successfully placed the target jar, confirmed by the proximity
B.2.5 CLOSE DRAWER
Task: Close one of the drawers (bottom, middle, or top) by sliding it shut.
filename: ues__drawer.py
Base task: open drawer.
Modified: The task involves closing the drawer instead of opening
Success Metric: The selected drawer is closed successfully, confirmed by the joint position of the drawer.
B.2.6 OPEN DRAWER SMALL
Task: Open one of the smaller drawers (bottom, middle, or top) by sliding it open.
filename: open_ ttraer_smalllpp
Base task: open drawer.
Modified: The task involves opening smaller drawer compared the base task, with adjusted camera settings for better visibility.
Success Metric: The selected drawer is opened successfully, verified by the joint position of the drawer.
B.2.7 CONDITION BLOCK
Task: Stack specified number of blocks and, if the black block is present, add it to the stack. filename: condition condition_block.py
Base task: stack blocks.
Modified: The task involves stacking specified number of blocks, with an additional requirement to include the black block if it is present.
Success Metrie: The correct number of target blocks are stacked, and if the black block is present, also correctly added to the stack.
B.2.8 PUSH BUTTON LIGHT
Task: Push the button that matches the color of a light bulb the first attempt.
filename: push_ buttons buttons_light.py
Base task: push button.
Modified: The task involves pressing a single button that matches the color of a light bulb. The button must be pressed correctly on the first attempt; repeated attempts are not allowed.
Success Metric:
The correct button matching the light bulb's color is pressed on the first attempt.
B.3 FAILURE CASES OF GRAVMAD
In this section, we analyze why GravMAD underperforms compared to the baseline model 3D Diffuser Actor on certain base tasks, particularly in the "Place Wine" task and drawer-related tasks.
As discussed in the main paper, GravMaps represent spatial relationships in 3D space, but this introduces challenge: areas close the sub-goal often share the same cost value, as seen in the value map the right side of Fig. (a). This uniform cost value can mislead the robot into assuming should complete the sub-goal within that area. For tasks requiring precise actions, such as the "Open Drawer" task, Grav coarse guidance may lead to suboptimal performance compared to 3D Diffuser Actor. In the left schematic of Fig. aT), the robot must grasp the center of small handle to achieve optimal performance in the "Open Drawer" task. This high precision demand on the end-effector results in a lower success rate for GravMAD. This limitation extends to the "Put in Drawer" task, which depends the successful completion of "Open Drawer". Similarly, in the "Place Wine " task, insufficient predictive accuracy causes the robot to misalign the bottle with the correct slot by one unit, leading to failure.
In the VLM setting, sub-goal accuracy often suffers, as shown in Fig, b)), further reducing model performance. These inaccuracies typically arise from factors: (1) may fail to accurately
25

---

Published as a conference paper at ICLR 2025
"stuck the wine bottle the left the rack"
target
"slide the middle drawer open". "
a Visualization of failure examples
Imprecise labels
Desired labels
(b) Comparison of imprecise labels and desired labels
Figure 11: Failure cause analysis, including (a) visualization of failure examples; (b) comparison of imprecise labels and expected labels.
identify ideal areas, leading to imprecise contextual information from the Detector module for tasks like "Place Wine", "Open Drawer". and "Put in Drawer": (2) the camera's positioning may not capture the full scene, leaving some task-relevant objects out of view, as seen in tasks like "Meat off Grill". To overcome these VLM limitations, potential solutions include: (1) integrating multi-view information into the Detector for more comprehensive scene observation; and (2) using a more granular segmentation model to provide GPT-4 with a wider range of labels, improving the quality of the context generated by the Detector.
C DISCUSSION
C. THE RELATIONSHIP AND DIFFERENCES BETWEEN GRAVMAP AND VOXPOSER
The GravMap in GravMAD and the value maps in Voxposer (Huang al. 2023 share the following connections and differences:
Number of value maps involved: Voxposer utilizes multiple value maps, including the cost map. rotation map, gripper openness map, and velocity map. In our method, we only combine the cost map and gripper map. and their numerical values remain identical at this stage.
Structure and processing: We further downsample the cost map and gripper openness map, transforming them into a point cloud structure containing position information and gripper states (z, 2, me, my), which we term GravMap. This sparse data structure not only efficiently represents sub-goals but also allows feature extraction using a point cloud encoder.
26

---

Published as a conference paper at ICLR 2025
C.2 THE REASON FOR NOT USING THE ROTATION MAP FROM VOXPOSER
Grav does not currently use the rotation map from Voxposer because incorporating the rotation map could introduce significant distributional shifts between the guidance provided during the training and inference phases. During training, precise rotation guidance can be derived from expert trajectories. However, during inference, off-the-shelf foundation models often struggle to accurately interpret rotation information from visual and linguistic inputs, making it challenging to provide precise rotation guidance. To address this issue, future research will explore integrating rotation information from expert trajectories with object poses to generate few-shot prompts for off-the-shelf foundation models (Yin et al.) 2024). This approach aims to enable LLMs to produce effective rotation guidance while reducing distributional shifts relative to the training data.
C.3 FURTHER DETAILS ON SUB-GOAL KEYPOSE DISCOVERY
C.3.1 WHY SUB-GOALS ARE EXTRACTED DIFFERENTLY DURING TRAINING AND INFERENCE
During the training phase of GravMAD, we use Sub-goal Keypose Discovery to extract sub- sub-goals and generate GravMaps based on them. In contrast, during the inference phase, sub-goals are inferred by foundation models to generate GravMaps. The reasons for adopting different methods to generate GravMaps during the training and inference phases are as follows:
Efficiency and reliability during training: Using Sub-goal Keypose Discovery to extract sub-goals during training is both simple and efficient. If foundation models were directly used to generate GravMaps as guidance during training, while they can indeed produce GravMaps, the results are generally coarser, less precise, and slower compared to expert trajectories. For example, due to limitations such as camera resolution or angles, foundation models may fail to fully observe the scene in some cases, leading to inaccurate aub-goal positions (failure cases quality of the training data cannot be guaranteed. Additionally, using foundation models to process large-scale data is practically infeasible due to their slow processing speed.
discussed in Appendix [.3).. Under such circumstances, the
Simplifying the problem by avoiding semantic reasoning: Extracting sub-goals from expert trajectories focuses solely on analyzing the robot's 's actions, thereby avoiding the complexity of semantic understanding and reasoning. Our key insight is that in task tra- jectories, certain actions in expert trajectories inherently carry semantic information (i.e., sub-goals, which may involve direct interactions with objects). These actions often exhibit distinctive features, such as the opening and closing of the gripper. The Keypose Discovery method (James & Davison 2022) has already performed an initial filtering of these key actions, narrowing the scope for sub-goal selection. Based on this, we can quickly identify sub-goals through heuristic methods, which are also effective for long-horizon tasks.
is worth noting that using different sub-goal generation methods during the training and inference phases may lead to distributional shift. This occurs because the sub-goals generated by foundation models during inference are often less precise compared to those derived from expert trajectories, resulting in discrepancy between the distributions of the training and inference phases. To address this issue, we apply data augmentation to the precise sub-goals generated from expert trajectories during the training phase. Specifically, as described in Line 279 of Algorithm []] introduce random offsets to the sub-goals generated during training (this processing is not applied to sub-goals generated during inference) and then generate GravMaps based on these perturbed sub-goals. This approach effectively reduces the risk of distributional shift to a certain extent.
C.3.2
WHY USE SUB-GOAL KEYPOSE DISCOVERY TO FILTER KEYPOSES
The Sub- gob-gool Keypose Discovery method is essential for GravMAD because the original keyposes include both sub-goal keyposes and the intermediate steps required to achieve these sub-goals. These intermediate steps may involve precise alignment of the robotic arm with objects. However, foundation models often struggle to generate these intermediate steps, and even if they can, the results may exhibit significant distributional shifts compared to the guidance provided during the training phase. Additionally, generating only sub-goals reduces the complexity and difficulty of task reasoning for the foundation model while also simplifying the prompt engineering.
27

---

Published as a conference paper at ICLR 2025
The Filtered Sub-goals
Sub-goal keypose Discovery
The Original Keyposes
Figure 12: A comparison between the original keyposes and the filtered keyposes in the long-horizon task put item in drawer.
Gripper Touch Ferces for Task: push buttons, Demo: 23
3373
Smoothed
orgina Farces
312
0:00
3.25
"push the maroon button, then push the green button, then push cyan button" "
Figure 13: Visualization of sub-goal keypose discovery determining significant changes in grip- per_torch torch force during the push button task.
As shown Fig. 12 for the long-horizon task put item druwer, if only traditional keypose discovery methods are used, the extracted sub-goal stages would include 11 stages. In contrast, when using our Sub-goal Keypose Discovery, the filtered sub-goals are reduced to just 4 stages, perfectly aligning with the most critical phases of the task. This significantly reduces model inference time and improves task execution efficiency.
C.3.3 CRITERIA FOR "SIGNIFICANT CHANGES" IN SUB-GOAL KEYPOSE DISCOVERY
To clearly explain the specific criteria for "significant changes" in our Sub-goal Keypose Discovery method, we visualized the changes in gripper_tooe touch force using the push buttons task as an example. As shown in Fig. [3] when the button pressed, the gripper_touu touch force value increases from nearly to 0.1 0.15. As the robotic arm lifts, the gripper touch force returns to O. By analyzing these force changes,
intuitively identify the sub-goal frames.
28

---

Published as a conference paper at ICLR 2025
Voxposer
Act3D
3D Diffuser Actor GravMAD (ours)
Avg. Inference Time for Keypose Prediction (secs)
0.04
1.78
1.81
Avg. Task Completion Time (secs)
448.01 01
14.08
49.45
97.04
Avg. Inference Time Sub-task Stage (secs)
90.47
40.64
Table 6: Comparison of Inference Times.
Models
Avg. Success
Avg. Rank
Close Jar
Open Drawer
Meat off Grill
Slide Block
Pur in Drawer
Voxposer
15.11
5.88
12.00
10.67
45.33
0.00
0.00
Voxposer (Manual)
22.06
4.92
13.33
18.67
69.33
0.00
0.00
ChainedDifluser (Oracle)
29.72
4.42
82.62
0.00
52.00
2.67
0,00
Act3D
34.11
5.38
61.33
41.33
60.00
78.67
49.33
3D Diffuser Actor
55.81
3.00
66.67
88.00
88.00
84.00
94.67
GravMAD (Manual)
69.17
1.63
100.00
76.67
89.33
93.33
78.67
GravMAD (VLM)
56.72
2.79
100.00
58.67
70.67
80.00
61.33
Models
Push Buttonss
Stack Blocks
Place Cups
Place Wine
Screw Bulb
Insert Peg
Stack Cups
Voxposer
80.00
6.67
4,00
0.00
1.33
Voxposer (Manual)
86,67
36.67
13.33
10.67
6.67
0,00
9.33
ChainedDiffuser (Oracle)
62.67
13.00
22.33
48.67
25.33
4,00
41.33
Act3D
66.67
0.00
45,33
6.67
0.00
0.00
3D Difluser Actur
94.67
13.67
5.33
82.67
29.33
2.67
20.00
GravMAD (Manual)
98.67
56.67
77.33
66.67
32,00
57.33
GravMAD (VLM)
97.33
51.33
5.33
33.33
54.67
18.67
49.33
Table 7: Additional Multi-task test results on base tasks.
Avg. Success
Avg. Rank
Close
Close Jar
Close Jar Condition Meal On Open Drawer Stuck cups
Push Buttons
Models
Drawer
Banana
Distractor
Block
Grill
Small
blacks
Light
Noxposer Huang et al
2023
34.29
1.25
96.10
17.33
22.67
25.00
38.67
6.67
0.00
68
Chained
Xian et 2025
43.22
2.75
81.33
82.67
85.00
18.00
29.00
0.00
41.33
30.00
Act3D
66.67
Dimics Anine
2024
29.38
3.375
81.33
48.00
42.67
27,00
0.00
2,67
2.67
30.67
GavMAD (VLM)
76,001
Table 8: Additional generalization results on novel tasks.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 INFERENCE TIME
We test the inference time of all models under the setting of 8 novel tasks using single NVIDIA 4090 GPU. The results, shown Table (in seconds), indicate the following: models like Act3D and 3D Diffuser Actor, which do rely foundation model inference, have shorter inference times but lower success rates. In contrast, Voxposer spends a significant amount of time synthesizing trajectories. Our GravMAD requires more time than Act3D and 3D Diffuser Actor because it waits for the foundation model to process information and infer sub-goals for sub-tasks.
D.2 ADDITIONAL BASELINE EXPERIMENTS
We introduce two additional baseline methods for performance comparison: Voxposer (Manual) and Chained Diffuser (Xian et al.) 2023) (Oracle). Voxposer (Manual) means that manually provide ground truth object pose information to Voxposer instead of relying on the inference results of the foundation model. In Chained Diffuser (Oracle), provide the ideal position for each keypose, with the connections between keyposes generated using the local trajectory diffuser module from Chained Diffuser. The performance comparisons of these baseline methods on 12 base tasks and novel tasks are shown in Table 7 and Table g respectively.
From the experimental results, we observe the following:
In the base task setting, Voxposer (Manual) shows slight performance improvement when provided with ground truth object information but still falls short compared to our GrayMAD (Manual).
29

---

Published as a conference paper at ICLR 2025
vo colesssmmeettan wtw. 20_0000
Vol wws.smmeeettaggg.. ett 00200025 0025
1tSDero
1van EcOert
S1Duma
24-
0.0
1.2
Adaas :00Camo
0.0
1.-
50Dore
3.2
Tvew 20Dore
100066
646.6
0E6622
340062
566629
LAKCE
JM0E5
45623
40008:
S66020
565693
Figure 14: Comparison of validation curves under varying viewpoints and data sizes.
Close Jar Button
Pour from Cup to Cup
Push Buttons Shape
Figure 15: Visualization of additional novel tasks.
For Chained Diffuser (Oracle), the keyposes come from ideal waypoints predefined in simulation, and the model effectively connects these keyposes, achieving high success rate. However, in real- world scenarios, manually providing each keypose is impractical. Even with precise keyposes, Chained Diffuser (Oracle) still performs worse than our GravMAD (VLM).
D.3 SCALABILITY OF GRAVMAD
To evaluate the scalability of our proposed method with respect to data volume, we conduct training comparisons using five different demonstration dataset sizes and visualize the corresponding valida- tion curves. The experimental results are presented in Fig. [4] with the validation curves reflecting two key metries:
1) The proportion of predicted positions in the validation set with error less than 0.01 (left subplot Fig. 14).
2) The proportion of predicted rotations in the validation set with an error less than 0.025 (right subplot in Fig. 14).
The results in Fig- cleearly demonstrate that the model's performance improves as the number of expert demonstrations and the number of viewpoints increase. The key observations are as follows:
With only 20 expert demonstrations, the model exhibits low overall performance, particularly in predicting rotation angles.
Models trained with four viewpoints achieve significantly better performance, but this improvement comes at the cost of increased training time.
As the number of expert demonstrations grows, the marginal improvement in model per- formance diminishes. This could be attributed to the model's parameter size not scaling proportionally with the increase in data volume.
These results highlight the benefits of larger datasets for enhancing model performance. However, they also underscore the need for further optimization in model architecture and resource allocation
30

---

0 Published as a conference paper at ICLR 2025
Task
Variation Type
Variations
Avg. Keyposes
Language Template
push button shape
color
20
2.0
"Press the buttons in order of their size. from smallest largest"
button close
20
close the jar, push batton"
pour from cup to cup
color
20
6.0
"pour liquid from the cup the cup"
Table Description of Additional Novel Tasks.
Additional Novel Task
Voxposer
Act3D 3D Diffuser Actor GravMAD (Ours)
Push Buttons Shape (Difficult Task)
0
0
62.66
Button Close Jar (Combination of Skills)
0
0
0
Pour From Cup Cup (Completely New)
0
0
Table 10: Generalization Performance Comparison Additional Novel Tasks.
to effectively harness the potential of large-scale data. Without such improvements, the diminishing returns observed with increasing data may limit scalability in practical applications.
D.4 ADDITIONAL NOVEL TASKS
We evaluate the performance of baseline methods and GravMAD three additional novel tasks, with detailed descriptions provided Table JablePand Fig. 15 These tasks include a highly challenging (Push Buttons Shape), task that requires integrating skills learned during training (Button Close Jar), and a task involving entirely new objects compared to the training set (Pour From Cup to Cup).
The results presented in Table TO The "Push Buttons Shape" task evaluates the model's ability to handle long-horizon planning, language reasoning, and robustness to visual perturbations. Under these conditions, all baseline methods fail to complete the task, whereas GravMAD performs well, showcasing its potential for generalization. For the "Button Close Jar" task, the results indicate that GravMAD still struggles with long- long-horizon tasks requiring the integration multiple skills. In the entirely new task "Pour From Cup to Cup", GravMAD successfully identifies task-relevant objects but fails to complete the task due incorrect actions. This failure likely caused by a significant mismatch between the training data and the test environment.
D.5 ADDITIONAL. ABLATION STUDY
To investigate the impact of the cost map on model performance, we perform more detailed experi- ments the "w/o Cost map" ablation setting. In this ablation study, due the inherent limitations of the encoder, the GravMap containing only the gripper map cannot be effectively processed. For instance, when the sub-goal requires the robotic becomes zero structure. Such an m cannot be properly parsed by the DP3 Encoder, resulting in gradient vanishing during the training process.
perform "close everywhere" operation, my
Base Tosks
Novel Taskk
$7.28 54
14.23
5: 43,03
GravMAD (VLM)
50
GrawMap
Success Rato
w/o, Cost map (Gripper -1)
Grip. map
15,00
w/o. Contra, loss
W. Guided Rot.
PointNet
DP3 Encoder - PointNet++
Figure 16: Additional Ablation Studies. We represent the gripper closure in the gripper map under "w/o. Cost map" as -1 instead of 0, enabling the encoder to correctly process this data structure.
31

---

Published as a conference paper at ICLR 2025
To address this issue, modify the gripper map in the "w/o Cost map" setting by changing the closed state representation from 0 to -1, enabling the encoder to correctly process this data structure. The experimental results are shown in Fig. [6 The results show that removing the cost map causes a significant performance drop compared to the original model: a decrease of 11.97% on 12 base tasks and 21.04% ..04% 8 novel tasks. These findings clearly highlight the critical role of the cost map in ensuring the performance of the GravMAD model.
D.6 REAL WORLD EVALUATION
Real-world Task
Open Drawer
Toy Drawer
Mouse Pad
Stack Cup
Stack Block Same
GravMAD (%)
100
Real-world Task
Place Cup
Stack Block
Stack Cup Blocks
Wired Mouse on Pad
Colored Toy Drawer
GrayMAD (%)
10
40
40
100
70
Table 11: Real-robot Results. Success rates of GravMAD on 10 real-world tasks. These tasks include both manipulation and placement challenges. Above the table are the point clouds and GravMaps for Stack Cup Blocks and Stack Block, respectively.
We use Franka Emika robot to validate Grav- MAD's multi-task generalization ability across 10 real-world tasks. Each task involves varia- tions in placement, and some tasks include color variations. Compared the base tasks, the novel tasks introduce new objects and new instructions. The base tasks include:
Real D345i
Open Drawer (task description: open top drawer)
Place Cup (task description: put the yellow toy the top drawer)
Mouse on Pad (task description: put
the wireless mouse on pad)
Stack Cup (task description: stack
Fruila Emika Punda
colorl cup on top of color2 cup)
Stack Block Same (task description:
Figure 17: Real-Robot Setup with RealSense D435i and Franka Panda.
stack blocks with the same color)
Place Cup (task description: place one cup on the cup holder)
The novel tasks involve:
Stack Block (task description: stack color1 block top of color2 block)
Stack Cup Blocks (task description: identify the most common color in the block pile, and stack the other cups on the cup that matches that color)
Wired Mouse Pad (task description: put the wired mouse pad)
Colored Toy in Drawer (task description: put the Black and white toy in the top drawer)
We position a RealSense D435i camera in front of the robot to capture images, which are downsampled from the original resolution of 1280x720 to 256x256, shown in fig. 17 During training, collect
32

---

Published as a conference paper at ICLR 2025
20 demonstrations for each base task to train the model. During inference, similar to the simulation setup, GravMAD predicts the next keypose, and we use the BiRRT planner provided by Movelt! ROS to guide the robot to reach the predicted keypose. For evaluation, we run 10 episodes for each task and report the success rate.
The inference performance of GravMAD 6 base tasks and 4 novel tasks is shown in Table [1 These results demonstrate that GravMAD can effectively reason about 3D manipulation tasks in real-world robotic scenarios, leveraging associated visual information and generalizing to novel tasks. The video demonstrations are available at: https:://gravin git.ub.thh........ io
E LIMITATIONS AND POTENTIAL SOLUTIONS
Despite GravMAD demonstrating strong generalization capabilities across the 3 categories and 8 novel tasks showcased, still has certain limitations. The following section discusses some of the limitations not covered in the main text and their potential solutions:
Limitations of heuristic Sub-goal Keypose Discovery: The current method relies on predefined heuristic rules, which may struggle adapt tasks with more complex ambiguous sub-goal structures. Future research could explore more adaptive or learning- based strategies, such as incorporating diffusion models (Black al. 2024) or generative models (Shridhar et al., 2024) to generate sub-goals, to further enhance the robustness and flexibility of the method.
Dependence Detector accuracy and inference time: inference phase has a significant impact on the results, and its relatively long inference time remains a bottleneck. Future work could integrate observations from multiple viewpoints to provide a more comprehensive scene understanding and improve detection accuracy. Alternatively, more granular segmentation models could be leveraged provide richer labels for foundation models, thereby improving the quality of the context generated by the Detector.
The Detector's accuracy during the
Limited guidance for end-effector orientation: The current Grav framework does not effectively guide the robot's end-effector orientation, limiting its applicability to tasks requiring precise orientation control. A potential improvement involves combining rotation information from expert trajectories with object poses to generate few-shot prompts for off fff-the-sheff foundation models (Yin al., 2024). By leveraging such few-shot prompts, foundation models could produce more precise and effective rotation guidance.
Challenges in generalization: While GravMAD performs exceptionally well on tasks similar to those seen during training. its generalization ability is still limited for tasks with significant differences from the training set, such as entirely unseen tasks or challenging tasks requiring combination of multiple learned skills. Expanding GravMAD's D's capability to flexibly integrate multiple learned skills will be a key direction for future research. One feasible direction combine exploration- based learning with reinforcement learning (Hao) et al. 2024).
Dependence on GravMap for Sub-goal Representation: The GravMap framework relies on point cloud struetures for sub-goal representation, which, while effective, may add unnecessary complexity in scenarios where simpler representations, such as single point or relative coordinates, could suffice. The competitive performance of the "w/o GravMap" variant novel tasks suggests that alternative representations could simplify the model without compromising performance. Defining sub-goals as relative coordinates with respect to the gripper's current position, leveraging proprioceptive information, is promising direction. This approach could possibly introduce more data variation, enhance adaptability spatial changes, handle imprecise sub-goals, and naturally encode directional information. Future research could explore this direction further to achieve a balance between simplicity and performance, potentially enhancing the generalization capability of the model while reducing reliance on GravMap.
By addressing these limitations, we anticipate that GrayMAD will demonstrate stronger adaptability and practical value in more diverse tasks.
33


Reviews:
[{"review_id": "Q1mbvzMjqD", "reviewer": "Reviewer_2Lap", "summary": "The paper proposes a pipeline of Grounded Spatial Value Maps-guided Action Diffusion (*GravMAD*), which is a subgoal-driven, language-conditioned action diffusion framework combining VoxPoser (Huang et al., 2023) with subtask imitation learning based on 3D Diffuser Actor (Ke et al., 2024). The idea is essentially to break tasks into sub-goals based on language instructions, allowing auxiliary guidance to help robots conceptually grasp tasks more accurately than directly using pre-trained foundation models.\n\nThe division of subtasks is conducted via Sub-goal Keypose Discovery, which is defined to happen when end effectors make contact or disengage with the objects. The proposed method uses pre-computed keyposes and touch&gripper change detection to segment subtasks for training epochs. During inference, the proposed method uses the LLM-based Planner and Composer from VoxPoser (Huang et al., 2023).\n\nWith the segmented subtasks, the proposed method first uses VoxPoser (Huang et al., 2023) to generate 3D value maps *GravMaps* and then uses 3D Diffuser Actor (Ke et al., 2024) to conduct subtask imitation learning and generate action sequences with the condition of tokens from *GravMaps*.\n\nThe authors test *GravMAD* on 20 RLBench (James et al., 2020) simulated tasks, including 12 original tasks selected from RLBench and  8 modified new tasks. Empirical results show that the proposed framework outperforms both VoxPoser and 3D Diffuser Actor in most seen and novel tasks, which validates GravMAD's superiority in both generalization and preciseness.", "questions": "1. Can you provide more details on the selection strategy of the 12 tasks over 100 language-conditioned tasks from RLBench? Why do you choose 12 and why these 12? The currently designed novel tasks feature scenes and objects similar to those in the training tasks. Can you design more complex novel tasks?\n2. Can you evaluate the performance of the proposed method in a real-world experiment? Most setups should be similar to 3D Diffuser Actor and not that hard.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, easy to follow, and presents its ideas clearly. \n2. The approach of combining VoxPoser's planning and perception ability with precise subtask imitation learning from 3D Diffuser Actor is well-motivated and clever to enhance the model’s long horizon task capability to achieve both VoxPoser's generalization and 3D Diffuser Actor's preciseness.\n3. The reported scores are evaluated across 3 seeds, which are statically robust.", "weaknesses": "1. The proposed Sub-goal Keypose Discover method is kind of naive. The proposed method simply uses the change in the gripper’s open/close state and the change in touch force to segment a whole task, which could suit simple tasks like moving the chicken but might not work for tasks with longer horizons. This segmenting requires complex reasoning over the concrete task description and comprehensive understanding. During inference, foundation models like VoxPoser can be better at dividing subtasks due to the reasoning ability of LLM. However, for this proposed pipeline where different segmenting methods are used in training and testing, there could even be a domain shift for the subtasks' distribution itself due to unaligned segmenting.\n2. The comparison and evaluation are limited. The selection strategy of the 12 tasks over 100 language-conditioned tasks from RLBench is not mentioned. The 8 modified tasks are not justified for their novelty, which might cause a doubt on opportunistic choices. \n3. Since both baselines of VoxPoser and 3D Diffuser Actor have conducted real-world evaluations, a real-world comparison for *GravMAD* will be more convincing.\n\nThough conducted in an A+B paradigm on VoxPoser and 3D Diffuser Actor, this work's engineering contribution is solid and the idea of subtask imitation learning is novel. I will raise my score if more validations are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJjyJQ7KQo", "reviewer": "Reviewer_fAE6", "summary": "This paper introduces GravMAD, a new approach that decomposes 3D manipulation tasks into key sub-goals, generates 3D spatial GravMaps to capture spatial relationships from these sub-goals, and employs an action diffusion policy to learn actions through imitation learning. This design aims to combine the precision of imitation learning with the generalization capabilities of foundation models. Extensive experiments conducted on RLBench show that GravMAD achieves improvements in both training tasks and novel tasks.", "questions": "* The use of a foundation model, such as GPT-4o, during inference might be time-consuming and could potentially hinder real-time responsiveness in robotic manipulation. Could the authors provide the average inference time for GravMAD at each sub-task stage?\n* During training, why not generate GravMap in the same way as during inference? (One possible reason may be that the foundation model’s inference time is too lengthy?) The GravMap generated from demonstration ground-truths may differ in distribution from the GravMap produced by the foundation model, which could impact inference performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The authors conduct extensive experiments, including comparisons with different types of baselines and a detailed ablation study.\n* The authors provide diverse visualization in the paper and on the website, enhancing readers' understanding.\n* The paper is well-organized and easy to comprehend.", "weaknesses": "* The novelty of the proposed method is limited, as it resembles a combination of previous works, Voxposer and 3D Diffuser Actor. Specifically, the authors use a similar approach to Voxposer by decomposing tasks and generating 3D spatial heatmaps for each sub-task, which are then integrated into the 3D Diffuser Actor architecture. The primary innovation appears limited to the proposed GravMAD framework.\n* The comparison between GravMAD and 3D Diffuser Actor on training tasks indicates that, although the introduction of GravMap improves performance on most tasks, GravMAD's performance heavily relies on both the Detector and GravMap. If the Detector’s predictions lack precision, it can even degrade the method's performance, potentially compromising its robustness in scenarios where camera capture is unclear or ambiguous.\n* In the ablation study w/o. Cost Map, it is surprising that the method achieves a zero score. Could the authors provide further explanation on why the predicted actions result in non-unit quaternions in this scenario? (This seems to conflict with the statement in Section 3.1, which mentions, \"to avoid quaternion discontinuities, we use the 6D rotation representation.\") Additionally, if the predicted quaternion is not-unit, we can normalize and convert it into a unit quternion. \n* It is advisable to conduct real-world experiments, as many related works, such as Voxposer and 3D Diffuser Actor, also include real-world demonstrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zoDnMKT20w", "reviewer": "Reviewer_SKBE", "summary": "This manuscript investigates 3D manipulation tasks through robot learning. Traditional imitation learning excels with familiar tasks but struggles with new, unseen ones. Recent approaches use large foundation models to mitigate this but lack task-specific learning, often leading to failures in 3D environments. To overcome this, the author introduces a novel framework termed Grounded Space Value Map-guided Action Diffusion (GravMAD). Due to its promising results, I am tentatively inclined to accept the paper. However, I find certain aspects unclear, especially the evaluation of design choices and overall performance. I am open to revising my evaluation if the authors can address these concerns comprehensively and convincingly detail the contributions of their work.", "questions": "1. What are the specific criteria for “significant changes” in Sub-goal Keypose Discovery?\n2. From Table 2, it is evident that the algorithm exhibits significant differences in performance on similar tasks, such as placing a cup and placing wine. Can you explain the reasons for this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. GravMAD shows strong generalization capabilities, achieving at least 13.36% higher success rates than state-of-the-art baselines on 12 base tasks encountered during training and surpassing them by 28.63% on 8 novel tasks.\n  2. The manuscript is well-written with a clear structure, logical flow, detailed technical descriptions, and intuitive chart presentations.", "weaknesses": "- Insufficient argumentation\n  1. The manuscript mentions in section 3.2 that different sub-goal keypose discovery rules were designed for different types of manipulation tasks, which seems to lack a reasonable explanation and generalizability.\n  2. The manuscript does not adequately justify the exclusion of GravMap’s guidance in rotation prediction, stating that “rotation actions are difficult to be guided explicitly through value maps.”\n- Inadequate Evaluation: \n  1.  Although it is claimed that this method is generalized, all experiments were conducted in the RLBench environment, and there is a lack of validation on other simulation environments or real robots.\n  2. The main evaluation metric used is success rate, overlooking other important dimensions such as completion time, smoothness of action, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KUa4VTbtRc", "reviewer": "Reviewer_vbZp", "summary": "This paper proposes a novel method for learning from demonstrations while incorporating knowledge from foundation models to tackle robotic manipulation. It learns to predict robot end-effector actions using a diffusion process and enhances the representation using 3D cost maps introduced in previous work. These cost maps are computed using heuristically extracted sub-goals during training, while foundation models are used instead at inference. In this way, the proposed method is able to combine strong generalisation capabilities from foundation models and learn precise robot actions from available demonstrations. The experiments show strong performance when compared to the considered baselines, while ablation studies provide valuable insights into different components of the method.", "questions": "- Some are raised in the Weaknesses section.\n- What is the difference between GravMaps and the cost maps described in VoxPoser? Is it the downsampling part?\n- Why are rotation maps from VoxPoser not used?\n- Do you have any intuition on how well your proposed method would scale with more data? Would Imitation Learning catch up at some point (with a lot of training data)?\n- Could you expand a bit more about the limitations of your proposed method?\n- GravMaps are just an overparameterization of sub-goals at a high level. The fact that the ablation variant (w/o. GravMap) performs so poorly is not very intuitive. Could you provide more details about this ablation variant and maybe you have some intuition about its performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The motivation of the paper is strong and well-explained.\n- Figures (both in the main paper and the appendix) are clear and very helpful in understanding the proposed method.\n- The technical challenge of integrating many components together in a unified framework is well-executed and appreciated.\n- Ablation studies are well-executed and provide valuable insights into the method.\n- Overall, utilising foundation models together with some demonstrations to achieve both, task generalisation and high precision seems like a great direction.", "weaknesses": "- Novelty and contribution claims are not very clear:\n    - 1) Discovering sub-goals and predicting them to guide robot actions has been done before [1,2,3,4]. Is the novelty here mainly the 3D part combined with foundation models? Maybe the difference could be emphasised more.\n    - 2) From the paper, it seems that GravMaps are Cost and Gripper openness maps used in VoxPoser, just renamed. Without properly explaining the difference, it shouldn’t be called a novel contribution.\n    - 4) Claiming to design two types of tasks (line 124) and then using base tasks from RLBench for one type can come off as being disingenuous about the contribution claims. Rephrasing the 4th contribution to focus more on the evaluation rather than task design would be beneficial.\n- Some parts of the method and design choices need to be better explained:\n    - How GravMap tokens are used in the network architecture is not described in the main text of the paper.\n    - Explaining in more detail why sub-goals are extracted differently during training and inference would greatly benefit the clarity of the paper. It becomes obvious while reading the paper, but initially, it can cause confusion.\n    - It is not clear why rotation maps from VoxPoser are not used. Justifying it would make the overall approach clearer.\n    - GravMap Synthesis during Inference is not explained clearly. Lines 305-309 are very dense with information, but it’s very difficult to understand how the sub-goals are actually extracted. If it is done just using existing methods, then contribution claims should be adjusted accordingly. \n- It is not clear if the proposed sub-goal keypose discovery is actually needed. There are no experiments using keyposes, without filtering them. If there are other reasons to filter keyposes, they should be explained.\n- Overall, sub-goal keypose discovery seems very tailored towards specific types of tasks. Limitations of it (and how they could be lifted) are not discussed.\n- What are 3 random seeds used for? Are the models re-trained using different seeds or just to change object poses and the stochastic diffusion process? If it’s the latter, then it is the same as having 3x episodes and the reported statistics don’t hold a lot of value.\n- GravMAD (Manual) seems like an unfair comparison to the baselines. Maybe providing the same ground truth object pose information to other methods would be more valuable for such comparison.\n- Given that the proposed method uses sub-goals to guide the diffusion process, selected baselines are not the most appropriate. Better baselines would be ChainedDiffuser [3] and/or Hierarchical Diffusion Policy [4].\n- There are some contradictory and not complete claims in the paper:\n    - Line 144 - Described methods already leverage demonstration data.\n    - Line 205 - Quaternions, although they have the double cover property, are continuous unless additional restrictions are imposed (e.g. by requiring that the scalar part w≥0).\n    - Line 206 stated that 6D rotation representation is used, but line 510 states that the model can predict non-unit quaternions. How is this possible?\n    - What are “3D tasks”? Tasks that involve 3D observations or 6D manipulation? It is not a very clear term.\n- Limitations of the proposed method are only superficially discussed.\n\n[1] Black, Kevin, et al. \"Zero-shot robotic manipulation with pretrained image-editing diffusion models.\" arXiv preprint arXiv:2310.10639 (2023).\n\n[2] Kang, Xuhui, Wenqian Ye, and Yen-Ling Kuo. \"Imagined subgoals for hierarchical goal-conditioned policies.\" CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP). 2023.\n\n[3] Xian, Zhou, et al. \"Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation.\" 7th Annual Conference on Robot Learning. 2023.\n\n[4] Ma, Xiao, et al. \"Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)