PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
Authors: Xiaogang Jia
Qian Wang
Anrui Wang
Han A. Wang
Balázs Gyenes
Emiliyan Gospodinov
Xinkai Jiang
Ge Li
Hongyi Zhou
Weiran Liao
Xi Huang
Maximilian Beck
Moritz Reuss
Rudolf Lioutikov
Gerhard Neumann
Venue: NeurIPS 2025 poster ()
Keywords: Imitation Learning
Point Cloud
Robot Manipulation
Primary area: reinforcement_learning
TL;DR: We present PointMapPolicy, a multi-modal imitation learning method that conditions diffusion policies on point maps

Abstract: Robotic manipulation systems benefit from complementary sensing modalities, where each provides unique environmental information.
Point clouds capture detailed geometric structure, while RGB images provide rich semantic context. Current point cloud methods struggle to capture fine-grained detail, especially for complex tasks, which RGB methods lack geometric awareness, which hinders their precision and generalization. We introduce PointMapPolicy, a novel approach that conditions diffusion policies on structured grids of points without downsampling. The resulting data type makes it easier to extract shape and spatial relationships from observations, and can be transformed between reference frames. Yet due to their structure in a regular grid, we enable the use of established computer vision techniques directly to 3D data. Using xLSTM as a backbone, our model efficiently fuses the point maps with RGB data for enhanced multi-modal perception.
Through extensive experiments on the RoboCasa and CALVIN benchmarks and real robot evaluations, we demonstrate that our method achieves state-of-the-art performance across diverse manipulation tasks. The overview and demos are available on our project page: https://point-map.github.io/Point-Map/

PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
Xiaogang Jia Qian Wang'
Anrui Wang'
Han Wang
Balázs Gyenes'
Emiliyan Gospodinov
Xinkai Jiang'
Ge Li Hongyi Zhou²
Weiran Liao'
Xi Huang'
Maximilian Beck3
Moritz Reuss!
Rudolf Lioutikov!
Gerhard Neumann
Karlsruhe Institute of Technology
Reality Labs, Meta
Johannes Kepler University Linz
Abstract
Robotic manipulation systems benefit from complementary sensing modalities, where each provides unique envirommental information. Point clouds capture detailed geometric structure, while RGB images provide rich semantic context. Current point cloud methods struggle to capture fine- aineggriiedd detail, especially for complex tasks, which RGB methods lack geometric awareness, which hinders their precision and generalization. We introduce PointMapPolicy. novel approach that conditions diffusion policies on structured grids of points without downsampling. The resulting data type makes it easier to extract shape and spatial relationships from observations, and can be transformed between reference frames. Yet due their structure in a regular grid, we enable the use established computer vision techniques directly to 3D data. Using xLSTM backbone, our model efficiently fuses the point maps with RGB data for enhanced multi-modal perception. Through extensive experiments evaluations, we demonstrate that our method achieves state-of ofte-of-thheart tt-hherrt performance across diverse manipulation tasks. The overview and demos are available on project page.
the RoboCasa, CALVIN benchmarks and real robot
1 Introduction
The advent of diffusion- based Imitation Learning (IL) has allowed robots to carry out complex, long-horizon tasks from raw image observations [1, 2]. RGB images a common observation modality for diffusion policies due to their ubiquitousness and rich semantic information. However, policies conditioned on only RGB images lack 3D geometric information about the scene. This 3D information is crucial for learning generalizable policies that can act precisely in complex 3D scenes, especially when using multiple camera views [3-6]. An alternative modality is point clouds, unstructured sets of 3D points that preserve geometric shape, distances, and spatial relationships. In addition, points captured from multiple camera views can be transformed into a common reference frame and concatenated, yielding natural and powerful way to fuse multiple cameras. Although numerous works use point clouds as an input modality [7-9], their irregular structure limits the network architectures that can be used with them. In contrast, RGB images are regular grid and can processed using convolutional operators, but are susceptible to changes in perspective and lighting.
Current 3D processing approaches face fundamental limitations that create critical gap between 3D geometric information and existing 2D vision architectures. Downsampling- methods [7],
"Correspondence jia2661 @ gmail.com This work is not related to Han A. Wang's 's position at Meta.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Image Encoder
FPS
Lifting
Point
Visual Encoder
Visual Encoder
Net
Max Pool
RGB Token
Point Map Token
Compact Token
3D Scene Tokens
(a) Down Sampling
(b) 3D Lifting
(c) Point Map
wwwnsampling-based based methods use Furthest Point Sampling (FPS) to reduce dense point clouds sparse representations, which PointNet then processes into compact tokens. Some variants employ FPS+KNN to generate structured point patches. (b) Feature-liftue lifting approaches first extract 2D features from images, then project these features into 3D space, creating semantically rich 3D points. (c) Our point map method structures the point cloud as grid with the same dimensions as corresponding images, enabling direct application of efficient visual encoders to each modality independently.
Figure 1: Different approaches for point cloud processing: (a) Downsampling
as shown in Figure (a), suffer from an inherent information-fidelity tradeoff:: they must dramat- ically reduce point density through techniques like Farthest Point Sampling (FPS)[10] to remain computationally tractable, inevitably discarding fine-grained geometric details essential for precise manipulation tasks. Feature-lifting approaches [11], as shown in Figure 1(b). face equally prob- lematic limitations as they aggregate 2D features through depth averaging and 3D transformations, introducing information loss while struggling maintain spatial structure during the lifting process. In this paper, we take inspiration from recent advances from the computer vision community in stereo reconstruction [12, 13] to propose using point maps, as shown Figure I(C). Point maps encode information in regular, 2D grid of Cartesian coordinates. This results in a structured data type that can be used with standard architectures such as ResNet [1 4], ViT [15], or ConvNeXt [16]. This obvi- ates the need for steps like K-Nearest Neighors (KNN) and Farthest Point Sampling (FPS) [10], which are computationally expensive operations common to point cloud methods [17. 18, 6]. At the same time, because they are geometrically grounded, point maps from multiple views can be transformed into the same reference frame, increasing robustness to perturbations in camera perspective.
We integrate point maps into a standard diffusion- based imitation learning framework based on EDM [19] demonstrate their effectiveness as drop-in replacement for RGB images or point clouds. We validate the effectiveness of point map observations two challenging benchmarks: RoboCasa [20] and CALVIN [21]. These benchmarks feature language-conditioned tasks and diverse scenes, and require spatial reasoning and long-term planning. Across both benchmarks, point map- based policies outperform basclines using RGB, depth maps, or point clouds, demonstrating superior learning efficiency and generalization. Our method is computationally efficient in training and inference, sometimes by an order magnitude.
Our contributions are the following: 1) we propose PointMapPolicc (PMP), method for diffusion-baan based imitation learning on point maps, powerful observation modality that has never been used in diffusion imitation learning: 2) we achieve state of the art results among policies trained from scratch on CALVIN benchmark [21], and outperform other observation modalities on RoboCasa [20]: 3) we present systematic ablations of point cloud processing methods, vision backbones (e.g. ResNet [14], ViT [15), ConvNeXt [16]), and paradigms for fusing color and geometry information.
Contributions:

---

2
Related Work
2D Visual Representations for Imitation Learning. Recent imitation learning approaches [1, 22- 25] rely predominantly on 2D visual representations such as RGB images or videos. Such representa- tions widely utilized due their capacity capture rich textural and semantic information, as well as their accessibility through low-cost cameras. However, 2D image modalities have inherent limitations: they contain 3D information only implicitly, are vulnerable viewpoint and lighting changes and occlusions, and typically underperform in tasks requiring detailed spatial reasoning and geometric alignment [3-6].
3D Visual Representations for Imitation Learning. To overcome these limitations, a growing amount of research incorporates explictly 3D representations such as depth maps, point clouds, or voxels. Voxel-based methods like C2F-ARM [26] and Perceiver-Actor [27] voxelize point clouds and 3D-convolutional network for action prediction, but require high voxel resolution for precision tasks, resulting in high memory consumption and slow training. DP3 [7] encodes sparse point clouds using FPS, followed by lightweight MLP to produce compact embedding vector of the observation. While efficient, this approach discards local geometric structure that can be critical for fine-grained tasks. In contrast, 3D Diffuser Actor [11] computes tokens by lifting 2D image features into 3D space by using averaged depth information and camera parameters, and applies FPS after the first cross-attention layer. FPV-Net [28] fuses RGB and point cloud modalities by injecting global and local image features into a point cloud encoder using adaptive normalization layers, but is still limited by the disadvantages of both modalities.
Multi- Representation. Complementary work, such as Robot Vision Transformer (RVT) [29], avoids working directly with raw point clouds by proposing novel multi-view representation. This approach re-renders the point cloud from a set of orthographic virtual cameras, deriving a 7-channel point map (RGBD XYZ) from each view. RVT-2 [30] improves this approach for high- precision tasks by introducing a multi-stage inference pipeline: it first identifies a region-of-interest truncates the observation this of interest, and then runs policy inference. However, neither of these methods use action diffusion, instead relying on key-frame based manipulation with a motion planner [27]. Furthermore, geometric and color information are fused naively at the channel level, whereas we investigate more sophisticated techniques for fusion.
Diffusion-Policy Backbones. Due the non-Markovian nature of human demonstrations, where successful eecision-making often depends on histories of past observations and actions, early work used -aN--ased architectures [31], but struggled with vanishing gradients and limited scalability. This led to the adoption of Transformer- based architectures, which offer global attention and paral- lelism, enabling superior performance in tasks requiring long-horizon reasoning [32-34], becoming the standard backbone for many methods [7, 11, 28, 22]. However, Transformers are computationally intensive and scale quadratically with the sequence length, which limits the number of tokens that can be used to encode the observation.
To mitigate these challenges, recent works [35, 36] explore State Space Models (SSMs) like Mamba [37], achieving linear-time complexity and improved sample efficiency, particularly in low-data regimes. Additionally, recent recurrent architectures such xLSTM [38] provide an appealing balance, maintaining the temporal modeling strengths of traditional RNNs while intro- ducing architectural innovations that improve gradient flow and expressiveness. Despite being less expressive than self-attention, XLSTM significantly reduces compute and memory costs, making it well-suited for real-time resource-constraaied applications. X-IL. [36] systematically compares different architectural parts, and finds that XL.STM performs competitively with Transformers in multi-modal imitation learning. Building on these insights. PointMapPolicy adopts xLSTM as its diffusion backbone, balancing temporal modeling capability with efficient training and inference.
3 Method
3.1 Problem Formulation
Imitation Learning aims to learn policy from expert demonstrations, Given a dataset of expert trajec- tories D,- (Tx Ji=1 { where each trajectory = ((s), a1), (S2, a2),... (Sx,ax)). The objective is to learn a policy (a ss) that maps observations s to a sequence actions ā = (az, ag+1;-... a H1), Predicting sequences of actions, i.c. action chunking, allows for more temporally consistent action
3

---

Prompt: eg..
"Push the Button"
Image
Point Map
Image
Map
Noise Token
Noised Action Tokens
Tokens
CLIP
FILM
Visual
Visual Encoder
Visual Encoder
Visual Encoder
nusker
Encoder
E
PE
PE
PE
PE
PE
PE
PE
RUSkoom
X-Block
Denolsed Actions
--
PE
Positionall Embedding
ResNet, ConwNett, DWVIT
1-+11
Figure 2: Overview of PMP. PMP integrates multiple modalities: a pretrained CLIP model, images processed by pretrained visual encoders, and point maps processed by visual encoders trained from seratch. Leveraging x-L.TTM its backhone architecture, PMP efficiently fuses these multi-modal inputs to generate denoised actions.
language instructions encoded by
prediction [39]. Each observation contains multi-view RGB-D images and language instruction for the current trajectory.
3.2 Score-based Diffusion Policy
Our approach employs the EDM framework for continuous- tietetuuus---eeeeeeo action diffusion [19, 33] to generate actions. Diffusion models are generative models that learn to generate new samples through learning to reverse a Gaussian Perturbation process. In PointMapPolicyy we apply score-based diffusion model formulate the policy representation To (ā]s). This perturbation and its inverse process can be expressed through Stochastic Differential Equation (SDE);
dā (3.o, oz) V. log (ālls)dt 22,,0,
where 3r determines the noise injection rate, dur represents infinitesimal Gaussian noise, and pr (āls) denotes the score function of the diffusion process. guides samples away from high-density regions during the forward process. To learn this score, we train a neural network Da via score matching [40]:
Lsm E,,a, (aoa c, -a
where @a + o, represents our trainable neural architecture.
After training, we can generate new action sequences beginning with Gaussian noise by iteratively denoising the action sequence with a numerical Ordinary Differential Equation (ODE) solver. Our approach utilizes the DDIM-solver, specialized numerical ODE-solver for diffusion models [41] that enables efficient action denoising in minimal number of steps. Across all experiments, our method uses 4 denoising steps.
3.3 Observation Tokenization
We are given an observation Sş in step k as well as a textual language instruction 2tang- The language instruction is first tokenized using a pretrained CLIP text encoder [42] generate language embeddings. For RGB inputs, we use Film-ResNe [43] with pretrained ImageNet weights to generate visual embeddings from the observation Sg.
We define that Point Map X E Hx is a dense 2D field of 3D points that establishes one-to-one mapping between image pixels and 3D scene points. For an RGB image of resolution И W, the corresponding Point Map X satisfies 4 + X1,; for all pixel coordinates (i,j) e (1...H) H) (1...W), W), where each pixel intensity It, corresponds 3D point X,,, E IR3 in world coordinates.
We convert each depth map D HW to structured point map representation:
M, (DD,,))))
M, Xx
(3)

---

where Kint are the camera intrinsic parameters obtained through calibration and & is depth unpro- jection operation. The result a multi-channe point map with the same spatial dimensions as the input depth map, where the channel dimension C' is typically 3. Points beyond maximum depth and below minimum depth are masked out. Point maps from all cameras are transformed into a common world reference frame using the extrinsic parameters of the camera Kex--
3.4 PointMapPolicy
PointMapPolicy uses EDM-based action diffusion for decision making and conditions on the multi- modal observation tokens generated from RGB and point map modalities. We present and explore multiple paradigms for fusing RGB and geometric data at various stages of processing. We also describe PMP-xyz, variant with tokens from only the point map modality, for tasks that do not condition on color information.
Fusion of image and point maps. A key advantage of point maps their ability to provide both geo- metric and visual embeddings for each camera view, enabling straightforward multimodal fusion. We in- vestigate both early and late fusion approaches. For early fusion PMP-6ch, we concatenate point maps with RGB values, creating six-channel inputs (XYZ + RGB). For late fusion, we first tokenize image and point map modalities from each view with separate encoders. Then we explore three methods to fuse encoded tokens, as illustrated in Figure 3: I) Add, element-wiss addition of tokens from both modalities, resulting in one token per view; 2) Cat, concatenation of tokens from all modalities and views; and 3) Attn, using a four-layer transformer module process tokens using pross-attentic shown in our ablation studies, we find Cat to slightly outperform other late fusion methods, so we choose this for PMP. An overview of PMP with Cat fusion is illustrated in Figure 2.
Transformer Block
Concatenate
CLS
Figure 3: Fusion methods. From left to right:
Add, Cut, and Attn.
to generate fused class tokens for each view. As
Backbones. Given the multi-modal tokens from Section 3.3, a learnable positional embedding is added to each token. PMP uses decoder-only backbone from X-IL [36] with x-LSTM as the core computational unit. All tokens are concatenated as inputs to the X-Block, which is the diffusion score network Dg. While Transformers dominate most imitation learning policies, X-IL demonstrated that the recent recurrent architecture xLSTM excels in robot learning tasks. The core computational element within X-Block is the m-LSTM layer, which serves analogous function to self-attention in Transformer architectures. The denoised action tokens produced by the X-Block are then used to guide the robot's 's behavior, resulting in policy that effectively leverages both the geometric precision of point maps and the rich semantic understanding from RGB images.
Simulation Experiments
We conduct experiments on two simulation benchmarks RoboCasa [20] and CALVIN [21]. We aim answer the following questions: Q1) How does PointMapPolicy compare to state-of-the tftte- 2D and 3D imitation learning policies? Q2) How do the fusion methods perform compared to other modalities?
Q3) How does point map representation compare other point cloud processing methods? Q4) Can current vision encoders effectively extract the geometric and semantic information from point maps required for robust deeciiiionmakinng
The RoboCasa benchmark [20] is a large-scale simulation framework designed to evaluate IL agents across wide range of household manipulation tasks. Built physically realistic environment with rich visual rendering, RoboCasa supports task diversity, long-horizon behaviors, and tine-grained physical interactions, making it compelling testbed for assessing both generalization and behavior diversity in policy learning. We use the RoboCasa benchmark to assess whether our proposed point map representation can enable effective learning and generalization across manipulation tasks of increasing complexity, object count, and behavioral variation.
RoboCasa:
The CALVIN benchmark [21] provides a large-scale framework for evaluating language- conditioned II. policies in visually rich, long-horizon manipulation tasks. The benchmark contains 34 distinct manipulation tasks such as button-presssigg drawer-opening, object-picking, and pushing.
CALVIN:
5

---

CALVIN Benchmark
RoboCasa
Real World
Train on diverse hours of play dats from 4 ansfronments.
with diverse scenes and objects
maripuattion and long honizon task completion
Env
Env
Env
Env
"Dweoplng"
"Pourigg
"Poldiog"
Drawsr
Stove
Butoo"
Complete 1800 eets of Tout Instructions Rew
Tleee
Mug"
Mup"
Mlug"
"Amangiigg
CCup Stacking
Figure 4: Overview of Simulation and Real World Experiments used to test PointMapPolicy. From left to right: CALVIN Benchmark [21], RoboCasa [20], and our Real World Setup.
Category
Task
BC
GR00T-N1
DP3
303
RGB
Depth
PMP-6ch
PMP-xyz
PMP
2.0
0.0
4.0z..e
0.0
10.0a4.
3.3ec.
9.Scsa
13.3es.
10.7ass
Pick Place
PaPCounter/oSink
2.0
1.0
0.7aca
0.0
5.3.1.
4.7
8.7+0,9
6.7
6.7
PaPMicrowave ToCounter
2.0
0.0
4.0a21
0.0
10.7a1
8.0a2.
12,0a:4
16.0s1.
16.0
PoPSinkToCConnts
8.0
1.3609
14.766.
3.3.ca
9.3
8.0g14
16.7.50
Open/Cloos Drawers
OpenDrawer
42.0
8t.7en.
58.7aau
00.0a.
60.0
56.0a82
Clowerrweer
80.0
96.1
60.0a14
84.0a73
92.0aoc
15.3
91.335
Twisting Knobs
TumOnStove
32.0
25.5
26.784:
18.0
18.7
22.7
S5.3aza
43.3
41.355
TamorfStooe
4.0
15.7
7.36
8.0
13.3
14.0a34
16.7
20.01a.
18.0an0
TumonnSikkFuue
35,0
59.8
42.0aa1
26.0
65.0
78.7+28
64.7.41
76.7
08.7a81
Turning Levers
Fuuuet
50.0
67.7
42.0a4.
44.0
63.3
76.0a5.
73.3
82.0g1.
06.7am4
SumSinkSpout
54.0
18.7
50.014
76.0144
00.3
76.011.0
48.7674
ColiochhessButton
14.7aaa
70.7s157
80.0a4
76.7ea.
92.0es.a
Pressing Buttons
Jumonniccrwwve
62.0
73.5
39.3a74
34.0
48.0a44
40.0aac
64.7a1a
49.3as
64.7434
TumortMicrowaeee
70.0
57.8
62.7657
30.0
60.3g9.c
08.0a7
10.3
70.0.44
84.016.5
Insertion
CoffeeServeMug
22.0
31.3
21.3aa3
0.0
60.0a2
57.3a52
68.7e7.
69.3+s.
49 49.3e54
CoffesSeupMing
0.0
2.0
4.0g28
2.0
16.0s2a
15.3ess
10.7aa.
16.7
26.7+s.4
Average Success Rale
32.25
36.28
27.04
18.50
40.16
44.00
43.12
49.12
47.22
Table 1: Success rate (%) for each task in RoboCasa [20]. The models were trained for 50 epochs with 50 human demonstrations per task and evaluated with 50 episodes for each task. The bold numbers highlight the best achieved success rate for that task among all the models.
Each rollout consists of a sequence of 5 language instructions, and the agent must complete one task before proceeding to the next. Policies are evaluated 1,000 such instruction chains per seed, and success is measured by the average number of correctly completed tasks in each sequence.
Experimental Setup: For RoboCasa, each model was trained for 50 epochs using three random seeds, with performance measured at the 30th, 40th, and 50th checkpoints, selecting the best result. To ensure fair comparison, all models across different modalities use identical backbone parameters. For the CALVIN benchmark, models were trained for 25 epochs, with the best success rate reported from the 10th, 15th, 20th, and 25th checkpoints.
Baselines: For RoboCasa, we benchmark against Behavioral Cloning (BC) [20], GROOT-NI [44], 3D Diffusson Policy (DP3) [7], and 3D Diffuser Actor (3DA) [11]. Note that GROOT-NI results use 100 demonstrations, while all other methods use 50 human demonstrations. To systematically evaluate the effectiveness of our representation, we further compare other against image-based baselines using only RGB data (RGB), and only depth data (Depth). We then compare PMP against multiple variants introduced in Section 3.4: PMP-6ch directly uses xyz coordinates as inputs. All five methods share the same architectures and parameters for fair comparison. Details be found in Appendix 6.
6-channel point maps inputs, and PMP-xyz only
For CALVIN, we primarily compare our approach against models without robot-specific pretraining, though we include all results for reference. DP3, 3DA, and CLOVER [45] selected representa-

---

Train-TTsst
Method
Action Type
No. Instructions a Row (1000 chains)
Avg. Len.
RohoFlamingo (48]
Cont.
82.4%
61.9%
46.6%
33.1%
23.5%
2.47
SuSIE [49]
Ditlusion
87.0%
69.0%
49.0%
38. 38.0%
26.0%
2.69
GR- [50]
Cont.
85.4%
71.2%
59.6%
49.7%
40. 10.1%
3.06
Lp
Discrete
91.3%
77.8%
62.0%
52.1%
43.5%
3.27
RohoDual
Diffusion
94.4%
82.7%
72.1% 1%
62.4%
54.4%
3.66
Seer [47]
79.9%
64.3%
MoDE [24]
Dittusion
96.2%
88.9%
81.1%
71.8%
63.35 59.5%%
Seer-Large [47]
Cont.
96.3%
91.69
86.1%
80.3%
74.0%
4.28
ABC->D
DP1 [7]
Diffusion
28.7%
2.7%
0.0%%
0.0%
0.0%
0.31
MDT [46]
Diflusion
63.1%
42.9%
24.7%
15.1%
9.1%
1.55
3DA
Diffusion
92.2%
78.7%
63,9%
51 .21%
11.
3.27
MoDE (scratch) [24]
79.2%
67.3%
CLOVER [45]
Diffusion
96.0%
83.5%
70.8%
57
Seer (seratch) [47]
Cont.
93.0%%
82.4%
72.3%
62.6%%
53.3%
3.64
Soor-Large (scratch) [47]
Cont.
92.7%
84.6%
76.1%
68 9%.
60.3%
3.83
RGB
Diffusion
89.9%
75.4%
60.8%
49 19.8%
39.1%
3.15
PMP-xyz (ours)
Diffusion
73.0%
51.9%
37.0%
24.5%
16. 1%:
2.03
PMP (ours)
Diflusion
96.1%
88.6%
80.5%
72.3%
63.6%
4.01
Table 2: Evaluation results on the CALVIN benchmark under ABC-D. All results report the average rollout length averaged over 1000 instruction chains.
tive policies using RGB-D inputs. MDT [46], MoDE [24], and Seer [47] are selected as RGB-based policies. Further details of these baselines be found in Appendix A.2.
4.1 Main Results
RoboCasa. We present the main results in Table 1. PMP-xyz demonstrates significant advantages over prior 3D baselines DP3 and 3DA, achieving an average success rate of 49. 12% 10%11eerrryy 20% higher. It also outperforms 2D baselines BC and GROOT-N1 by approximately 13%. The ahove results address Q1. Our cross-modality evaluation using consistent architectures reveals that incorporating 3D information consistently improves performance in RoboCasa. Specifically, PMP-xyz shows 6% improvement over the Depth-only model, highlighting the value of structured point maps. While PMP (47.22%) outperforms PMP-6ch, demonstrating the benefits of late fusion, it still falls 2% short of PMP-xyz. This pattern suggests that most RoboCasa tasks favor geometric information, likely due the diversity of objects and scenes.
CALVIN. On the CALVIN benchmark, PMP achieves a score of 4.01, outperforming all other models trained from scratch and many models that leverage pretrained data, as shown in Table 2. Our method even outperforms Seer-Large (scratch) which scores 3.83 despite using 24 Transformer layers compared to smaller model using only 10 x-L.TMM blocks. This answers Q1 in the affirmative.
PMP-xyz performs poorly with an average rollout length of only 2.03, while the RGB-only model achieves respectable score of 3.15. This performance disparity stems from CALVIN's heavy reliance on color information for task execution, with many instructions explicitly referencing colors (e.g., "red block", "pink block"). This finding highlights a key limitation of purely geometric representations in color-dependent scenarios.
These contrasting results between RoboCasa and CALVIN benchmarks underscore the complemen- tary nature of geometric and visual information. While PMP-xyz excels geometry- -heay tasks (RoboCasa), it struggles with color-dependent tasks (CALVIN). This demonstrates that multimodal fusion approaches like PMP provide the most robust and versatile performance across diverse task domains by adaptively leveraging the most relevant modality for each scenario, addressing Q2.
4.2 Point Cloud Encoding
While our main results demonstrate the effectiveness PMP compared to other methods, these comparisons involve different policy backbone architectures. To isolate the contribution of our point cloud encoding approach, we conduct a controlled ablation study where we fix the policy backbone (X- Block) and systematically vary only the point cloud encoder.
7

---

PolutNet-xye
PointNes color
0.8
PotutPutch
Surees Iate:
PMP--ya (ours)
0.6
0.4
0.2
PaPCoaater TooloaeeeTolllrrre
PaPSieroean Tat Comnee
PaPSial Conntrr
Openiuar
ChaeDrwe
Farat Mftteee
Thrn ThrnCesia Fatoor
Thackttiad Fascos
TornBisk
Caffee Pifferrondflll PronrPaftttt
ThmmChblle rowerr
Thrnt Miwieroeeeeee
Ceffer Resperre
Amrage
Pa PoPCounta
Figure 5: Ablation study comparing point cloud encoders with fixed X-Block policy backbone. Our PMP-xyz method substantially outperforms baseline encoders across all manipulation tasks, demonstrating that our improvements arise from the point cloud encoding approach.
ReRNNet00 Gorroo
0.5
0.8
5000080 Date
S0t000s tite
0.6
0.6
0.4
0.4
0.2
PeP
Drawer
Koob
Levee
Buston
Insertion
Avg.
PnP
Drawer
Kzob
Lever
Dutton
Insertion
Avg.
Figure 6: Left: Performance comparison of various fusion methods between point maps and images. Right Performance comparison of different visual encoders for point map processing.
We conducted controlled experiments RoboCasa using identical xLSTM backbones with different point cloud processing encoders: 1) PointNet-xyz: Following DP3 [7], we gather point clouds from camera views and use Furthest Point Sampling (FPS) downsample 1024 points, then apply MLP with maxpooling to create a compact 3D token. 2) PointNet-color: Same process as PointNet-xyz but using colored points with XYZRGB information. 3) PointPatch: We use FPS to sample 256 center points, apply k-Nearest Neighbors to create 256 point patches with 32 points each, tokenize each patch using MLP with maxpooling, then process the resulting tokens with a transformer generate compact 3D representations. 4) 3D-Lifting: We extract CLIP features (frozen) from each camera view and lift the 2D features into 3D space, then use transformer to process the lifted tokens. The 3D tokens are then passed to the diffusion policy with an identical X- X-Block backbone.
Figure 5 presents the success rates 16 RoboCasa tasks. Our PMP-xyz achieves average success rate of 49.12%, substantially outperforming all baselines. The consistent improvements demonstrate that the point maps approach effectively captures the spatial understanding necessary for robotic manipulation, independent of the downstream policy architecture, addressing Q3.
4.3
Ablation Study
We additionally conduct three ablations across the categories of RoboCasa:
Fusion of Images and Point Maps. One key advantage of point maps over traditional point cloud processing methods is their structural similarity to RGB images from corresponding views. This alignment enables direct fusion of visual representations with point cloud data on a per-view basis. We evaluate three fusion strategies-Aiee comparative performance these fusion strategies presented in Figure 6. Although the performance differences are modest, Cat consistently emerges the most effective fusion approach.
Cat, and Aunhwhich are described in Figure 3. The

---

Vision Encoders for Point Maps. To assess how well existing visual architectures process Point Map representations, we conduct comparative analysis of three prominent visual encoders: FiLM- ResNer50, ConvNeXt-v2, and DaVIT. The results in Figure 6 demonstrate that while all encoders can effectively process point maps, ConvNeXtv2 consistently outperforms the others across all RoboCasa tasks, addressing Q4
Understanding Model Attention Patterns. To uncover where the model attends during action prediction, we apply Grad-CAM++ [52] to highlight the regions most influential for action decisions across different modalities. For detailed visualizations, see Appendix D.
4.4 Computation Resources and Inference Time
For the CALVIN experiments, PMP employs Film-ResNet50 as encoders for both images and point maps, with x-Blocks backbones (512 latent dimensions), totaling 147M trainable parameters. Training utilizes Nvidia RTX 6000 Ada GPUs with 128 samples per GPU (512 total batch size). Each epoch completes in approximately 13 minutes, allowing full training (25 epochs) in under hours, excluding evaluation time. More details can be found in Appendix E.
Regarding computational efficiency, conducted inference latency benchmarks for our models using ConvNeXt-v2 encoders on a single Nnidia RTX 5080 GPU (batch size 1). Across 1000 prediction eycles, PMP-xyz demonstrates remarkable efficiency with an average inference time of 2.9 ms, while PMP requires only 3.9 ms, maintaining real-time performance.
5 Real-World Experiments
We evaluate PMP on six challenging real-world robot manipulation tasks: Arranging, Folding, Cup- Stacking, Drawer, Pouring, and Sweeping. An overview of our robot setup is shown in Figure 7. The robot's perception system consists of two RGB-D cameras mounted on the left and right sides of the workspace.
5.1 Real-World Benchmark
Real-world Setup. We evaluate our policies on a 7-DOF Franka Panda robot in six challenging tasks. Visual information is captured by two Orbbec Femto Bolt cameras, positioned provide left and right views. These sensors provide both RGB and depth images, which are used to generate calibrated 3D point clouds. All RGB and depth images resized to 180x320 resolution. The robot operates in an 8-dimensional action space, including joint positions and gripper state.
Datasets. For collecting demonstrations, we use telcoperation system consisting of leader robot and follower robot. For each task, we collect varying numbers of language-conditionee trajectories detailed in Table 3. To ensure robust evaluation,
randomly initialize the object and goal states, introducing significant variation in the objects used. For instance, in the sweep task, the broom can appear in 10 different areas, and the garbage in 4 different areas. In addition, the number, positions, and even categories of trash items are varied in the collection and evaluation.
5.2 Baselines and Metrics
Baseline. To evaluate the effectiveness of the point-map representation, benchmark methods against RGB-only policy, sharing with the same backbone. Each method is evaluated over trials per task at training checkpoints 70,000, 80,000, and 90,000, using randomized initial object states to ensure robustness. We report results from the best-performing checkpoint for each method.
Metrics. Given the long-horizon nature and complexity of the tasks, we introduce structured scoring metric to enable fair and detailed comparisons. Each task decomposed into multiple stages, with each successfully completed stage contributing point to the overall score. The final task score is the sum of the completed intermediate stages, providing more granular measure of progress and policy effectiveness. The details of our scoring metrics can be found in Table 5.

---

Table shows average completed stages. Max. indicates the total number of stages task.
Tisks
Denon Per Task
Methods with Scores
PMP-xyz
Max
Arranging
80
2.05
2.10
2.25
3
Fight
Folding
45
2.1
0.80
3
Cameua
Careeaa
Cup-Stacking
75
1.40
0.45
3
Dowwer
120
2,00
2.16
2.40
Pouring
1.00
Sweeping
90
1,80
0.80
2.15
Figure 7: Real world experiments consisting of six tasks. The left figure shows our setup and the Drawer task. The Table shows the average completed stages out of 20 evaluations.
5.3 Real- World Main Results
We evaluate all the methods with 20 rollouts per task. As can be seen in Table 3, our proposed PMP policy consistently outperforms all baselines across all evaluated real-world tasks. Compared to the RGB-only policy, PMP achieves at least a 0.2-point improvement in accumulated scores, demonstrating the effectiveness of fusing both point-map and RGB modalities. Notably, on the Folding task, PMP increases the score from 2.1 to 2.5 using only 45 demonstrations, showcasing strong sample efficiency.
Interestingly, the PMP-xyz also outperforms the RGB-only baseline on several tasks, underscoring the value of spatial structure in guiding action prediction. However, its performance drops significantly in tasks involving deformable objects, such as Folding and Sweeping, where object geometry would change over actions. In these scenarios, the lack of appearance cues leads coarse and less reliable action predictions. This is especially evident in Cup-Stacking, a task that explicitly requires reasoning about object color, further highlighting the importance of RGB input. Overall, these results validate the effectiveness and generalizability of PMP in handling diverse and challenging manipulation tasks in the real world.
Limitation and Future Work
Our current approach has two main limitations. First, simply concatenating the point map and RGB tokens may not optimally leverage the complementary information each modality. More sophisticated fusion mechanisms could potentially extract richer cross-modal relationships and further improve performance. Second, our point map visual encoders are trained entirely from scratch, which constrains their performance compared to the RGB modality that benefits from ImageNet pretraining. For future work, developing pretraining objectives specifically designed for point map encoders represents promising direction. Just as vision models benefit substantially from pretraining on large image datasets, establishing similar paradigms for point map representations could dramatically improve performance, enabling more robust geometric feature learning before fine-tuning specific robotic tasks.
7 Conclusion
We present PointMapPolicy (PMP), novel diffusion-basedr imitation learning framework that effectively integrates 3D geometric reasoning with standard vision techniques. By projecting depth pixels into a multi-channel image of XYZ coordinates, PMP leverages existing visual encoders. while an efficient XLSTM-based diffusion network denoises action tokens to generate precise control sequences. Empirical results on RoboCasa and CALVIN demonstrate that PMP not only achieves state-of-the-art performance but also offers significantly faster training and inference. Comprehensive ablations on observation modalities and fusion strategies further highlight the clear advantages of structured map representations. Looking forward. we plan to explore large-scale pretraining of point-map models to extend generalization across diverse robotic tasks.
10

---

8 Acknowledgments
This work was supported by the European Research Council (ERC) under the European Union's ss Horizon Europe programme through the project SMARTP (Grant Agreement No. 101 (01177393). The authors also acknowledge funding from the German Research Foundation (DFG) within the framework of Collaborative Research Centre SFB 1574 "Circular Factory for the Perpetual Product" (Project No. 471687386). This work was also supported by funding from the pilot program Core Informaties of the Helmholtz Association (HGF). NS and GN were supported by the Carl Zeiss Foundation under the project JuBot (Jung Bleiben mit Robotern.. Xiaogang Jia and Xinkai Jiang acknowledge the support from the China Scholarship Council (CSC). The authors also acknowledge support by the state of Baden- Württemberg through the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden- Würtemberg, and by the German Federal Ministry of Education and Research.
References
[1] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-1 1):1684-1704 2025.
[2] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. T0: A vision- iagiionlanguage-cction flow model for general robot control, 2024. URL htps://arxiv. wrg/abs/2410.24164
[3] Xiaogang Jia, Denis Blessing, Xinkai Jiang, Moritz Reuss, Atalay Donat, Rudolf Lioutikov, and Gerhard Neumann. Towards diverse behaviors: A benchmark for imitation learning with human demonstrations. arXiv preprint arXiv:2402.14600 2024.
[4] Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces on robot learning. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https::/ wwtp:/neeteoeaiieeeeeeeeeeeeeeae wdw.ormmmmd.-2nnnooeo....
[5] Skand Peri, lain Lee, Chanho Kim, Li Fuxin, Tucker Hermans, and Stefan Lee. Point cloud models improve visual robustness robotic learners, 2024. URL https: //axxiv..//ii//a//aa org/ /bs/ 2404.18926.
[6] Balazs Gyenes, Nikolai Franke, Philipp Becker, and Gerhard Neumann. PointpatchRL masked reconstruction improves reinforcement learning on point clouds. In 8th Annual Conference on Robot Learning, 2024. URL https: htt.:::ttt///eeeeeeee //pp//eeppeeeeeeee////ee new/ffouummiiiii j
[7] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.0030 03355, 2024.
[8] Eugenio Chisari, Nick Heppert, Max Argus, Tim Welschehold, Thomas Brox, and Abhinav Valada. Learning robotic manipulation policies from point clouds with conditional flow match- ing. In 8th Anmual Conference Robot Learning, 2024. URL https:: /pwp/eveeeeeeeeeeeee/eeeeeeeeeee net/ forum? dd1tEnn.JW2..
[9] Zhou Xian, Nikolaos Gkanatsios, Theophile Gervet, Tsung- Keung-Weii Ke, and Katerina Fragkiadaki. Chaineddiffuser:
Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference Robot Learning, 2023. URL https: http:::////pp/vv//////e net/ forum? 1d=W0zg22BBTABB.
[10] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[11) Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations, arXiv preprint 4rXhv:2402.10885 2024.
11

---

[12] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 20697-20709, June 2024.
[13] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71-91. Springer, 2024.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. pages 770-778, 2016. URL httpst/s///ooeaaceees thecvf. com/ content_ paper html.
cvpr aypp__216/html///ee 2tt01///ttt//eeeeeeeeeeeeeeeeeee httppeeeiidull__eeeee httpge/esssaaa___aaaaen_____PPPO
[15]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https://openreviie
www.runiddiiicaiTT YicbFaNTTy.
[16] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976-1 1986, June 2022.
[17] Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked Autoencoders for Point Cloud Self-super Learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, European Conference on Computer Vision 2022, pages 604-621, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20086-1. doi: 10. 11-1007977783-03-20006-----
[18] Guangyan Chen, Meiling Wang. Yi Yang. Kai Yu, Li Yuan, and Yufeng Yue. PointGPT: Auto- regressively Generative Pre-training from Point Clouds, May 2023. http://arxxi.. org/ abs/2300 /205 11487. arXiv:2305.1 1487 [es].
[19] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Flucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreviic wttppeerrrreeeeeeeeoormaaaaaaaee www.:e/e.oorrreeeeeea/atooo.
[20] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint XiXX:224660022233 2024.
[21] Oier Mees, Lukas Hermann, Erick Rosete- Beass and Wolfram Burgard. Calvin: A benchmark for condittone policy lcarning for long-horizer robot manipulation tasks. IEEE Robotics and Automation Letters, 733)7732777334, 2022.
[22] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Xu, Hang Su, and Jun 7hu. RDT-1b: a diffusion foundation model for bimanual manipulation. In The Thirteenth International Conference on Learning Representations, 2025. URL. https: / //opereviiw wwwriiiwwweeeffoooo htt///ooommiiddaaaaaaaaaa
[23]
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Open- VLA: An open-source vision-languaag-aation model. In 8th Annual Conference on Robot Learning, 2024. URL https:t::/// wwwp:/wwwwwwwwwwweeeeeeeeeeeeeee net/teruuuriieee wwt.ffoommmddddddeeeeeee ZMM.dDDZAE.......
[24] Moritz Reuss. Jyothish Pari, Pulkit Agrawal, and Rudolf Lioutikov. Efficient diffusion trans- former policies with mixture of expert denoisers for multitask learning. In The Thirteenth International Conference on Learning Representations, 2025. URL http//oorummid-DDmlllla.m
[25] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source
12

---

generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
[26] Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew Davison. Coarse-to- Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 13729 13738, Los Alamitos, CA, USA, June 2022. IEEE Computer Society. doi: URL attps://don iececompttersocietyy org/10.
PWW.9CCCVRR2288222222.02 1199/CVPR5200 2022.0132
[27] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022. URL https:
/ // /openreview net/forum? id=PS_ _es___SS_____ WCvD.
[28] Atalay Donat, Xiaogang Jia, Xi Huang, Aleksandar Taranovic, Denis Blessing, Ge Li, Hongyi Zhou, Hanyi Zhang. Rudolf Lioutikov, and Gerhard Neumann. Towards fusing point cloud and visual representations for imitation learning. In 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025. URL https: https: /ttprr/eee///eeeee////// mwttforumm id= 5cG7ilWX11
[29] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. RVT: Robotic view transformer for 3d object manipulation. In 7th Annual Conference Robot Learning, 2023. URL https: /ttp////peeeeeeeeeeeeee net
[30] Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. RVT-2: Learning precise manipulation from few demonstrations. In RSS 2024 Workshop on Geometric and Algebraic Structure in Robot Learning, 2024. URL https: httpttppp/pppppeeeeeeeeeeee T6c4qoc6PG.
meet/forum?dd
[31] Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martnn. What matters in learning from offline human demonstrations for robot manipulation. In Sth Annual Conference on Robot www/teetturiieee wawerrmmiddJJsffBteeeee.
Learning, 2021. URL httpps:/
[32] Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning Sk$ modes with one stone. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttps://openrevicee www.eeeweeeeeeeeeeeeeeeeeeeeeeee wost//ooummahi iidiiddaap agdddda-T-rrsssss
[33] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023. URL
https://doli org/10. 15607 /RSS. 2023.0222 XIX. 028.
[34] Homanga Bharadhwaj. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In First Workshop on Out- Distribution Generalization in Robotics at CoRL 2023, 2023. URL https: //tp:r/////peeevieee
[35] Xiaogang Jia, Qian Wang, Atalay Donat, Bowen Xing. Ge Li, Hongyi Zhou, Onur Celik, Denis Blessing, Rudolf Lioutikov, and Gerhard Neumann. Mail: Improving imitation learning with mamba, 2024. URL https: / /arxiv.vvvvv...i//aax/aaaaaa/ www. ab///b///4000 082344.
[36] Xiaogang Jia, Atalay Donat, Xi Huang, Xuan Zhao Denis Blessing, Hongyi Zhou, Han A Wang, Hanyi Zhang, Qian Wang, Rudolf Lioutikov, al. X-il: Exploring the design space of imitation learning policies. arXiv preprint arXiv:2502. 12330, 2025.
[37] Albert Gu and Tri Dao. Mamba: Linear- sequence modeling with selective state spaces, 2024. URL /tttps://pperreeeee wwwpeearwwiieeeeeeooooorrreeeeee ftt.://fffrrrrrdd-1111.ffff
[38] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Ex- tended long short-term memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: htttt::tttp//reeeeee http:/ppwwwreeeeeeeeeeeee wwwp:e/.wweeettoodnddeeeddaadaae
13

---

[39] Tony Z Zhao. Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint XXi::2304.13705, 2023.
[40] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674 2011. doi: 10.1 10.1162/NECO__000 _01122
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[43] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[44] Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang. et al. Gr00t nl: An open foundation model for generalist humanoid robots. arXiv preprint 2rXIv:2503.14734 2025.
[45] Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan, Ping Luo, Heming Cui, Yi Ma, and Hongyang Li. Closed-loop visuomotor control with generative expectation for robotic manipulation. Advances in Neural Information Processing Systems, 37:139002-139029, 2024.
[46] Moritz Reuss, Ömer Erdinç Yagmurlu, Fabian Wenzel, and Rudolf Lioutikov. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals. In Robotics: Science and Systems, 2024.
[47] Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15100 2024.
[48] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. In International Conference on Learning Representations, 2024.
[49] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 10639, 2023.
[50] Hongtao WU, Ya Jing, Chilam Cheang. Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference Learning Representations, 2024.
[51] Qingwen Bu, Hongyang Li, Li Chen, Jisong Cai, Jia Zeng, Heming Cui, Maoqing Yao, and Yu Qiao. Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arXi::2410.0800) 2024.
[52] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. :rad-cam+++: Generalized gradient- craddeet-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839-847, 2018. doi: 10. 1 http://CC.2018.00099900
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution novel and efficient representation. This is clearly outlined in the abstract and expanded in the introduction and method sections. Claims about performance are verified through experiments on simulation and real robot evaluations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
We discuss the limitations of this work in Sec. 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the baselines, evaluation details of simulation benchmarks in A, the real world experiment in B, and corresponding hyperparameters to reproduce the experiment results in the Appdendix. C.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open source the codes in the near future once they are cleaned up and anonymity is not concern anymore. All the experiments we conducted were using open- datasets. In the experiments section and appendix we provide information to get access to the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ public/ /twggddee/oeeuuuiieeeeeeee
wwm.imissiioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//n //iips.co. httpscccpppbbcciiicceeae guwdes/eeeeeeee SCodeSubmiei is onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We list the hyperparameter with the optimizer for each of the algorithms in Tab. 6, and how they were chosen is in the appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The experiment of RoboCase provides confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The compute resources are described in Appendix. E
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps:///ne h/tpprip///////..e//// cccppublic/hthiccuuueeeeeeeeee
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
19

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we do used pretrained models and open source code base for baselines, which is clearly stated in both experiment section and appendix.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We plan to open source the code in the future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development of this paper does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non non-standard components. Please refer to our LLM policy (https / :tttps://neur//e //euripp......// wCc.Coooenene////// (at.cc/ee////////// /LLM) for what should or should not be described.
21

---

A Simulation Experiment Details
A.1 RoboCasa Benchmark
RoboCasa [20] is a large-scale simulation framework developed to train generalist robots in realistic and diverse home environments, with particular focus kitchen scenarios. The benchmark comprises 100 tasks, including 25 atomic tasks with 50 human demonstrations and 75 composite tasks with auto-generated demonstrations. These tasks are centered around eight fundamental robotic skills relevant to real-world home environments: (1) pick-and-plaat place, (2) opening and closing doors, (3) opening and closing drawers, (4) twisting knobs, (5) turning levers, (6) pressing buttons, (7) insertion, and (8) navigation.
To comprehensively evaluate our method, we selected five tasks from the atomic tasks described in Table 4, each representing a distinct skill.
Task Name
Description
Pick-and-Place Place Tasks
PickPlace Counter To Microwave
Pick an object from the counter and place it inside the microwave (door is open).
PickPlace Counter To Sink
Pick an object from the counter and place it the sink.
PickPlace Microwave MiccPlaceMicreuaee _ooeee_TTeeeeeeee To__oo_Cnoo Counter
Pick an object from the microwave and place on the counter (dour open).
PickPlace Sink Counnter
Pick an object from the sink and place the counter the sink.
Drawer Tasks
Open Drawer
Open drawer.
Close Drawer
Close drawer.
Stove Tasks
Stove_Or
Turn a specific stove burner by twisting its knob. Turn off specific stove burner by twisting its knob.
Stove_ Off
Sink Tasks
SinkFaucet On
Turn on the sink faucet to start water flow. Turn off the sink faucet to stop water flow. Rotate the sink spout.
SinkFaucet Off
Tarn_Sink Sink_Sinkoor
Coffee Machine Tasks
Coffee_Prest Button
Press the button to pour coffee into the mug. Place the mug into the coffee machine's mug holder.
Coffee Serve_Mee
Remove the mug from the holder and place it the counter.
Microwave Tasks
Microwave On
Start the microwave by pressing the start button. Stop the microwave by pressing the stop button.
Microwave_ Off
Table 4: RoboCasa task set.
A.2 CALVIN Benchmark
Benchmark Setup. The CALVIN benchmark [21] is long-horizon manipulation benchmark featuring four visually distinct tabletop environments (A-D), each containing a common set of objects and 34 manipulation tasks. Agents are given natural language instructions deseribing sequences of up to tasks to be executed in order. The primary evaluation involves completing 1000 such instruction chains in environment D. Agents are scored by the number of tasks successfully completed per chain, with a maximum rollout length of 5.
Evaluation Protocol. We evaluate PointMapPolicc on one standard CALVIN settings: ABC-D. where the policy is trained on environments A, B, and C, and evaluated zero-shot on D. Only 1% of the play data is paired with language, requiring models to learn from primarily unlabeled data. The ABC-D setup tests visual and environmental generalization, while D->D emphasizes efficiency in low-resource, language-scarce settings.
Baselines. We compare against a broad set of state-of-the-ar language-coodditonee policies span- ning imitation, diffuuiin,, and senddatioommdelll
architectures:
RoboFlamingo [48]: based on OpenFlamingo, this model integrates a frozen VLM with lightweight policy head. is pretrained on large-scale laniunne data and finetuned CALVIN using supervised behavior cloning.
22

---

Drawer
Stack
Fold
Score
Open the upper/lower drawer
Pick the correct cup
Pick the towel
1
Pick the object and place into drawer
Stack failed
Fold the towel
2
Close the Drawer
Stack the cups
Fold the towel perfectly
3
Pour
Sweep
Arrange
Score
Pick cup
Pick the broom
Open mixer machine
Pour the contents
Sweep 20% of garhage
Pick the container and place on pad
2
Pour all contents into container
Sweep 50% of garbage
Close the mixer machine
3
Put the cup back
Sweep all garbage
Table 5: Task score metric details of real robot and evaluation standards.
SuSIE [49]: a scalable instruction-folloving diffusion policy. It is pre-trained on curated robot demonstrations and uses an iestruction-condiiionne denoising process with significant offline finetuning.
GR-1 [50]: a powerful decoder-only transformer trained on large-scale synthetic video data. The model is capable of generating long sequences of actions and is finetuned CALVIN for grounding.
CLOVER [45]: video diffusion planner that predicts intermediate visual goals via video generation and closes the loop using low-level policy feedback. It does not require internet- scale pretraining and achieves strong multi-step rollout success.
MoDE [24]:
model with sparse routing. supports both small (non-pretrained) and large (pretrained) variants. The pretrained variant achieves top performance while maintaining low inference
Seer / Large [47]: large-scale transformer models pretrained on 1000+ hours of robot play data. Seer incorporates language, vision, and action streams into a unified transformer and achieves strong generalization, particularly when scaled up.
B Real World Experiment Details
We conducted six real-world experiments on a Franka Panda Robot: Drawer, Stack, Pour, Sweep, Fold, and Arrange.
B.1
Task Metric
Given the complexity and long-horizon nature of the tasks, we decompose each task into several discrete stages. The final score is computed as the total number of successfully completed stages. Details of the scoring metric design are provided in Table
B.2 Task Description
Drawer: In the Drawer task, there a cabinet with two drawers and two different objects, a cube and cylinder. The robot must follow manguage-specifeen instruction to open the designated drawer. pick up the target object, place it inside the drawer, and then close the drawer. The key challenges involve handling the random initialization of both the cabinet's position and the objects' locations.
Stack: the Stack task, four cups of different colors and sizes are provided. The robot must stack the cups in specific order based on their colors. The main challenges lie in accurately recalling the stacking sequence and executing precise placement, as the cups are closely sized and must fit together properly.
Pour: the Pour task, three distinct cups and three different containers are placed in randomized initial positions. The robot must generalize to novel object configurations while maintaining the
23

---

precision necessary to pour the contents from the cups into the containers without spilling. The primary challenge lies in adapting to varying spatial arrangements while executing controlled and accurate pouring motions.
Unlike standard Pick-and-Plan tasks, this task requires the robot acquire novel sweeping skill. In the Sweep task, the positions of the broom, dustpan, and trash vary across trials, and even the number of trash items changes. The key challenge is manipulating deformable trash materials that differ from those encountered during training, requiring the policy to exhibit strong generalization and adaptability.
Sweep:
The Fold task requires precise manipulation skills. The goal is to neatly fold a towel that is randomly oriented at the start of each trial. The primary challenge lies in accurately handling the soft, deformable material to achieve a clean and consistent fold despite varying initial conditions.
Fold:
Arrange:
In the Arrange task, the setup includes mixing machine and a container. The robot must follow a specific sequence: first, open the mixing machine; next, place the container on the designated pad; and finally, close the machine. This task primarily emphasizes long-horizon planning, requiring the robot to execute a multi-step procedure in the correct order.
C Hyper Parameters
Hyperparameter
CALVIN ABC
RoboCasa
Real World
Number of x-Blocks
10
8
6
Attention Heads
Action Chunk Size
10
10
10
History Length
1
1
Embeddiing Dimension
2048
768
2048
Image Fncoder
Fil M-ResNet50
ComoNextV2
Fil.M-ReNNetS CLIP VIT-B/32
Goal Lang Eneader
CLIP VIT-B/32
CLIP VIT-B/32
Attention Dropout
0.3
0.3
0.3
Residual Drupout
0.1
MI.P Dropout
0.1
0.1
0.1
Optimizer
AdamW
Adumm
Adamn
Betas
[0.9,
[0.9, 0,95)
[0.9, 0.95]
Learning Rate
le-4
le-4
le-4
Transformer Weight Docay
0.05
0.05
0.05
Otber weight decay
0.05
0.05
0.05
Batch Size Train Steps in Thousands
128
128
25
15
30
80
80
0.001
0.001
0.001
d=
0.5
0.5
0.5
EMA
True Expomential DDIM I1I
True
Time steps
Exponential DDIM
Expomential
Sampler
DDIM
Trainable Parameters (Millions)
147
96
Table 6: Summary of all the Hyperparameters for our experiments.
We export all the hyper parameters across three benchmarks for reproduction.
D Activation Map Analysis
To gain qualitative insights into what regions the visual encoders attend to during action inference, we visualize activation maps using Grad-CAM++ [52]. Unlike classification tasks, our diffusion- policy does not predict discrete categories, therefore, we apply Grad-CAM+H using the diffusion loss as the target signal, following the approach of highlighting input regions that most influence the denoised trajectory prediction. We generate the heatmaps using the Grad-CAM++ implementation" and compute activations for each camera view across three RoboCasa tasks: OpenDrawer, Turn On Sink Faucet, and Coffee Serve Mug. In all figures, we use a 2 encoder and extract Grad-CAM++ heatmaps from the final convolutional block before normalization. Each visualization consists of six images arranged in two rows. The top row shows the raw visual input (RGB or XYZ visualized in color), and the bottom row displays the corresponding Grad-CAM++ heatmaps for each
http://ghhhbbbooooccoogpp///opp/ -rw---rddddaaa cam
24

---

of the three camera views: static left, static right, and wrist-mounted. These maps highlight spatial regions with the greatest impact on predicted actions.
Overall, the attention patterns are consistent with task-relevant visual cues. For example, activations commonly foeus the rohot gripper, the manipulated object, the goal location, depending on the modality and perspective.
Figures 8, 9 and 10 show results for the RoboCasa tasks Coffee Serve Mug, Open Drawer and Turn On Sink Faucet, respectively.
25

---

comerr
right comeno
gdpper
(a) RGB-only ConvNeXtv2 encoder. Top: raw 128x RGB input frames provided to the agent. Bottom: Grad-CAM++ heatmaps from the final convolutional layer.
18
18
PMP-xyz ConvNeXtv2 encoder. Top: 128x128 XYZ input visualized RGB. Bottom: Grad- CAM++ + heatmaps from the final convolutional layer.
Figure 8: Raw RGB, XYZ input frames and Grad-CAM++ activations the Coffee Serve Mug
RoboCasa task for RGB-only and Point-map-only ConvNeXtv2 visual encoders.
26

---

comera
right cameno
gdpper camsen
92
(a) KGB-only ConvNeXtv2 encoder. Top: raw 128x RGB input frames provided to the agent. Bottom: -rad-CAM++ heatmaps from the final convolutional layer.
left samera
right comera
gripper cornera
12
(b) PMP-xyz ConvNeXtv2 encoder. Top: XYZ input visualized as RGB. Bottom: Grad-CAM++ heatmaps from final convolutional layer.
Figure 9: Raw RGB, XYZ input frames and Grad-CAM++ activations on the Open Drawer RoboCasa task for RGB-only and Point-map-only ConvNeXtv2 visual encoders.
27

---

comera
rihtt
grlper
(a) KGB-only ConvNeXtv2 encoder. Top: raw 128x RGB input frames provided to the agent. Bottom: Grad-CAM+- heatmaps from the final convolutional layer.
left samera
right comera
gripper comera
18
14
(b) PMP-xyz ConvNeXtv2 encoder. Top: XYZ input visualized as RGB. Bottom: Grad-CAM++ heatmaps from final convolutional layer.
Figure 10: Raw RGB, XYZ input frames and Grad-CAM++ activations the Turn On Sink Faucet RoboCasa task for RGB-only and Point-map-only ConvNeXtv2 visual encoders.
28

---

E Compute Resources
For the CALVIN experiments, PMP-Cat employs Felm-ResNNe550 encoders for both RGB images and point maps, with 10 x-Blocks as backbones (512 latent dimensions), totaling 147M trainable parameters. Training utilizes Nvidia RTX 6000 Ada GPUs with 128 samples per GPU (512 total batch size). Each epoch completes in approximately 13 minutes, allowing full training (25 epochs) in under hours, excluding evaluation time.
For the RoboCasa experiments, PMP-Cat employs ConvNeXtv2 as encoders with 8 x-Blocks using 512 latent dimensions. Training utilizes 1 NVIDÍA A100-SXM4-40GB with 128 batch size.
For the real-robot experiments, PMP-Cat employs Film-ResNeet00 as encoders for both images and point maps, with 6 x- B-Blocks using 256 latent dimensions. Training utilizes Nvidia RTX 6000 Ada GPUs with 128 batch size.
29


Reviews:
[{"review_id": "e1xgDBL8H8", "reviewer": "Reviewer_twzC", "summary": "This paper proposes PointMapPolicy (PMP), a multi-modal fusion framework for imitation learning that integrates both RGB images and point maps. The authors compare different multi-modal fusion methods, including early fusion, late fusion, and attention-based fusion. Simulation and real-world experiments show that the proposed method outperforms policies trained using only RGB or Depth modalities.", "questions": "How does the proposed point map method compare to the point cloud method? The paper only compares to point cloud-based methods that have different backbones.", "limitations": "How to maintain performance when one modality is performing well and one is not? For example, the two modalities have large success rate differences for the TurnSinkSpout task in RoboCasa, resulting in PMP performing worse than using either modality. The authors should discuss these kinds of edge cases and describe them in the limitations section. Additionally, Figure 5 doesn’t show a lot of difference in the different fusion methods.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper is clearly written and well-structured, making it easy to follow.\n2. The paper studies an underexplored area in robot manipulation: policy learning from 3D input. \n3. The paper provides a thorough comparison across RGB, depth map, and point map input modalities. The paper highlights the benefits of combining them.\n\n\nWeakness:\n1. The visual encoder and the backbone in the proposed model are trained from scratch, while prior work shows that pretrained vision encoders can significantly boost performance. The authors should clarify their decision not to use pretrained encoders or pretrained backbones and discuss how existing pretrained models could be adapted to handle additional modalities like point maps. \n2. The ablation study compares different visual encoder choices. However, the performance of imitation learning also depends on the backbone choice. It is unclear how much the performance gain is from multi-modal fusion itself versus the choice of backbone. More controlled experiments are needed to isolate the contribution of fusion.\n3. The introduction states that point clouds struggle to capture fine-grained detail, motivating the use of point maps. However, this claim would be more convincing with a quantitative comparison between point clouds and point maps, considering the availability of convolutional networks like KPConv that process unstructured point clouds effectively. \n4. The related work could be strengthened by discussing more recent approaches in multi-modal policy learning. In particular, the authors should consider comparing with:\nZhu, Yichen, Zhicai Ou, Feifei Feng, and Jian Tang. \"Any2Policy: Learning Visuomotor Policy with Any-Modality.\" Advances in Neural Information Processing Systems 37 (2024): 133518-133540.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work and for the many helpful comments and suggestions. We hope the following replies address the questions and concerns raised. \n\n---\n\n> The visual encoder and the backbone in the proposed model are trained from scratch, while prior work shows that pretrained vision encoders can significantly boost performance. The authors should clarify their decision not to use pretrained encoders or pretrained backbones and discuss how existing pretrained models could be adapted to handle additional modalities like point maps.\n> \n\nWe thank the reviewer for bringing this concern to our attention and apologize for any confusion. We train point map encoders from scratch because no existing pretrained models are designed for XYZ coordinate inputs. Standard ImageNet-pretrained models expect RGB inputs with specific statistical properties that don't apply to geometric coordinates. Our RGB encoders do use ImageNet pretraining (FiLM-ResNet50), which is standard practice [1,2,3,4]. For point maps, we show that training from scratch with ConvNeXtv2 consistently outperforms other modalities (Table 1), suggesting our approach effectively learns geometric features. We agree that developing pretrained models for point map representations is valuable future work - since point maps can leverage any existing visual encoder architecture (ResNet, ViT, etc.), they could potentially be pretrained using visual objectives (detection, segmentation) or directly on robot learning datasets.\n\n---\n\n> The ablation study compares different visual encoder choices. However, the performance of imitation learning also depends on the backbone choice. It is unclear how much the performance gain is from multi-modal fusion itself versus the choice of backbone. More controlled experiments are needed to isolate the contribution of fusion. …… How does the proposed point map method compare to the point cloud method? The paper only compares to point cloud-based methods that have different backbones.\n> \n\nWe thank the reviewer for suggesting ablations to demonstrate point map effectiveness. We ensure fair evaluation by using identical xLSTM architectures across all modalities. Table 1 shows PMP-xyz (49.12%) vs RGB (40.16%) and Depth (44.00%) with identical backbones, demonstrating 9% and 5% improvements respectively from modality choice, not architecture differences. \n\nAdditionally, Reviewer Dr9P raised similar concerns about separating input modality effects from policy backbone influences. We conducted extensive controlled ablations comparing different point cloud processing methods using identical xLSTM backbones. These results demonstrate that our improvements stem from the point map representation itself, not architectural choices. **We kindly refer the reviewer to our detailed response to Reviewer Dr9P for the complete ablation study.**\n\n---\n\n> The introduction states that point clouds struggle to capture fine-grained detail, motivating the use of point maps. However, this claim would be more convincing with a quantitative comparison between point clouds and point maps, considering the availability of convolutional networks like KPConv that process unstructured point clouds effectively.\n> \n\nWhile convolutional architectures for point clouds such as KPConv exist, they empirically perform worse than other architectures. Point Transformer [5] and its recent variant Point Transformer v3 [6], both attention-based architectures, significantly outperform KPConv. We compare against 3DA, which is also attention-based. Similarly, Point-BERT [7], a point patch architecture, outperforms KPConv. We present an ablation against a Point Patch architecture in our response to reviewer Dr9P, where PMP outperforms it. Given that convolutional methods are consistently shown to be inferior in the point cloud classification and segmentation literature, we opt to compare against attention-based and aggregation-based architectures, which show more promise.\n\n---\n\n> The related work could be strengthened by discussing more recent approaches in multi-modal policy learning. In particular, the authors should consider comparing with: Zhu, Yichen, Zhicai Ou, Feifei Feng, and Jian Tang. \"Any2Policy: Learning Visuomotor Policy with Any-Modality.\" Advances in Neural Information Processing Systems 37 (2024): 133518-133540.\n> \n\nWe appreciate this reference and will add it to our related work. Any2Policy focuses on cross-modal policy transfer, while our contribution is specifically the structured point map representation that enables direct application of vision architectures to 3D data. These are complementary approaches.\n\n---\n\n> How to maintain performance when one modality is performing well and one is not? For example, the two modalities have large success rate differences for the TurnSinkSpout task in RoboCasa, resulting in PMP performing worse than using either modality. The authors should discuss these kinds of edge cases and describe them in the limitations section. Additionally, Figure 5 doesn’t show a lot of difference in the different fusion methods.\n> \n\nWe thank the reviewer for this insightful observation. Table 1 reveals an important limitation: while PMP shows significant improvements on tasks like PnPSinkToCounter, TurnOffMicrowave, and CoffeePressButton, it performs similarly or worse than individual modalities on others (e.g., TurnSinkSpout). This suggests that simple concatenation fusion may not be optimal when one modality dramatically outperforms the other.\n\nFor instance, in the Turning Levers category, PMP-xyz substantially outperforms RGB, yet PMP performs similarly to RGB alone. This indicates the fusion mechanism may be dominated by the weaker modality rather than leveraging the stronger one. This represents a key limitation of our current approach that we will add to our limitations section. This analysis points toward important future research on adaptive fusion mechanisms, such as learned modality weights or dynamic modality selection, that could better exploit complementary strengths rather than averaging performance.\nDespite this limitation, concatenation fusion shows consistent advantages across diverse task categories (Figure 5), and PMP demonstrates substantial improvements over individual modalities on CALVIN (4.01 vs RGB: 3.15, PMP-xyz: 2.03), demonstrating the effectiveness of this approach.\n\n---\n\nWe express our gratitude to the reviewer for their valuable comments and suggestions. We are pleased to address any additional questions or concerns that may arise.\n\n---\n\nReferences\n\n[1] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" *The International Journal of Robotics Research* (2023).\n\n[2] Nasiriany, Soroush, et al. \"Robocasa: Large-scale simulation of everyday tasks for generalist robots.\" RSS 2024\n\n[3] Reuss, Moritz, et al. \"Efficient diffusion transformer policies with mixture of expert denoisers for multitask learning.\" ICLR 2025\n\n[4] Wen, Junjie, et al. \"Dexvla: Vision-language model with plug-in diffusion expert for general robot control.\" *arXiv preprint arXiv:2502.05855* (2025).\n\n[5] Zhao, Hengshuang, et al. \"Point transformer.\" ICCV 2021.\n\n[6] Wu, Xiaoyang, et al. \"Point transformer v3: Simpler faster stronger.\" CVPR 2024.\n\n[7] Yu, Xumin, et al. \"Point-bert: Pre-training 3d point cloud transformers with masked point modeling.\" CVPR 2022."}, {"review_id": "cEuF4lZXEd", "reviewer": "Reviewer_Dr9P", "summary": "This paper proposes PointMapPolicy for diffusion-based imitation learning in robotic manipulation. Its core contribution is to use a Point Map observation that projects the XYZ coordinates of a point cloud back onto the image plane, supplying rich geometric cues for policy learning. Extensive experiments in both simulation and the real world show that PointMapPolicy outperforms existing baselines.", "questions": "1. As mentioned in Weakness 2, separating the evaluation of the input modality and the policy backbone would strengthen the paper. Although Table 1 shows that the proposed method is outperforming DP3 and 3DA, is it because Point Map is a better input modality, or because the X-Block is better than the UNet used in DP3 and the transformer used in 3DA? Adding an additional baseline that uses a point cloud encoder + X-Block variation would be beneficial.\n2. Moreover, another baseline that uses 3D Lifting + X-Block will also better demonstrate the advantages of Point Map, particularly because 3D Lifting is listed as a comparison in Figure 1.\n3. A clearer contribution summary would be useful. How is the proposed method different from other robot learning approaches that use a similar Point Map concept, like RVT?\n4. Why do the authors list their model as no pre-training in Table 2? Figure 2 states `PMP integrates multiple modalities: language instructions encoded by a pretrained CLIP model, images processed by pretrained visual encoders, and point maps processed by visual encoders trained from scratch.`. Line 198 seems to argue that the proposed method does not use `robot-specific pretraining`, but it is important to clarify what is the difference here, especially because the proposed method underperforms the pretrained baselines in Table 2.\n5. The overall writing could be polished; for instance, the abstract does not read very smoothly.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strength\n1. The paper tackles input-modality selection in robotic manipulation, a very important problem.\n2. The proposed point-map representation can be easily adopted by different policy-learning frameworks.\n3. I appreciate the experimental videos on the website, especially those testing diverse initial configurations.\n\nWeakness\n1. The technical contribution appears incremental. Point Maps are an existing concept [10, 11], and prior work has used similar inputs in manipulation. E.g., RVT uses a similar input format: ```Specifically, for each view, we render three image maps with a total of 7 channels: (1) RGB (3 channels), (2) depth (1 channel), and (3) (x,y,z) coordinates of the points in the world frame (3 channels).```\n2. The goal is to advocate Point Maps as an input modality, but the experimental justification is not rigorous. Specifically, Table 1 mixes input modality with backbone architecture, where PMP uses an X-Block, while DP3 (UNet) and 3DA (Transformer) use different policy backbone, making it unclear whether the improvement come from the modality or the policy backbone. \n3. Some alternative point-cloud approaches, such as the 3D-Lifting method in Figure 1, are also missing.", "was_revised": true, "final_justification": "Most of my concerns are addressed by the authors. The new experiment clearly demonstrates the advantages of using point maps as the input modality for policy learning. While I have reservations about the novelty of the paper, I partially agree with the authors’ claims and arguments, so I increase my rating to borderline accept.", "rebuttal": "We thank the reviewer for taking the time to review our work and for the many helpful comments and suggestions. We hope the following replies address the questions and concerns raised. \n\n---\n\n> The technical contribution appears incremental. Point Maps are an existing concept [10, 11], and prior work has used similar inputs in manipulation. E.g., RVT uses a similar input format\n……. A clearer contribution summary would be useful. How is the proposed method different from other robot learning approaches that use a similar Point Map concept, like RVT?\n> \n\nWe thank the reviewer for this important concern. While Point Maps exist as a concept in 3D vision, our contribution lies in demonstrating their effectiveness for general imitation learning. We provide a clear summary of our contributions and key differences from RVT:\n\n- RVT is a learning-based motion planner that predicts keyframe poses and relies on traditional motion planners for execution. In contrast, PMP is the first to apply point maps to end-to-end behavior cloning using diffusion policies, enabling direct action prediction without separate planning stages.\n- While RVT uses point map-like inputs, it makes a number of other design decisions that make it hard to disentangle the benefits of point maps compared to other 3D representations. For instance, they re-render a point cloud with virtual cameras, ignoring self-occlusions by unobserved geometry. Our extensive controlled experiments across modalities with identical backbones provide the first rigorous evidence that point maps outperform traditional 3D representations in robotic learning contexts.\n- RVT naively concatenates geometric and color information at the channel level, while we systematically investigate multiple fusion strategies (Add/Cat/Attn) and demonstrate their robustness.\n\n---\n\n> Some alternative point-cloud approaches, such as the 3D-Lifting method in Figure 1, are also missing.\n> \n\nWe apologize for any confusion about our presentation regarding the 3D-Lifting methods. **3D Diffuser Actor (3DA) is indeed the 3D-Lifting method** shown in Figure 1(b), as stated in lines 70-72: \"3D Diffuser Actor computes tokens by lifting 2D image features into 3D space\".  Additionally, the Downsampling-based method in Figure 1(a) refers to DP3, as stated in lines 67-68: “DP3 encodes sparse point clouds using FPS”. We will make the baseline descriptions clearer in the final version.\n\n---\n\n> Table 1 mixes input modality with backbone architecture, where PMP uses an X-Block, while DP3 (UNet) and 3DA (Transformer) use different policy backbone, making it unclear whether the improvement come from the modality or the policy backbone. …… separating the evaluation of the input modality and the policy backbone would strengthen the paper. …… Adding an additional baseline that uses a point cloud encoder + X-Block variation would be beneficial. …….. Moreover, another baseline that uses 3D Lifting + X-Block will also better demonstrate the advantages of Point Map\n> \n\nWe thank the reviewer for this valuable suggestion and agree on the need for more controlled comparisons.\n\n**Additional Controlled Ablations:** To directly address backbone vs. modality effects, we conducted controlled experiments on RoboCasa using **identical xLSTM backbones** with different point cloud processing encoders: 1) **PointNet-xyz**: Following DP3, we gather point clouds from 3 camera views and use Furthest Point Sampling (FPS) to downsample to 1024 points, then apply MLP with maxpooling to create a compact 3D token. 2) **PointNet-color**: Same process as PointNet-xyz but using colored points with XYZRGB information. 3) **PointPatch**: We use FPS to sample 256 center points, apply k-Nearest Neighbors to create 256 point patches with 32 points each, tokenize each patch using MLP with maxpooling, then process the resulting tokens with a transformer to generate compact 3D representations. 4) **3D-Lifting**: We extract CLIP features (frozen) from each camera view and lift the 2D features into 3D space, then use a transformer to process the lifted tokens. The 3D tokens are then passed to the diffusion policy with an identical X-Block backbone. \n\n|  | PointNet-xyz | PointNet-color | PointPatch | 3D-Lifting | PMP-xyz (ours) |\n| --- | --- | --- | --- | --- | --- |\n| PnPCounterToMicrowave | $0.7 \\scriptstyle{\\pm 0.9}$ | $1.3 \\scriptstyle{\\pm 0.9}$ | $0.7 \\scriptstyle{\\pm 0.9}$ | $3.3 \\scriptstyle{\\pm 1.9}$ | $\\mathbf{13.3 \\scriptstyle{\\pm 3.4}}$ |\n| PnPCounterToSink | $1.3 \\scriptstyle{\\pm 0.9}$ | $0.7 \\scriptstyle{\\pm 0.9}$ | $0.7 \\scriptstyle{\\pm 0.9}$ | $0.7 \\scriptstyle{\\pm 0.9}$ | $\\mathbf{6.7 \\scriptstyle{\\pm 2.5}}$ |\n| PnPMicrowaveToCounter | $0$ | $0$ | $0$ | $4.0 \\scriptstyle{\\pm 1.6}$ | $\\mathbf{16.0 \\scriptstyle{\\pm 1.6}}$ |\n| PnPSinkToCounter | $0$ | $0$ | $0$ | $3.3 \\scriptstyle{\\pm 1.9}$ | $\\mathbf{8.0 \\scriptstyle{\\pm 1.6}}$ |\n| OpenDrawer | $16.7 \\scriptstyle{\\pm 2.5}$ | $10.7 \\scriptstyle{\\pm 0.9}$ | $25.3 \\scriptstyle{\\pm 2.5}$ | $26.7 \\scriptstyle{\\pm 2.5}$ | $\\mathbf{60.0 \\scriptstyle{\\pm 4.3}}$ |\n| CloseDrawer | $92.0 \\scriptstyle{\\pm 1.6}$ | $92.0 \\scriptstyle{\\pm 1.6}$ | $93.3 \\scriptstyle{\\pm 0.9}$ | $64.7 \\scriptstyle{\\pm 9.6}$ | $\\mathbf{96.0 \\scriptstyle{\\pm 1.6}}$ |\n| TurnOnStove | $42.0 \\scriptstyle{\\pm 0.0}$ | $31.3 \\scriptstyle{\\pm 3.4}$ | $38.0 \\scriptstyle{\\pm 4.9}$ | $28.7 \\scriptstyle{\\pm 5.2}$ | $43.3 \\scriptstyle{\\pm 5.7}$ |\n| TurnOffStove | $19.3 \\scriptstyle{\\pm 1.9}$ | $19.3 \\scriptstyle{\\pm 1.9}$ | $19.3 \\scriptstyle{\\pm 5.0}$ | $14.7 \\scriptstyle{\\pm 2.5}$ | $20.0 \\scriptstyle{\\pm 3.3}$ |\n| TurnOnSinkFaucet | $43.3 \\scriptstyle{\\pm 4.1}$ | $28.7 \\scriptstyle{\\pm 3.8}$ | $32.0 \\scriptstyle{\\pm 0.0}$ | $30.0 \\scriptstyle{\\pm 5.9}$ | $\\mathbf{76.7 \\scriptstyle{\\pm 4.1}}$ |\n| TurnOffSinkFaucet | $50.0 \\scriptstyle{\\pm 7.1}$ | $51.3 \\scriptstyle{\\pm 3.4}$ | $48.7 \\scriptstyle{\\pm 5.7}$ | $41.3 \\scriptstyle{\\pm 8.4}$ | $\\mathbf{82.0 \\scriptstyle{\\pm 1.6}}$ |\n| TurnSinkSpout | $62.0 \\scriptstyle{\\pm 1.6}$ | $64.0 \\scriptstyle{\\pm 1.6}$ | $58.7 \\scriptstyle{\\pm 3.4}$ | $64.0 \\scriptstyle{\\pm 0.0}$ | $\\mathbf{76.0 \\scriptstyle{\\pm 1.6}}$ |\n| CoffeePressButton | $8.0 \\scriptstyle{\\pm 1.6}$ | $8.7 \\scriptstyle{\\pm 5.7}$ | $8.0 \\scriptstyle{\\pm 3.3}$ | $10.0 \\scriptstyle{\\pm 4.3}$ | $\\mathbf{82.7 \\scriptstyle{\\pm 5.0}}$ |\n| TurnOnMicrowave | $36.0 \\scriptstyle{\\pm 3.3}$ | $26.7 \\scriptstyle{\\pm 5.0}$ | $35.3 \\scriptstyle{\\pm 9.6}$ | $25.3 \\scriptstyle{\\pm 2.5}$ | $\\mathbf{49.3 \\scriptstyle{\\pm 5.0}}$ |\n| TurnOffMicrowave | $48.0 \\scriptstyle{\\pm 0.0}$ | $39.3 \\scriptstyle{\\pm 5.2}$ | $46.7 \\scriptstyle{\\pm 4.1}$ | $28.0 \\scriptstyle{\\pm 7.1}$ | $\\mathbf{70.0 \\scriptstyle{\\pm 4.9}}$ |\n| CoffeeServeMug | $12.0 \\scriptstyle{\\pm 2.8}$ | $9.3 \\scriptstyle{\\pm 0.9}$ | $12.7 \\scriptstyle{\\pm 0.9}$ | $24.7 \\scriptstyle{\\pm 2.5}$ | $\\mathbf{69.3 \\scriptstyle{\\pm 6.6}}$ |\n| CoffeeSetupMug | $2.0 \\scriptstyle{\\pm 1.6}$ | $2.7 \\scriptstyle{\\pm 0.9}$ | $2.7 \\scriptstyle{\\pm 0.9}$ | $4.7 \\scriptstyle{\\pm 2.5}$ | $\\mathbf{16.7 \\scriptstyle{\\pm 3.8}}$ |\n| Average | $27.08$ | $24.12$ | $26.38$ | $23.38$ | $\\mathbf{49.12}$ |\n\nAll methods are using the same training scheme as PMP-xyz, and we report the results from 3 seeds. Results show **PMP-xyz consistently outperforms** these alternatives with identical architectures, demonstrating that improvements stem from our point map representation rather than backbone choice.\n\n---\n\n> Why do the authors list their model as no pre-training in Table 2? Figure 2 states `PMP integrates multiple modalities: language instructions encoded by a pretrained CLIP model, images processed by pretrained visual encoders, and point maps processed by visual encoders trained from scratch.`. Line 198 seems to argue that the proposed method does not use `robot-specific pretraining`, but it is important to clarify what is the difference here, especially because the proposed method underperforms the pretrained baselines in Table 2.\n> \n\nWe apologize for the confusion and the missing definition of the pre-training (PrT) in Table 2. PrT in our paper refers to robot-specific pertaining, for example, MoDE [1] is pre-trained on a diverse mix of multi-robot datasets curated from the OXE dataset Collaboration; Seer [2] is pre-trained on large-scale robotic datasets, such as DROID.\n\nRegarding our PMP model, we clearly state:\n\n- RGB encoders use ImageNet pretraining (standard vision pretraining)\n- Point map encoders trained from scratch (no existing pretraining for XYZ inputs)\n- No robot demonstration pretraining (unlike Seer, RoboFlamingo, etc.)\n\nPerformance Context: Compared to the Seer-Large scratch version (3.83) and pretraining version (4.28), our method achieves competitive results (4.01) without robot-specific pretraining.\n\nAdditionally, we also discussed the potential of pre-training point maps in the Limitation and Future Work.\n\n---\n\n> The overall writing could be polished; for instance, the abstract does not read very smoothly\n> \n\nWe will carefully revise the abstract and manuscript to improve clarity, flow, and overall readability in the final version.\n\n---\n\n> Should the table caption be moved to the top of the table?\n> \n\nThank you for pointing this out. We will correct this formatting in the final version.\n\n---\n\nOnce again, we sincerely thank the reviewer for their many insightful questions, which have greatly contributed to improving the quality of our work. We welcome any further concerns or questions they may have and would be glad to address them.\n\n---\n\nReferences\n\n[1]  Reuss, Moritz, et al. \"Efficient diffusion transformer policies with mixture of expert denoisers for multitask learning.\" ICLR 2025\n\n[2] Tian, Yang, et al. \"Predictive inverse dynamics models are scalable learners for robotic manipulation.\" ICLR 2025"}, {"review_id": "cvQmWqvj9N", "reviewer": "Reviewer_r4FG", "summary": "This paper attempts to augment vision-based policies with 3D information to improve the motion accuracy and hence the task success rate. Unlike previous works exploring point clouds or depth-image, the paper suggests using point map, converted from depth image via known camera intrinsic parameters. The main benefit is the possibility of using a similar image encoder and position embeddings as the RGB modality, in contrast to point cloud data. The approach explores different ways of fusing point map and RGB encodings, identifying the benefits of concatenation. The design is utilised in training a language prompted diffusion policy, leveraging xLSTM for light-weight encoding and training of the history. The approach is compared to several baselines with/without input carrying 3D information on two simulation benchmarks and real robot experiments. The results show improvement on the task success rate and the length of following the language instructions.", "questions": "* Can more insights be shed on why point map brings more performance gains comparing to other 3D especially depth map? It seems like depth map information is just distorted with a scale by not knowing the camera intrinsics. Any other factors may contribute to the performance gap?\n* What is the implication for the extra requirement of knowing camera intrinsics? Would that be a limitation for utilising large-scale video action data without camera parameters?\n* Can additional ablation study be done to show the significance of the xLSTM design choice?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Simple yet effective design to fuse 3D information in vision-policy training.\n* The motivation for having a unified visual encoding architecture for capturing 3D information is reasonable.\n* Extensive studies with multiple baselines and benchmarks, including real-robot experiments.\n* The design has the potential to augment multiple views/sensors input, for the ease of manipulating image encodings.\n\nWeakness:\n* Though the empirical performance is positive to the main hypothesis, more insightful discussion or studies on why point map is favoured, especially comparing to depth image, is lacking.\n* The significance of xLSTM is not very clear in the comparison.", "was_revised": true, "final_justification": "The authors' additional ablation helps to clarify some confusions in the remark. The significance of the proposed PointMap seems eclipsed somehow by the xLSTM used in peer works. But overall the paper provides an idea that seems simple yet effective for learning 3D vision-based manipulation.", "rebuttal": "We thank the reviewer for taking the time to review our work and for the many helpful comments and suggestions. We hope the following replies address the questions and concerns raised. \n\n---\n\n> Though the empirical performance is positive to the main hypothesis, more insightful discussion or studies on why point map is favoured, especially comparing to depth image, is lacking. \nCan more insights be shed on why point map brings more performance gains comparing to other 3D especially depth map? It seems like depth map information is just distorted with a scale by not knowing the camera intrinsics. Any other factors may contribute to the performance gap?\n> \n\nWe thank the reviewer for this question. Point maps provide explicit 3D world coordinates (X,Y,Z) while depth maps only provide distance from camera. This difference is crucial for manipulation tasks. Point maps enable the network to directly reason about object positions, orientations, and spatial relationships in world coordinates, while depth requires the network to implicitly learn the camera projection model. Beyond camera Intrinsics, the transformation also handles extrinsic parameters (camera pose), enabling fusion of multi-view point maps in a common reference frame - something not directly possible with depth maps.\n\n---\n\n> What is the implication for the extra requirement of knowing camera intrinsics? Would that be a limitation for utilising large-scale video action data without camera parameters?\n> \n\nWe thank the reviewer for raising this important practical consideration. Indeed, the camera parameter requirement limits applicability to large-scale uncalibrated datasets. However, point maps serve as an alternative to other 3D representations, which also require camera calibration, so this limitation is not unique to our approach. Additionally, recent foundational models like VGGT [1] can infer 3D attributes, including camera parameters and point maps, from just a few views. By leveraging such methods, we could generate point maps from any 2D dataset, opening possibilities for large-scale pretraining of point map encoders. We will add this promising direction to our future work discussion in the final version.\n\n---\n\n> The significance of xLSTM is not very clear in the comparison. …… Can additional ablation study be done to show the significance of the xLSTM design choice?\n> \n\nOur xLSTM choice is motivated by X-IL [2], which systematically compared xLSTM vs Transformers in multi-modal imitation learning, showing xLSTM's advantages for both RGB and point cloud inputs. However, we agree that additional ablation would strengthen our work since their comparison didn't include point map representations. We conducted controlled experiments comparing xLSTM vs Transformer-based PointMapPolicy using matched model sizes and identical training schemes on a subset of RoboCasa tasks:\n\n|  | PMP-xyz (Transformer) | PMP-xyz (xLSTM) |\n| --- | --- | --- |\n| TurnOnSinkFaucet | $50.7 \\scriptstyle{\\pm 8.1}$ | $\\mathbf{76.7 \\scriptstyle{\\pm 4.1}}$ |\n| TurnOffSinkFaucet | $52.7 \\scriptstyle{\\pm 10.5}$ | $\\mathbf{82.0 \\scriptstyle{\\pm 1.6}}$ |\n| TurnSinkSpout | $45.3 \\scriptstyle{\\pm 10.9}$ | $\\mathbf{76.0 \\scriptstyle{\\pm 1.6}}$ |\n| CoffeePressButton | $80.0 \\scriptstyle{\\pm 2.8}$ | $\\mathbf{82.7 \\scriptstyle{\\pm 5.0}}$ |\n| TurnOnMicrowave | $41.3 \\scriptstyle{\\pm 11.5}$ | $\\mathbf{49.3 \\scriptstyle{\\pm 5.0}}$ |\n| TurnOffMicrowave | $50.0 \\scriptstyle{\\pm 10.7}$ | $\\mathbf{70.0 \\scriptstyle{\\pm 4.9}}$ |\n| CoffeeServeMug | $56.0 \\scriptstyle{\\pm 5.9}$ | $\\mathbf{69.3 \\scriptstyle{\\pm 6.6}}$ |\n| CoffeeSetupMug | $14.7 \\scriptstyle{\\pm 3.4}$ | $\\mathbf{16.7 \\scriptstyle{\\pm 3.8}}$ |\n\nThe results demonstrate that xLSTM consistently outperforms Transformer-based policies when using point map representations, validating our architectural choice and suggesting that xLSTM's efficiency advantages extend to our structured 3D representation.\n\n---\n\nWe would like to thank the reviewer again and welcome the opportunity to address any additional concerns or questions that the reviewer may have.\n\n---\n\nReferences\n\n[1] Wang, Jianyuan, et al. \"Vggt: Visual geometry grounded transformer.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.\n\n[2] Jia, Xiaogang, et al. \"X-il: Exploring the design space of imitation learning policies.\" arXiv preprint arXiv:2502.12330 (2025)."}, {"review_id": "RxBR85syKs", "reviewer": "Reviewer_QAhT", "summary": "This paper proposes a multi-modal imitation learning method, namely PointMapPolicy. It uses point maps to organize raw point cloud data. The diffusion policy for action generation is integrated based on the EDM framework. The authors have tested their proposed method in both simulation and real-world environments. The main results demonstrate that the proposed PointMapPolicy outperforms the current 2D and 3D imitation learning baselines. The ablation studies also present the effectiveness of some devised components in the point-map-based scheme.", "questions": "The authors should address the concerns listed in the Weaknesses section. Here are some suggestions:\n\n* The authors should clarify their motivation more clearly by providing the weaknesses of the current schemes in the introduction.\n\n* Please consider modifying the existing diffusion-based imitation learning scheme to better incorporate point maps, rather than using the off-the-shelf ones.\n\n* The definition of the point map should be provided.\n\n* Please provide key ablation studies as mentioned in Major Weaknesses.\n\n* When the authors propose using some configurations such as the number of Transformer layers, please clarify the motivation or provide related ablations.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* This paper is easy to follow.\n* The citations are proper, and the related works are well reviewed.\n* Using point maps to organize point clouds for imitation learning is interesting.\n* The proposed method is compared with multiple baseline methods in both simulation and real-world setups, providing extensive experimental results.\n\nWeaknesses:\n\nMajor:\n\n* While the introduction provides an overview of existing schemes, it does not clearly include their limitations, particularly the down-sampling-based scheme and feature-lifting scheme. This obscures the gaps that the proposed point-map-based method aims to address in the context. In addition, Figure 1, as one of the most important figures in the paper, is not referenced in the main text. \n\n* The contributions of this work are limited. The only major contribution is using point maps as the input for imitation learning. The action diffusion module is off-the-shelf.\n\n* The current formulation of Equation (3) provides only how to get point maps. However, a rigorous mathematical definition of the point map should be explicitly specified, since it is the major contribution of this work.\n\n* In Table 2, PMP-xyz shows very poor performance compared to other baselines. Does this demonstrate that the whole PMP approach actually works mostly because of RGB input? The reviewer thinks that point maps struggle to capture useful geometric patterns since only using xyz cannot lead to good performance. \n\n* Key ablation studies are missing. The authors should replace the point map encoding with the current point-cloud-format encoding (e.g., down-sampling-based encoding) to demonstrate its superiority.\n\n* There is no analysis on some experimental results. For example, in Fusion of Images and Point Maps, the reason why Cat consistently emerges as the most effective fusion approach should be provided.\n\nMinor:\n\n* How to determine the number of denoising steps in PMP?\n* Why have the authors used Film-ResNet rather than CLIP or other backbones to encode RGB images?\n* Figure 3 omits the illustration of the concatenation operation.\n* Why do the authors use a 'four-layer' transformer for the Attn scheme?", "was_revised": true, "final_justification": "The authors have addressed all my concerns. I will raise my rating to borderline accept.", "rebuttal": "We thank the reviewer for taking the time to review our work and for the many helpful comments and suggestions. We hope the following replies address the questions and concerns raised. \n\n---\n\n> While the introduction provides an overview of existing schemes, it does not clearly include their limitations, particularly the down-sampling-based scheme and feature-lifting scheme. This obscures the gaps that the proposed point-map-based method aims to address in the context. In addition, Figure 1, as one of the most important figures in the paper, is not referenced in the main text. …… The authors should clarify their motivation more clearly by providing the weaknesses of the current schemes in the introduction.\n> \n\nWe thank the reviewer for the important feedback and apologize for the missing reference to Figure 1. Here we provide a clear motivation and will add it in the final version:\n\nCurrent 3D processing approaches face fundamental limitations that create a critical gap between 3D geometric information and existing 2D vision architectures (Figure 1). **Downsampling-based methods** (Figure 1a) [1] suffer from an inherent information-fidelity tradeoff: they must dramatically reduce point density through techniques like Furthest Point Sampling to remain computationally tractable, inevitably discarding fine-grained geometric details essential for precise manipulation tasks. **Feature-lifting approaches** (Figure 1b) [2] face equally problematic limitations as they aggregate 2D features through depth averaging and 3D transformations, introducing information loss while struggling to maintain spatial structure during the lifting process.\n\nOur point map approach (Figure 1c) directly addresses both limitations by preserving complete spatial information in a structured 2D format, enabling direct application of established vision architectures without sacrificing geometric fidelity or computational efficiency. This representation maintains natural correspondence between visual appearance and geometric structure, eliminating the traditional tradeoff between precision and tractability.\n\n---\n\n> The contributions of this work are limited. The only major contribution is using point maps as the input for imitation learning. The action diffusion module is off-the-shelf. …… Please consider modifying the existing diffusion-based imitation learning scheme to better incorporate point maps, rather than using the off-the-shelf ones.\n> \n\nWe thank the reviewer for this concern. We clarify that our work is the first to systematically validate point maps for end-to-end imitation learning using diffusion policies. Our contribution is introducing point maps as a representation that enables any policy framework to leverage 3D geometric information through standard vision encoders.\nJust as RGB images work with any vision architecture (ResNet, ViT, ConvNeXt), point maps can be directly processed by existing encoders without modification. This eliminates the need for specialized point cloud networks and enables seamless integration into any visual imitation learning framework. Our systematic validation demonstrates the effectiveness of this representation across diverse tasks and benchmarks, providing evidence that enables the broader community to adopt point maps in their preferred policy architectures. \n\n---\n\n> The current formulation of Equation (3) provides only how to get point maps. However, a rigorous mathematical definition of the point map should be explicitly specified, since it is the major contribution of this work.\n> \n\nWe really appreciate this important suggestion. We provide the definition of the Point Map and will add it to the final version of the paper:\n\nA Point Map $X \\in \\mathbb{R}^{W \\times H \\times 3}$ is a dense 2D field of 3D points that establishes a one-to-one mapping between image pixels and 3D scene points. For an RGB image $I$ of resolution $W \\times H$, the corresponding Point Map $X$ satisfies $I_{i,j} \\leftrightarrow X_{i,j}$ for all pixel coordinates $(i,j) \\in \\\\{1 \\ldots W\\\\} \\times \\\\{1 \\ldots H\\\\}$, where each pixel intensity $I_{i,j}$ corresponds to a 3D point $X_{i,j} \\in \\mathbb{R}^3$ in world coordinates. This representation assumes that each camera ray intersects a single 3D point, ignoring the case of translucent surfaces [3].\n\n---\n\n> In Table 2, PMP-xyz shows very poor performance compared to other baselines. Does this demonstrate that the whole PMP approach actually works mostly because of RGB input? The reviewer thinks that point maps struggle to capture useful geometric patterns since only using xyz cannot lead to good performance.\n> \n\nWe thank the reviewer for this important question. The performance gap between RGB and XYZ inputs on the CALVIN tasks is a task-dependent limitation, and not a limitation of point maps in general. As we discuss in lines 216-220, PMP-xyz's lower performance on CALVIN (2.03 vs RGB's 3.15) stems from CALVIN's heavy reliance on color information, with many instructions explicitly referencing colors (e.g., \"red block,\" \"pink block\"). Without color cues, the model cannot complete color-dependent tasks regardless of geometric precision.\n\nHowever, this task-dependency actually validates our approach: different modalities excel at different tasks. On geometry-heavy RoboCasa tasks, PMP-xyz (49.12%) significantly outperforms RGB (40.16%), while on color-dependent CALVIN tasks, RGB dominates. Most importantly, our multi-modal PMP achieves the best of both worlds - outperforming RGB on both benchmarks (CALVIN: 4.01 vs 3.15; RoboCasa: 47.22% vs 40.16%) by adaptively leveraging the most relevant information for each task.\n\nThis demonstrates the core value of our approach: complementary modalities that together provide more robust performance across diverse manipulation scenarios.\n\n---\n\n> Key ablation studies are missing. The authors should replace the point map encoding with the current point-cloud-format encoding (e.g., down-sampling-based encoding) to demonstrate its superiority. …… Please provide key ablation studies as mentioned in Major Weaknesses.\n> \n\nWe appreciate the reviewer’s valuable suggestions. In fact, Reviewer Dr9P raised similar concerns about separating input modality effects from policy backbone influences. We conducted extensive controlled ablations comparing different point cloud processing methods using identical xLSTM backbones. These results demonstrate that our improvements stem from the point map representation itself, not architectural choices. **We kindly refer the reviewer to our detailed response to Reviewer Dr9P for the complete ablation study.**\n\n---\n\n> There is no analysis on some experimental results. For example, in Fusion of Images and Point Maps, the reason why Cat consistently emerges as the most effective fusion approach should be provided.\n> \n\nCat fusion performs well because it preserves all information from both modalities without loss by directly concatenating all tokens as input to the diffusion policy, avoiding any intermediate processing that could discard useful features.\n\n---\n\n> How to determine the number of denoising steps in PMP?\n> \n\nWe follow the established practice from diffusion policy literature [4] and use 4 denoising steps, which provides a good balance between generation quality and computational efficiency.\n\n---\n\n> Why have the authors used Film-ResNet rather than CLIP or other backbones to encode RGB images?\n> \n\nFiLM-ResNet is a standard choice for robotic vision tasks due to its effective conditioning mechanisms and has been established in prior work including RT-1 [5], MT-ACT [6], and MoDE [7].\n\n---\n\n> Figure 3 omits the illustration of the concatenation operation.\n> \n\nWe will add concatenation illustration for clarity in the final version.\n\n---\n\n> Why do the authors use a 'four-layer' transformer for the Attn scheme? …… When the authors propose using some configurations such as the number of Transformer layers, please clarify the motivation or provide related ablations.\n> \n\nWe additionally provide ablation results on different transformer layer counts (4, 6, 8) for the Attn fusion scheme across three RoboCasa task categories. The results below show that increasing the number of attention layers does not improve performance.\n\n|  | Drawers | Buttons | Insertion |\n| --- | --- | --- | --- |\n| Attn-4 | $72.7$ | $75.3$ | $34.0$ |\n| Attn-6 | $69.6$ | $72.9$ | $29.0$ |\n| Attn-8 | $68.0$ | $72.0$ | $30.0$ |\n\n---\n\nWe sincerely thank the reviewer for providing specific and constructive suggestions to strengthen our work, and hope that our responses have successfully addressed the questions and concerns raised. Should the reviewer have any additional questions or further concerns, we would be more than happy to provide further clarifications or answers.\n\n---\n\nReferences\n\n[1] Ze, Yanjie, et al. \"3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.\" RSS 2024\n\n[2] Ke, Tsung-Wei, Nikolaos Gkanatsios, and Katerina Fragkiadaki. \"3d diffuser actor: Policy diffusion with 3d scene representations.\" CoRL 2024\n\n[3] Wang, Shuzhe, et al. \"Dust3r: Geometric 3d vision made easy.\" CVPR 2024\n\n[4] Reuss, Moritz, et al. \"Goal-conditioned imitation learning using score-based diffusion policies.\" RSS 2023.\n\n[5] Brohan, Anthony, et al. \"Rt-1: Robotics transformer for real-world control at scale.\" RSS 2023.\n\n[6] Bharadhwaj, Homanga, et al. \"Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking.\" ICRA 2024.\n\n[7] Reuss, Moritz, et al. \"Efficient diffusion transformer policies with mixture of expert denoisers for multitask learning.\" ICLR 2025"}]

Author rebuttal:

Meta-review: The paper introduces a framework for imitation learning that integrates both RGB and depth information through point maps. The authors argue that point maps offer the advantage of a grid-like structure, allowing them to be processed with standard image backbones. Experiments on RoboCasa and CALVIN demonstrate strong performance compared to recent baselines. Reviewers acknowledged the paper’s contribution to advancing 3D policy learning, while noting the lack of direct, “apples-to-apples” comparisons with point cloud-based methods. These concerns were addressed in the rebuttal and subsequent discussion.
Decision: Accept (poster)