Title: PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Imitation Learning

Point Cloud

Robot Manipulation
Abstract: Robotic manipulation systems benefit from complementary sensing modalities, where each provides unique environmental information.
Point clouds capture detailed geometric structure, while RGB images provide rich semantic context. Current point cloud methods struggle to capture fine-grained detail, especially for complex tasks, which RGB methods lack geometric awareness, which hinders their precision and generalization. We introduce PointMapPolicy, a novel approach that conditions diffusion policies on structured grids of points without downsampling. The resulting data type makes it easier to extract shape and spatial relationships from observations, and can be transformed between reference frames. Yet due to their structure in a regular grid, we enable the use of established computer vision techniques directly to 3D data. Using xLSTM as a backbone, our model efficiently fuses the point maps with RGB data for enhanced multi-modal perception.
Through extensive experiments on the RoboCasa and CALVIN benchmarks and real robot evaluations, we demonstrate that our method achieves state-of-the-art performance across diverse manipulation tasks. The overview and demos are available on our project page: https://point-map.github.io/Point-Map/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multi-modal imitation learning method, namely PointMapPolicy. It uses point maps to organize raw point cloud data. The diffusion policy for action generation is integrated based on the EDM framework. The authors have tested their proposed method in both simulation and real-world environments. The main results demonstrate that the proposed PointMapPolicy outperforms the current 2D and 3D imitation learning baselines. The ablation studies also present the effectiveness of some devised components in the point-map-based scheme.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* This paper is easy to follow.\n* The citations are proper, and the related works are well reviewed.\n* Using point maps to organize point clouds for imitation learning is interesting.\n* The proposed method is compared with multiple baseline methods in both simulation and real-world setups, providing extensive experimental results.\n\nWeaknesses:\n\nMajor:\n\n* While the introduction provides an overview of existing schemes, it does not clearly include their limitations, particularly the down-sampling-based scheme and feature-lifting scheme. This obscures the gaps that the proposed point-map-based method aims to address in the context. In addition, Figure 1, as one of the most important figures in the paper, is not referenced in the main text. \n\n* The contributions of this work are limited. The only major contribution is using point maps as the input for imitation learning. The action diffusion module is off-the-shelf.\n\n* The current formulation of Equation (3) provides only how to get point maps. However, a rigorous mathematical definition of the point map should be explicitly specified, since it is the major contribution of this work.\n\n* In Table 2, PMP-xyz shows very poor performance compared to other baselines. Does this demonstrate that the whole PMP approach actually works mostly because of RGB input? The reviewer thinks that point maps struggle to capture useful geometric patterns since only using xyz cannot lead to good performance. \n\n* Key ablation studies are missing. The authors should replace the point map encoding with the current point-cloud-format encoding (e.g., down-sampling-based encoding) to demonstrate its superiority.\n\n* There is no analysis on some experimental results. For example, in Fusion of Images and Point Maps, the reason why Cat consistently emerges as the most effective fusion approach should be provided.\n\nMinor:\n\n* How to determine the number of denoising steps in PMP?\n* Why have the authors used Film-ResNet rather than CLIP or other backbones to encode RGB images?\n* Figure 3 omits the illustration of the concatenation operation.\n* Why do the authors use a 'four-layer' transformer for the Attn scheme?\n\nQUESTIONS:\nThe authors should address the concerns listed in the Weaknesses section. Here are some suggestions:\n\n* The authors should clarify their motivation more clearly by providing the weaknesses of the current schemes in the introduction.\n\n* Please consider modifying the existing diffusion-based imitation learning scheme to better incorporate point maps, rather than using the off-the-shelf ones.\n\n* The definition of the point map should be provided.\n\n* Please provide key ablation studies as mentioned in Major Weaknesses.\n\n* When the authors propose using some configurations such as the number of Transformer layers, please clarify the motivation or provide related ablations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to augment vision-based policies with 3D information to improve the motion accuracy and hence the task success rate. Unlike previous works exploring point clouds or depth-image, the paper suggests using point map, converted from depth image via known camera intrinsic parameters. The main benefit is the possibility of using a similar image encoder and position embeddings as the RGB modality, in contrast to point cloud data. The approach explores different ways of fusing point map and RGB encodings, identifying the benefits of concatenation. The design is utilised in training a language prompted diffusion policy, leveraging xLSTM for light-weight encoding and training of the history. The approach is compared to several baselines with/without input carrying 3D information on two simulation benchmarks and real robot experiments. The results show improvement on the task success rate and the length of following the language instructions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Simple yet effective design to fuse 3D information in vision-policy training.\n* The motivation for having a unified visual encoding architecture for capturing 3D information is reasonable.\n* Extensive studies with multiple baselines and benchmarks, including real-robot experiments.\n* The design has the potential to augment multiple views/sensors input, for the ease of manipulating image encodings.\n\nWeakness:\n* Though the empirical performance is positive to the main hypothesis, more insightful discussion or studies on why point map is favoured, especially comparing to depth image, is lacking.\n* The significance of xLSTM is not very clear in the comparison.\n\nQUESTIONS:\n* Can more insights be shed on why point map brings more performance gains comparing to other 3D especially depth map? It seems like depth map information is just distorted with a scale by not knowing the camera intrinsics. Any other factors may contribute to the performance gap?\n* What is the implication for the extra requirement of knowing camera intrinsics? Would that be a limitation for utilising large-scale video action data without camera parameters?\n* Can additional ablation study be done to show the significance of the xLSTM design choice?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PointMapPolicy for diffusion-based imitation learning in robotic manipulation. Its core contribution is to use a Point Map observation that projects the XYZ coordinates of a point cloud back onto the image plane, supplying rich geometric cues for policy learning. Extensive experiments in both simulation and the real world show that PointMapPolicy outperforms existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper tackles input-modality selection in robotic manipulation, a very important problem.\n2. The proposed point-map representation can be easily adopted by different policy-learning frameworks.\n3. I appreciate the experimental videos on the website, especially those testing diverse initial configurations.\n\nWeakness\n1. The technical contribution appears incremental. Point Maps are an existing concept [10, 11], and prior work has used similar inputs in manipulation. E.g., RVT uses a similar input format: ```Specifically, for each view, we render three image maps with a total of 7 channels: (1) RGB (3 channels), (2) depth (1 channel), and (3) (x,y,z) coordinates of the points in the world frame (3 channels).```\n2. The goal is to advocate Point Maps as an input modality, but the experimental justification is not rigorous. Specifically, Table 1 mixes input modality with backbone architecture, where PMP uses an X-Block, while DP3 (UNet) and 3DA (Transformer) use different policy backbone, making it unclear whether the improvement come from the modality or the policy backbone. \n3. Some alternative point-cloud approaches, such as the 3D-Lifting method in Figure 1, are also missing.\n\nQUESTIONS:\n1. As mentioned in Weakness 2, separating the evaluation of the input modality and the policy backbone would strengthen the paper. Although Table 1 shows that the proposed method is outperforming DP3 and 3DA, is it because Point Map is a better input modality, or because the X-Block is better than the UNet used in DP3 and the transformer used in 3DA? Adding an additional baseline that uses a point cloud encoder + X-Block variation would be beneficial.\n2. Moreover, another baseline that uses 3D Lifting + X-Block will also better demonstrate the advantages of Point Map, particularly because 3D Lifting is listed as a comparison in Figure 1.\n3. A clearer contribution summary would be useful. How is the proposed method different from other robot learning approaches that use a similar Point Map concept, like RVT?\n4. Why do the authors list their model as no pre-training in Table 2? Figure 2 states `PMP integrates multiple modalities: language instructions encoded by a pretrained CLIP model, images processed by pretrained visual encoders, and point maps processed by visual encoders trained from scratch.`. Line 198 seems to argue that the proposed method does not use `robot-specific pretraining`, but it is important to clarify what is the difference here, especially because the proposed method underperforms the pretrained baselines in Table 2.\n5. The overall writing could be polished; for instance, the abstract does not read very smoothly.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PointMapPolicy (PMP), a multi-modal fusion framework for imitation learning that integrates both RGB images and point maps. The authors compare different multi-modal fusion methods, including early fusion, late fusion, and attention-based fusion. Simulation and real-world experiments show that the proposed method outperforms policies trained using only RGB or Depth modalities.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is clearly written and well-structured, making it easy to follow.\n2. The paper studies an underexplored area in robot manipulation: policy learning from 3D input. \n3. The paper provides a thorough comparison across RGB, depth map, and point map input modalities. The paper highlights the benefits of combining them.\n\n\nWeakness:\n1. The visual encoder and the backbone in the proposed model are trained from scratch, while prior work shows that pretrained vision encoders can significantly boost performance. The authors should clarify their decision not to use pretrained encoders or pretrained backbones and discuss how existing pretrained models could be adapted to handle additional modalities like point maps. \n2. The ablation study compares different visual encoder choices. However, the performance of imitation learning also depends on the backbone choice. It is unclear how much the performance gain is from multi-modal fusion itself versus the choice of backbone. More controlled experiments are needed to isolate the contribution of fusion.\n3. The introduction states that point clouds struggle to capture fine-grained detail, motivating the use of point maps. However, this claim would be more convincing with a quantitative comparison between point clouds and point maps, considering the availability of convolutional networks like KPConv that process unstructured point clouds effectively. \n4. The related work could be strengthened by discussing more recent approaches in multi-modal policy learning. In particular, the authors should consider comparing with:\nZhu, Yichen, Zhicai Ou, Feifei Feng, and Jian Tang. \"Any2Policy: Learning Visuomotor Policy with Any-Modality.\" Advances in Neural Information Processing Systems 37 (2024): 133518-133540.\n\nQUESTIONS:\nHow does the proposed point map method compare to the point cloud method? The paper only compares to point cloud-based methods that have different backbones.\n\nLIMITATIONS:\nHow to maintain performance when one modality is performing well and one is not? For example, the two modalities have large success rate differences for the TurnSinkSpout task in RoboCasa, resulting in PMP performing worse than using either modality. The authors should discuss these kinds of edge cases and describe them in the limitations section. Additionally, Figure 5 doesn’t show a lot of difference in the different fusion methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}