Title: Object-centric 3D Motion Field for Robot Learning from Human Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning from videos; Robot Learning; 3D Perception
Abstract: Learning robot control policies from human videos is a promising direction for scaling up robot learning. However, how to extract action knowledge (or action representations) from videos for policy learning remains a key challenge. Existing action representations such as video frames, pixelflow, and pointcloud flow have inherent limitations such as modeling complexity or loss of information. In this paper, we propose to use object-centric 3D motion field to represent actions for robot learning from human videos, and present a novel framework for extracting this representation from videos for zero-shot control. We introduce two novel components. First, a novel training pipeline for training a ``denoising'' 3D motion field estimator to extract fine object 3D motions from human videos with noisy depth robustly. Second, a dense object-centric 3D motion field prediction architecture that favors both cross-embodiment transfer and policy generalization to background. We evaluate the system in real world setups. Experiments show that our method reduces 3D motion estimation error by over 50% compared to the latest method, achieve 55% average success rate in diverse tasks where prior approaches fail ($\lesssim 10$\%), and can even acquire fine-grained manipulation skills like insertion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Summary\n\nThis paper introduces a novel approach for robot learning from human videos, leveraging an object-centric 3D motion field to represent actions. The core of their framework is a 3D Motion Field Estimator, which is initially trained using simulated data to refine noisy motion fields. This estimator is then applied to improve the quality of human manipulation data, which is subsequently used to train a robot policy.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* The paper is well-structured, making it easy to follow with a clear and logical flow.\n* The proposed method is simple, direct, and easy to understand. Its workflow demonstrates a rigorous and effective engineering design.\n\n---\n### Weaknesses\n\n* The paper's method bears a resemblance to previous approaches, and some of its novel capabilities appear to stem from integrating existing models. I'll elaborate on this specific point in the \"Questions\" section.\n\nQUESTIONS:\nBroadly speaking, the core idea of this paper—predicting the **point cloud flow (or motion)** of manipulated objects and then converting that into robot control signals—bears a strong resemblance to prior work like Track2Act and GFlow.\n\nWhile the paper undeniably presents a more refined workflow, incorporating elements such as dataset augmentation, deployment-time optimization, and leveraging SAM2 for object-centric understanding, it feels more like a highly engineered and detailed implementation of existing concepts rather than a breakthrough in innovation or a source of new insights. For instance, some of the \"new\" capabilities, such as **distractor generalization**, seem to stem from the integration of SAM2, and **depth robustness** appears to be a benefit of introducing the **3D Motion Estimator**.\n\nTherefore, I'd like the authors to elaborate further on the **innovative aspects and core insights** of your proposed method. Specifically:\n\n* Are there any capabilities that **only the proposed method can achieve** and that previous methods simply cannot?\n* Or is the proposed method primarily a **better-designed, more deployment-ready version** of existing approaches?\n\nI'm hoping for a discussion that highlights the unique contributions and fundamental advancements that set this work apart, beyond just improved engineering and integration of other state-of-the-art components.\n\nLIMITATIONS:\nPlease see Questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates that the motion field is key to learning from human videos for robotic policy learning. It first analyzes the drawbacks of current action representation and points out that the proposed object-centric motion field has clear advantages. Then, it argues that the learning objective from the human video is object 3D motions and demonstrates the necessity of using RGBD video. After that, it raises the challenges when learning 3D motions from videos, which is the accumulated noise. It finally designs the 3D motion field estimator to reduce noise, which makes it possible to train video policy models from the accurate motion fields. It also discusses the deployment issues. Experiments on the real-world setups demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n- The writing is quite clear and fluent. The motivation of the proposed method is well established and discussed.\n- The proposed model that learns motion field from video is interesting and has clear advantages compared to previous learning strategies.\n- Experiments show that the proposed learning strategy performs well in real-world scenarios.\n\n## Weakness\n\n- In Phase 2, the model only takes object information as input, which is pre-processed by SAM2. Is it suitable for real-world settings? We may always have to manually instruct SAM2 to obtain the segmentation of the target object.\n\n- I found that when training the policy model in Phrase 2, it only uses RGBD image to infer the 3D motion field. This confuses me because object motion in one image can correspond to multiple potential motion fields. How can the predicted motion field be ensured is the desired motion field? For example, given a bottle image, the policy model could predict arbitrary motion trajectories of that bottle. It seems that it cannot control the predicted motions. Do I miss something?\n\nQUESTIONS:\nThe paper is well-written and acceptable. The motivation is clear, the proposed method is effective, and the discussion of each part of the proposed model is adequate. I only have a few concerns (see above).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an object-centric 3D motion field as the action representation for learning robot control policies from human videos. The motion field is defined as a 4-channel image, where the first channel is depth and the next three channels are 3D movements of each pixel from the current image frame to the next. The paper proposes a two-phase learning framework. In phase 1, a denoising 3D motion field estimator (with extra intrinsic map features as input) is pretrained in simulated environments. In phase 2, object instances are segmented (SAM2) from human videos. A policy network is then trained on lifted (CoTracker3) and denoised (phase 1) 3D motion field extracted from human videos. The proposed 3D motion field-based policy is evaluated on 5 real-world scenarios, in which *Tool Use II: Wrench* and *Insertion* are difficult manipulation tasks. The experiment results demonstrate that the proposed method succeeds even when no robot demonstrations are provided, and the evaluation performance is better than the baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper proposes 3D motion field. This representation is aligned to observation frames (unlike pointcloud tracking methods) and can be integrated with input image observations.\n2. This paper introduces the intrinsic map as an extra feature input to the model, and the authors explain the motivation and insights for introducing it.\n3. This paper is clearly written.\n\nWeaknesses:\n1. (Motion field movement difference) The proposed motion field covers the 3D movement of each pixel between the current frame and the next frame. However, the amount of the movement is determined by the motion speed of the human/robot between the frames. Typically, humans move the object at a much faster speed, leading to a larger movement between the frames. The paper does not provide the explanation of how the 3D Motion Field addresses the differences in 3D motion fields arising from the speed differences between humans and robots when performing tasks.\n2. (Motion field frame interval) In addition to 1., the exact value of the frame interval between the current frame and the next frame used in the extraction of 3D motion fields is not directly provided. There is information on the fps of the camera used in deployment, but it is unclear whether that fps aligns with 3D motion field extraction.\n3. (Duplicated input channel) In phase 2, the input contains both the 3D motion field and the segmented RGBD. As the 3D motion field already contains the depth information, the depth channel in the input seems to be duplicated. The paper provides no explanation or analysis of this potentially redundant input format.\n\nQUESTIONS:\n1. (Weakness 1) How to address the 3D motion field difference (the pixels' 3D movement part) caused by the speed difference between humans and robots?\n2. (Weakness 2) What is the exact frame interval used to compute the 3D movement used in the extraction of 3D motion field?\n3. (Weakness 3) Is the depth channel duplicated in the phase 2 model's input? If yes, why is this redundancy introduced?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes an object-centric pipeline for learning robot policies from human videos. One contribution of this paper is a fine object 3D motion estimator, and the other contribution is the prediction algorithm of object-centric 3D motion (the scene after the target is manipulated). The algorithm mostly operates in 3D, with compact usage of classic techniques such as ICP and RANSAC. Overall, this is a good work. Please see my comments below.

STRENGTHS AND WEAKNESSES:
1. As the authors do not opensource the model, can the human video lists be available? Otherwise the results are not reproducible. 

2. The paper does not mention how the manipulator is controlled. Through inverse kinematics? 

3. Did the authors test the algorithm in complex background scenes? The paper says that the sensor is mounted 40cm~50cm away from the target. Therefore, are the figures in Fig.7 cropped images? not the raw data? 

4. I assume that the hole structure shown in Fig.7 is fixed on the table.

QUESTIONS:
please see my comments above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03