Object-centric 3D Motion Field for Robot Learning from Human Videos
Authors: Zhao-Heng Yin
Sherry Yang
Pieter Abbeel
Venue: NeurIPS 2025 spotlight ()
Keywords: Learning from videos; Robot Learning; 3D Perception
Primary area: applications

Abstract: Learning robot control policies from human videos is a promising direction for scaling up robot learning. However, how to extract action knowledge (or action representations) from videos for policy learning remains a key challenge. Existing action representations such as video frames, pixelflow, and pointcloud flow have inherent limitations such as modeling complexity or loss of information. In this paper, we propose to use object-centric 3D motion field to represent actions for robot learning from human videos, and present a novel framework for extracting this representation from videos for zero-shot control. We introduce two novel components. First, a novel training pipeline for training a ``denoising'' 3D motion field estimator to extract fine object 3D motions from human videos with noisy depth robustly. Second, a dense object-centric 3D motion field prediction architecture that favors both cross-embodiment transfer and policy generalization to background. We evaluate the system in real world setups. Experiments show that our method reduces 3D motion estimation error by over 50% compared to the latest method, achieve 55% average success rate in diverse tasks where prior approaches fail ($\lesssim 10$\%), and can even acquire fine-grained manipulation skills like insertion.

Object-centric 3D Motion Field for Robot Learning from Human Videos
Zhao-Heng Yin
Sherry Yangl.2 Pieter Abbeel
BAIR, UC Berkeley BECS Google DeepMind
https://2haaheengiin.g git.hg.....b..........
Abstract
Learning robot control policies from human videos is promising direction for scaling up robot learning. However, how to extract action knowledge (or action representations) from videos for policy learning remains key challenge. Existing action representations such video frames, pixelflow, and pointcloud flow have inherent limitations such as modeling complexity or loss of information. In this paper, propose to object-centric 3D motion field to represent actions for robot learning from human videos, and present a novel framework for extracting this representation from videos for zero-shot control. We introduce two novel components in its implementation. First, a novel training pipeline for training "denoising" 3D motion field estimator to extract fine object 3D motions from human videos with noisy depth robustly. Second, dense object-centric 3D mo- tion field prediction architecture that favors both cross-embodiment transfer and policy generalization to background. We evaluate the system real world setups. Experiments show that method reduces 3D motion estimation error by over 50% compared to the latest method, achieve 55% average success rate diverse tasks where prior approaches fail (S 10%), and can even acquire fine-grained manipulation skills like insertion.
Train
Human Videos
Robot Skills
We propose a novel framework for robot learning from human demonstration videos without relying on any robot-collected data. Our approach learns control robots by extracting and modeling 3D object motion fields from RGBD human videos.
Figure 1:
1 Introduction
Data is the primary bottleneck in robot learning collecting large-scale high quality robotic data in real world at scale for training control policies not only expensive but also mentally challenging for humans in complex tasks 55). Recently, human-object interaction videos stand out as particularly promising avenue to overcome this challenge. These videos are not only scalable -iven the vast
Correspondence to: zhachengyin@cs berkeley.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Noisy
(Simulation Pretraining)
oz)
3D Pizel
Flow
3D Motion Field Estimator
Z
dx
Depth
3D Motion Fiekd
Carrent
dz
Learning to See 3D Motion Field
3D Motion Field Representation
[Extractio)
3D Poel Flow
3D Motion Field
Observation
Estimator
Dipth
3D Motion Field Predictor
Action (SE(3))
Motion Field
Esgmerted A560
3D Motion Field
Label
Labeled Videns
II-A. Seeing 3D Motion Field Videos
BI-B. Predicting 3D Motion Field for Control
Figure 2: Overview proposed learning framework. We first pretrain a 3D motion field estimator in simulation (Phase I) and use it estimate the 3D motion field in noisy RGBD human videos (Phase II-A). Then, we train policies to predict the estimated 3D motion field and use them to control robots zero-shot manner (Phase II-B). Unlike existing 3D tracking works that assume depth groundtruth reference, we recover accurate 3D object motion from noisy depth.
amount of footage available from internet or wearable device recordings but they also capture rich, naturalistic demonstrations of complex tasks. Moreover, collecting data through human hands is inherently easier, cheaper, and more intuitive than through robotic teleoperation
[4
Due to this data collection challenge, many works look into the feasibility of using real-world action- free videos for robot learning. Some recent works, such as UniPi [ and UniSim (54). directly apply video prediction for control, which essentially view future video frames as action representations and use inverse dynamics model translate the predicted future frame actions. Although this line of work achieved some preliminary success, video frames turn out to be an overly noisy, redundant action representation, which not only unnecessarily complicates training and inference but also makes policy non-robust due to blurry details in videos. Therefore, recent works also explored more compact action representations extracted from videos, such pixel-flow 49 52 B) point-cloud flow (59), and SE(3) pose transformation
[15]. Nevertheless, each of these representations suffers from certain drawbacks. Pixel-flow is 2D representation and drops the important 3D movement information for recovering actions. Point-cloud 3D flow is noisy and cannot represent motion accurately. SE(3) pose extraction relics on object 3D models and limits themselves to rigid bodics. We refer readers to the related works (Section 6) for a complete discussion and comparisons. In conclusion, far, it remains unclear what serves as a good and feasible action representation for video-based robot learning.
In this work, we propose to use an object-centric 3D motion field representation the action representation for control. Specifically, a dense position and motion field over the object pixels in the input image, representing how each observable point on an object should move in each task (Figure 2}. This action representation has many advantages compared to previous works: preserves minimal sufficient 3D information for robot control; 2. It is image-based representation and allows the use of well-studied powerful image generative models; beboodiment-agnostie simplifying cross embodiment transfer; 4. Its extraction fully depends RGBD video and does not assume additional information like object 3D models.
object-centric and
While it may sound appealing, extracting such information from video turns out to be challenging. Although it relatively simple to extract accurate 2D pixel flow movement part using the latest tracking model, the depth channel is full of noisy measurement values (e.g. missing wrong values) and directly using the raw depth values with for computing 3D motion will only result in inaccurate motion. Our key insight is that we can build "denoising" 3D motion field estimator to reconstruct 3D position and motion flow robustly from the noisy depth measurement with simulation data, as depth noise has some easy-to timmullte characteristics. Since this task is fully geometrical and does

---

not involve complex RGB textures, this simulation trriined-ddeddd estimator can transfer to real world well. Utilizing this, we can train control policies to predict reconstructed high-quality 3D motion flow, which is then translated to robot actions in downstream tasks for control. We evaluate our proposed components and system in several real world object manipulation and tool-use tasks. Our system reduces 3D motion field estimation error by over 50% compared to latest works and achieves ~55% zero-shot success rate on average where prior approaches fall short (S 10%). Remarkably, we also show that the policy trained solely on human (hand) videos is capable of performing fine-grained manipulations, such as precise insertions - a level of skill not previously shown in this setting.
In summary, our main contributions are follows. 1). We propose to use object-centric 3D motion field for robot learning from videos and present a novel learning framework for extracting this representation for control. 2). We present a simple and novel architecture that can learn to see and predict object-centric 3D motion field in the real world for control. With this, we can teach robots new skills with human videos the only training data. 3). We validate our proposed components in the real world. We demonstrate that our motion extraction pipeline can significantly reduce motion estimation error by over 50%. For its roboties application, approaches and show that our policies trained solely on human videos can achieve finegrained manipulation skill for the first time, demonstrating the potential of video-based robot learning.
significantly outperformed existing
2
Preliminaries
2.1 What Can Be Learned from Videos?
We begin by discussing what knowledge can be learned from videos and why focus on extracting and learning object movements. Recently, some works propose to extract low-level human finger
Although this might be feasible approach if robot embodiment is highly similar to human hand, we argue that learning direct embodiment control is both unnecessary and challenging. On one hand, the ultimate knowledge we want to learn is how each object in the view should move in each task, and it does not matter too much how we control a robot to generate such object movement as long as we can achieve il The state-of-ther of-thhe-rtt robots have built-in functionality to realize arbitrary object movements: i.e., by calling well-established foundational grasping policy and task-space SE(3) movement commands, and therefore we should focus on extracting object motion. On the other hand, generating reliable action through naive retargeting is hard (5 24]. In some cases, the movement performed by a human hand might not be realizable by robot (e.g. when robot has different or fewer fingers) and action retargeting will be undefined. Actually, existing methods typically require the human hand to act as a gripper and avoid in-hand manipulations in the training videos. This is not natural for human at work and introduces extra burdens. We conclude with the following assumption and proposal.
movement from videos and aim at directly retargeting them to robots [51 36
32
20 21
Assumption 1 attaching/detachinn)) policy. This is reasonable assumption given success in previous work on robot grippers or hands 61 (66.. Different robots should have their specific grasping policy.
We assume we have access to a reliable robot grasping/ungrasping (i.e. object
Proposal We propose to focus on extracting and learning object 3D motions for robot learning from videos. Combining an object flow prediction model with the assumed grasping policies above, a robot can solve a wide range of tasks.
2.2 Necessity of Depth Perception
To extract the object 3D motion from video effectively, we assume RGBD videos as training data. While there is growing interest in leveraging RGB-only information for action extraction, we argue that incorporating depth information is necessary for accurate estimation and learning. Otherwise, there exist infinitely many 3D transformations to realize observed pixel motion. Even if we assume some priors over underlying 3D movement, this is still extremely challenging and brittle. For example, suppose that we have a point (X, 0, Z) in the camera frame and it translating along z-axis. Our goal is to recover this small a-ax motion AZ from pixels. Assuming a pinhole camera with focal fX 7 AZ, and we
length f, the observed pixel movement on r-axxs is At, z+47
fX
fX
%
R
However, we may require some "functional knowledge" as constraints: e.g. do not hold the sides of a cup/ should grasp the tool handle. Therefore in the long run, besides object motion, we also need to consider extracting contact (semantical affordance).
3

---

z2 Aup. - rearrange it as AZ = 7x TxAAp.
The problem is that Az, might be noisy in practice and it can
lead to huge estimation error in AZ due to the large slope ffx especially near X (i.e. when the camera is looking at the object), and even 1-pixel error can result in over I centimeter difference! This error disastrous for robot manipulation task requiring high accuracy to overcome this we need robust subpixel-level trackers. Although this might be possible in the future, having RGBD video instead of RGB video can make learning much easier. Therefore:
Assumption We require access RGBD videos collected through pinhole cameras with known camera intrinsics (i.e. camera focal length fz: fy and center Ca, Cy), instead of general RGB videos collected by unknown camera with distortions. Although it might seem like a waste not reuse RGB videos already available on the internet, accumulating new RGBD videos can be easier than one may think of Million hours of videos are being produced every day [58] and many existing daily camera devices are already equipped with depth sensing (e.g. latest mobile phones and wearable devices).
2.3 3D Motion Field
As we have discussed, our goal is to extract an object-centric 3D motion field from RGBD videos for control. In this paper, 3D motion field 42 23 40 a dense, image-based representation defined as follows.
Definition (3D Motion Field) Given a pair of H x W images To, J (current frame and next frame), the 3D motion field F of To is defined as 4-channel image tensor in TI xx The first channel :l..;,0, denoted as Fdeyet, is the depth value of each pixel in lo. The remaining three channels F[:,;, 4], denoted as Pmooionn represent the underlying 3D movement of each pixel between the 2 frames under the lo camera frame.
Note that we include depth in this definition. Then, given the camera intrinsics and the 3D motion field, we can reconstruct the position and movement of every pixel in the 3D space.
3 Phase I: Seeing 3D Motion Field in Noise
Our first step is to extract accurate 3D motion fields from noisy RGBD videos. We first discuss a very simple pipeline for this purpose as suggested by latest works [59] and its fundamental limitations, and then we introduce our improved approach. The discussion below assumes the depth observation of two consecutive images Lo and I1, and dense pixel correspondence computed by a video tracker.
3.1 Direct Approach
We first assigns the camera depth to the Edepth Fl:.: :, depth channel. Then, we run a pixel tracker (e.g. Cotracker (17) to decide the pixel correspondence between Tu and JT. Let a pixel (x, y) in lo correspond to (a's s/) in It. Since we have camera intrinsics and their depth values Z and Z', we apply camera inverse projection to get their 3D coordinates (X, Y, Z) and (X', Y', Z"), and set F Fmotion = Fly, x, 1 4] = (X', Y', Z') (X, Y, Z), i.e., the 3D space movement. The procedure above assumes a static camera but also applies to moving camera (transform (X', Y', Z') to lo coordinate frame first).
The direct method can be effective if we have perfect depth and perfect tracker, but unfortunately, this is untrue in practice. First, the commonly used depth cameras are sensitive to lighting and partieularly erroneous when it comes to moving objects there are numerous holes (missing values) and white noises across the depth image. Second, even if we might have high quality learning- based binocular depth sensing method to improve the depth accuracy, noises from the pixel tracker can also lead to significant errors in motion. For instance, if the pixel tracker mistakenly maps foreground object pixel Io to background pixel or a hidden pixel the next frame 7, (which is common for object boundary pixels), X' and Y' as X' = Z'''/ fa and Y' = Z'y' /fy In both cases, we will obtain a wrong motion field. Due to this, prior works typically use intensive heuristics-basee filtering to reduce outliers in observations. Nevertheless, heuristics-base problem thoroughly. This will result in noisy motion field representation and make the subsequent prediction a hard problem.
will obtain incorrect depth Z' and consequently wrong
filtering can be unreliable, hence not eliminating the
3.2 Our Improved Approach: Learning to See 3D Motion Field
To remedy this issue, we propose to learn a 3D motion field estimator to reconstruct the groundtruth smooth 3D motion field with the noisy sensor measurements. As shown in Figure 2 and the

---

Motion
Hitt Point
Image
t-a
Projection
Plane
Source
Edge Noiso
Ray Casting
Flow
Missing Value
Generation
Object Samples
Data Augmentations
Figure 3: (Left) Phase Synthetic Data Generation. We randomly generate object and 3D motions, and use ray casting and projection to obtain 3D pixel flow input and the 3D motion field label. (Middle) Random Objects for Data Generation. (Right) Depth Samples and Data Augmentations.
(Naisy Sensors)
(Diffusion)
3D Motion Field
Timestap
concal
Depth
concal
Dopth
Phase II
(Diffusion)
UNet Blocks
Segmented RGBD
3D Motion Field
Noisy
Map
Intrinsics
Motion
Input
Camera Intrinsics
Dual Head UNet
Output
Figure 4: Model Architecture. The most important design is dense intrinsics map feature concate- nated to the input, which contains key information for reconstructing the groundtruth 3D flow. Note that Phase I and II train separate models: Phase uses noisy object depth and 3D pixel flow imput and Phase uses segmented RGBD and a noisy 3D motion field (in the case of Diffusion Model). For Phase I, also validity mask value channel indicate which feature values defined.
main inputs this model are noisy dense object 3D pixel flow (2D pixel flow (æ,, y-axis) and depth flow (z-axis)) and the noisy depth image. The output is the reconstructed 3D motion field. Since all the input and output information are geometrical (no RGB textures), we propose to use simulator to generate training data due to minimal sim-to-real gap for geometrical data. The dataset, model, and training method are as follows.
Dataset Generation We the objects in the ShapeNet dataset and some randomly generated regular rigid bodies as training objects. We rescale all the objects to regular daily object sizes, which about 4~200m on each dimension. Then, we randomly generate a camera with random field of view (FoV) between 40~552 (common camera FoV), and randomly place the object in the camera view to render the initial depth frame. Then, we generate random twist motion S [v, w] to move (translate and rotate) this object for several steps. We calculate both the 3D pixel movement and the groundtruth 3D motion of each observed pixel the initial frame through ray casting followed by transformations shown in Figure 3 We generate 8M samples at 256 x 256 resolution for training, which can be produced with NVIDIA L40 GPU less than hours.
Data Augmentation During training, we use diverse data augmentations to simulate the noise effect of each sensor observations, and the underlying idea is relevant to the Denoising Autoencoder 43 which reconstructs signals from sensor noises for downstream processing. The most common noise for depth is random missing values, white noises, and wrong values (especially for moving objects), and we apply these effects on the depth image. For the pixel flow input, since we find existing trackers are usually off by few pixels, we apply random Gaussian noise augmentation. Besides, also apply random dropout on the pixel flow input, i.e., only using part This also allows us to partial, sparse pixel flow for inferring 3D motion field, so that speed
flow vectors for prediction.
5

---

up the data labeling process, Finally, we also apply subset masking to the input feature map. With this, we can approximate complex object contours with simple objects.
Model: 3D Motion Field Estimator We use dual head UNet [33 model as our 3D motion field estimator f. This model predicts Edepth and Fmotion through two separate low-level decoder branches faepth and fmotion. This is to reduce interference near the output as these predicted values have different semantical meanings. Besides, we also append a dense, "intrinsic" map feature Imap E R 3xxxxx4 to the image, whose elements given by
I mop [V, x] = (vy cy)/ fy. (x -c)/ 1/ f/: 1/fa).
Iapp contains crucial low-level information for accurate Fmotion prediction. To understand this, let us consider motion prediction along z-axis as an example. Since X = (x cz Z/ by taking differential we have dXX (Z/fa) [( fa)/faldZ. The (noisy) 2D pixel motion dr, depth motion dZ, and depth Z are already included as input, while the remaining 1/.a. (x c terms cannot be inferred by a CNN architecture without position embeddings. Hence, also include them explicitly as input as well for predicting r-axis motion dXX. These features will directly propagate to the last layers in UNet through skip concatenations. We show their importance in Figure 8
Training We apply weighted Huber loss (Il - (]) as a stable supervision to train this model:
- C- M @ (fdepan (z) - Fdepst)ll Il + allM @ (fmotion (x) - Fmotson)ll Il-
(2)
In the loss function above, Dsim is the generated synthetic dataset. M is the object mask and @ is the (broadcast) elementwise product, so the dense motion field prediction loss is only applied on the object part. is weighting hyperparameter. We will also present a iitlusion-bassed architecture and objective in the next section, but for this motion estimation task, a direct prediction without further refinement is already good enough. We use the A optimizer [22] train this model and the training procedure takes about 1 day with 16 NVIDIA A A100-40GB GPUs.
Discussion I: Motion and Geometry Synergy A key idea in our prediction task is the synergy between the motion and geometry (depth)- the motion clue can be used to recover the missing or wrong depth value. One example is shown in Figure If we observe that some pixels move together with others whose depth values are known, we can reasonably infer their depth -epth-Rf particularly when the observed object is rigid. This formulation can also be considered as masked pretraining of 4D (time plus 3D) geometrical data. Most importantly, the tracks on the 2D plane are usually very visually accurate (< 3 pixel er- ror), making such "masked" reconstruction feasible. In the experiments, we find that adding motion and camera intrinsics information can significantly improve depth pre- diction accuracy.
What is
Noisy Depth
there ?
object
future
Observed Tracks
Figure 5: Guess what is there? Object tracks can be used to recover missing or wrong depth values.
Discussion II: Extension to Moving Camera In our current implementation, assumed static camera. However, since the data is produced in simulation, this framework can be extended to estimating 3D motions in videos captured by moving cameras (through simulating camera movement). Then again, one can append an additional dimension of (noisy) camera motion as input to the estimator and it process general videos (the camera motion in videos can be inferred via monocular simultaneous localization and mapping (SLAM) 10]
We leave this extension as a future work.
4 Phase II: Predicting Object 3D Motion Field for Control
Now that we have a model to see the groundtruth 3D motions from noisy sensors, we are ready to build our control policies with human videos.
Human Videos We only require human object interaction video dataset Dhumen to train our control policy. We first process the dataset through our learned estimator and existing foundation models. Specifically, we first apply foundation segmentation model SAM2 [31 in video mode to extract the segmentation of all the task-relevant objects each frame. Then, we use pixel trackers (CoTracker3 (17) to extract the noisy 3D pixel flow of each object point and lift them to an
Dataset:

---

Motion Comparison
Depth Comparison
Ours
6DoF
Baseline
RGBD Camera
(Direct)
Figure 6: (Left) Experimental Setup. (Right) Qualitative Results on "Pen" (Left Figure). Our method produces smoother motion field and depth compared baseline. This is essential for accurate control.
1mm
4cm
Figure 7: A rollout of fine-grained insertion. Our method can achieve high precision, even if we are observing the motion from 40em away without wrist camera. However, we observe an adjustment behavior resembling bang-bang control it is still challenging for the policy to insert accurately in one-shot as human. Fortunately, still captures rough directions for adjustment to finish the task.
accurate 3D motion field through our pretrained estimator. However, an underlying requirement that require the object consistently visible (i.e. not fully occluded) throughout the video segments, so we have to discard the fully occluded segments and learn from all the remaining partially occluded and fully visible segments. We leave the full occlusion case to future investigation.
Model and Training Then, we train a policy network π to predict these labeled 3D motion field with the segmented RGBD image as input. Since our motion field is image-shaped, we can use either a Gaussian diffusion model (policy) [14 38 for its accurate prediction. We reuse most of the dual-head UNet architecture our policy We apply the following general regression objective for training (for both diffusion and Gaussian policy):
L = Bo,,F,M-Daumor | IM ④ (Tdepth (o, F,t) Fdepth))l a @ (Tmotion (o, F,t) F Footio ||-
(3)
In this objective above, is the segmented RGBD image observation, F is the groundtruth object 3D motion field (desired action over the object) labeled by our estimator, and M
mask of the
corresponding object extracted by SAM2. (F. () the noised motion field sample and timestep and only apply diffusion model. As policy network only uses task-relevant object information as input and does not observe any cmbodiment-speccie and robot domain is minimal. However, still find it important apply random masking data augmentation to objects. Besides, for diffusion model we also find it useful to use "masked noise sample" as input- this simplifies training and makes the model more robust.
information, the gap between human domain
Deployment In the inference time, we need convert the predicted 3D motion field F to the robot action. As the object is already firmly grasped by the robot, this conversion is straightforward. For each pixel on the object mask, given F. we compute their current and future 3D coordinates in the camera frame through camera inverse projections, resulting in two point clouds Po, Pi E IR Importantly, these two point clouds have point- wise correspondence (as opposed to unordered point clouds that require ICP (2) so that we can directly solve a SE(3) transformation T, = {R. t} for aligning them. We minimize RRP + t Pf 1|2 which has closed form solution (Kabsch method [66). As there are outliers in Po and P. inevitably, the quality T.. Then, suppose the camera pose robot base frame {b} is The our desired robot action can be computed as T.- The T.T-I be The conversion from object 3D motion field to SE3 action is very fast at around 300- 1000100 introducing minimal overhead control loop.
also use RANSAC [3] to improve

---

-30-
3D
Success
Makon
Foceiie
Durs (eat
Our
19-4
Reath mains
23-
1.3
64-
6:3-
62
23-4
Keci Sample
J1-383 2-2.00
0-4-46
140bad
No RK. Focal Len
0utt (F48
Boc
Balk
Figure 8: (Leftt) SE3 motion estimation performance real world. Our method achieves lower error compared to baseline. (Middle) Intrinsics Map Ablation Studies. Both inverse focal length and coordinate map are crucial. (Right) Real world Task Success Rate (3 seeds). Baseline is the latest General Flow [59]. Other recent methods fail on our setup due to their limitations (Table 2
5 Experiments
In this section, demonstrate the effectiveness of our 3D motion field estimator and our control policy through real world experiments.
System Setup We use a widely-used Intel D435 RGBD camera at 640 480 resolution for video dataset collection at 30Hz (Figure 6 During training, crop the image to 480 x 480 and rescale it to 256 256. To get high-quality raw depth, we use the native temporal and spatial filters provided by the camera SDK. We use XArm7 robot arm with parallel-jaw gripper for the test dataset collection and robot experiments. We did not use wrist camera in the experiments as previous works. In the experiments, the manipulated object is typically 40~50em away from the camera.
5.1 Evaluating 3D Motion Field Estimator
Setup We capture RGBD video of randomly moving objects in the real world with their corre- sponding ground truth transformations as the test set. We program the robot arm to grasp random objects and wave them in front of the camera. Since the robot gripper grasps the object, we can directly obtain the groundtruth object 3D transformation by reading and calculating robot gripper pose transformation. We use objects of diverse aspect ratios and apply diverse robot motions.
Main Results We show the results of our method and the latest baseline "direct" method in Figure g We report the MSE of recovered object translation and the F-norm of object rotation matrix error. Note that use the gripper frame as the base for representing these transformations. We find that approach has significantly lower error compared to the baseline. We also visualize some examples in Figure 6 Our method successfully reproduces smooth depth and motion field even if the input depth is very noisy. We refer readers to appendix for more details.
Adversarial Robustness We test robustness further through adversarial attack in real world ex- periments by injecting Gaussian noise of different intensities into the depth observation (which then propagates into the computed 3D pixel flow input). We find that the baseline performance deteriorates quickly, while the error of our method remains at the same level. However, this is somewhat expected since the same kind of noise is also applied during our training process.
We find that both coordinates and inverse focal length are crucial for successful learning, confirming our derivation. Surprisingly, the focal length value plays a critical role in motion prediction even for a relatively small FoV variation around 10 degrees.
Ablation Studies We also ablate intrinsic maps as shown in
5.2 Robot Learning from Videos
Real world Tasks In this section, we test if our method can acquire object manipulation skills from human videos. We introduce the following tasks to benchmark the performance:
1. Pick, Rotate, and Place. The robot is required to pick up object and/or rotate it and put it at target location. This task is considered as successful only the object correct pose in the end.
2. Line Tracking. The robot is required to pick up a pen-like flashlight and control it to track cable on the table. This task is considered successful if the robot can finish the tracking trajectory with spotlight focusing on the cable in the process. The procedure is monitored by human.

---

Table 2: Technical Feature Comparisons. Our method is free from many limitations of existing works.
Note that this list is non-exhaustive and we refer readers to the text for more discussion.
UniPi
ATM
Track2Act
GFlow Im2Flow2Act
SPOT
TI
Ours
Method
9 23
49
24
31
222
15 24
25
No Pose Estimation
X
No Robot Data (Zero-shot)
Close-loop Motion Control
V
V
V
X
V
V
Depth Robustness
N/A
N/A
N/A
X
N/A
N/A
N/A
Distractor Generalization
X
X
X
X
X
Tool Use Pushing. The robot required to pick up a tool and push one object to a goal.
4. Tool Use II: Wrench. In this task, the robot is required to pick up a wrench and use it to tighten nut by a round. This task can be viewed as a more challenging version of pick, rotate, and place since the process is kinematically constrained and requires precision.
Insertion. In this task, the robot is required pick, rotate, and insert an item into a slot (hole). There is only 2.5mm tolerance so this requires very high precision.
We collect around 50-150 human videos for each of these tasks. The data collection procedure for each of the tasks lasts around 2-15 minutes, depending on task complexity. During the evaluation, we ensure that the grasping and the object segmentation correct for each of the evaluated method (otherwise we restart that trial).
We show the success rate of different methods Figure Right. We find that our method significantly outperformed the other evaluated methods. During the deployment, the baseline method will quickly deviate from the correct moving direction/trajectory. Our method not only solves the tasks but also follows the -emman--tmmnstrrteed path accurately throughout the evaluation (see Appendix). We attribute its success to accurate and smooth motion estimation. Besides this, we also observe the expected robustness to the background variations during experiments due to the use of object-centric input representation.
Main Results
A Ablation Studies We also study the design choices of our policy architecture and training. We find that for fine-grained tasks, it is important to apply a diffusion model even the human has tried to act consistently as possible. Compared to the Gaussian policy head, the diffusion policy can produce high- quality, accurate motion fields which is important for success. Besides, we also find important to mask out unnecessary regions in the diffusion model reverse step. Otherwise, the irrelevant noise in non-object regions can slow down training and harm performance. Finally, find it important apply object masking augmentation during training, as the object's ' silhouette under the robot gripper differs from that under a human hand, which leads to subtle domain gap.
Table 1: Policy Learning Ablation for Fine-grained Tasks.
Setting
Success
w/o Diffusion (Diff.)
0.0%
w/o Diff. Masking.
0.0%
w/o Masking Data Aug.
5.0%
Full
35.0%
Related Works
Robot Learning from Videos We summarize the most relevant studies in Table A key aspect of existing methods is how they represent actions. Some approaches rely on direct video frame for control while recent research explores compact and informative repre- point cloud flow [55 59]. and 3D poses [15 6). While these approaches offer certain ad advantages, each has notable limitations, as previously discussed. Among them, point cloud flow is most relevant to approach. However, we distinguish our work by employing an image-based 3D motion representation with a denoising architecture to refine the extracted 3D information. Another related line of research involves embodiment-speciff action retar- geting [5T 32 20). which can be infeasible in general setups. In contrast, our bobooddment-ggnosste approach aims to extract common motion knowledge for control.
prediction 12
sentations, such as 2D pixel flow 49
52 3
In addition to action representation, another challenge is domain grgemment-ensurinn that the learned model can effectively transfer the robot domain. Recent works tackle this by employing ego

---

masking or inpainting 220 [8]. We have a similar idea but adopt a more object-centric approach by using task-relevant objects as imput. While not entirely new 662, we are the first apply this concept within the context of learning from human video demonstrations. In a broader context, human videos have also been used as auxiliary data source to pretrain aasion-languaee
models/policies [?.37] in simulation, which are orthogonal directions.
or define RL tasks [29 30 28
3D Vision and Video Analysis Our method is related to 3D reconstruction from videos 10
34 Some works in this area have utilized 3D representations such as Neural Radiance Fields (NeRF) 25 Gaussian Splatting [19]. and Pointmaps [47 60 46) for scene geomerry reconstruction, while our work focuses motion reconstruction. In recent 3D motion extraction research, some works propose both optimization-basee
26
Our method shares a similar goal but differs significantly in focus and underlying assumptions. These works typically focus on long-range (time) tracking and assume the raw depth as the reference even if it can be noisy and inaccurate. In contrast, we focus on recovering depth and precise instantaneous motion from noisy temporal depth sensing. In this sense, our work is more aligned with depth estimation [53 48 areas traditionally explored within computational photography for 3D. While many existing approaches aim to recover depth from monocular or stereo RGB images as well, our method takes a distinct direction by leveraging the interplay between depth and motion to improve 3D understanding. Finally, we note that our approach is compatible with the methods above it can be integrated with existing depth and track estimation methods by using their depth and track outputs as inputs.
and end-to-end tracking method 227 50 45
7 Conclusion
In this paper, we have demonstrated novel framework for learning robot control policies from human videos using object-centric 3D motion field representations. Our approach overcomes key limitations of existing representations by introducing robust 3D motion estimator and dense flow prediction architecture, enabling better cross eooos-embodiment transfer and background generalization. Experiments demonstrate substantial improvements in motion estimation and success rates across diverse real-world tasks compared to prior works, and the unprecendented effectiveness in handling precise manipulation tasks. Our approach opens up new possibilities for leveraging scalable human video datasets to train versatile and generalizable robotic agents.
Acknowledgement
This work is part of the Google-BAII, Commons project. The authors gratefully acknowledge Google for providing computational resources. Zhao-Heng Yin is supported by the ONR MURI grant N00014-22-1-2773 UC Berkeley. Pieter Abbeel holds concurrent appointments as Professor at UC Berkeley and as an Amazon Scholar. This research was conducted at UC Berkeley and is not affiliated with Amazon.
Broader Impact
Our method targets one of the most significant challenges in robot learning: data. By addressing this issue through human video data, we open the door to scaling up data for the development of foundational robotic agents.
Limitations
We identify the following limitations this work as directions for future research. First, we did not consider knowledge extraction in the case of full occlusion we may need separate action representations for mining action knowledge from fully occluded data. Second, we mainly consider grippers and it would be interesting to study the case of robotic hands for which we may need to train motion-conditioned control policy rather than solving an optimization problem as we did. Third, in this study, we assumed fixed camera although we have shown a potential way to scale it to moving cameras (Section 4 Discussion II). Finally, it would be interesting to handle the case of (very) soft-bodies like cloth for which we should not extract an SE(3) transformation from whole object movement, but instead only consider subset of motion field around the contact point. In summary, there are several directions for extending this work and making a useful industry-level solution.
10

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the summary part of introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See the limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
There is some short discussion on the limit of existing assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details on collecting dataset and model training.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce
12

---

the model (e.g., with an spen-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release the trained model and learning pipeline.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ltipps.ccpublic/guiiico hwwli//ggisss//eee http://p//./ggiiiieoo//o//aaaaaa
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We discuss the setup thoroughly in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
13

---

Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, this is reproducible several tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discussed them in implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /ttp///eerip.../// htt.urpp..cpppli///thiiiiii cc/ppbllc/tthhcccuueeeelll.
Answer: [Yes]
Justification: Yes.
Guidelines:
14

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
See broader impact section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
15

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite them properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We use some simulation generated assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coddemoodll as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
16

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should be described.
References
[1] Belkhale, Y. Cui, and D. Sadigh. Data quality in imitation learning. In Neural Information Processing Systems (NeurIPS), 2023.
[2] Besl and McKay. Method for registration of 3-d shapes. Sensor Susion control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
17

---

[3] H. Bharadhwaj. R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. European Conference on Computer Vision (ECCV), 2024.
[4] K. Black, M. Nakamoto, Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pretrained image-editing diffusion models. In International Conference on Learning Representations (ICLR), 2024.
[5] A. X. Chang, T. Funkhouser, L. Guibas, P. Hamrahan. Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv:1512.0300 1512000012.. 2015.
[6] H. Chen, C. Zhu, Y. Li, and K. Driggs-Campbell. Tool-as-interface: Learning robot policies from human tool usage through imitation learning. arXiv preprint arXiv:2504.0400 2025.
[7] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273666. 2023.
[8] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and Song. Universal manipulation interface: wn-the-wild robot teaching without in-the-wild robots. In Robotics: Science and Systems (RSS), 2024.
[9] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. In Neural Information Processing Systems (NeurIPS), 2023.
[10] H. Durrant-Whyte and T. Bailey. Simultaneous localization and mapping: part i. IEEE robotics & automation magazine, 13(2):99-110, 2006.
[11] H.-S. Fang, C. Wang. M. Gou. and C. Lu. Graspnct-lbilion: A large-scale benchmark for general object grasping. In JEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2020.
[12] C. Finn and Levine. Deep visual foresight for planning robot motion. International Conference Robotics and Automation (ICRA), 2017.
[13] M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981
[14] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems (NeurIPS), 2020.
[15] C.-C. Hsu, Wen, J. Xu, Y. Narang, X. Wang, Y. Zhu, Biswas, and S. Birchfield. Spot: Se (3) pose trajectory diffusion for object-centric manipulation. arXiv preprint arXhv:2411.00955. 2024.
[16] W. Kabsch. A solution for the best rotation to relate two sets of vectors. Foundations of Crystallggruhhy 32(5):922-923, 1976.
[17] N. Karaev, I. Makarov, J. Wang, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker3: Sim- pler and better point tracking by pscudo-labelling real videos. arXiv preprint arXiv:2410.11831, 2024.
[18] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang. J. Hoffman, and D. Xu. Egomimic: Scaling imitation learning via egocentric video. In International Conference Robotics and Automation (ICRA), 2024.
[19] B. Kerbl, G. Kopanas, T. Leimküüler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4): 2023.
[20] M. Lepert, J. Fang, and Bohg. Phantom: Training robots without robots using only human videos. arXiv preprint 25X1022503.0077,, 2025.
[21] V. Liu, A. Adeniji, H. Zhan, R. Bhirangi, P. Abbeel, and L. Pinto. Egozero: Robot learning from smart glasses. arXiv preprint trXiv:2505.20290 2025.
18

---

[22] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), 2018
[23] W.-C. Ma, S. Wang. R. Hu, Y. Xiong, and R. Urtasun. Deep rigid instance scene flow. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[24] Z. Mandi, Y. Hou, D. Fox, Narang, A. Mandlekar, and S. Song. Dexmachina: Functional retargeting for bimanual dexterous manipulation. arXiv preprint urXiv:2505.2485. 2025.
[25] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[26] R. Mur-Artal, J. M. M. Montiel, and D. Tardos. Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions on Robotics, 51(5):11471166 2015.
[27] T. D. Ngo, P. Zhuang, C. Gan, E. Kalogerakis, S. Tulyakov. H.-Y. Lee, and C. Wang. Delta: Dense efficient long-range 3d tracking for any video. International Conference on Learning Representations (ICI.R), 2025.
[28] A. Patel, A. Wang. Radosavovic, and J. Malik. Learning to imitate object interactions from internet videos. arXiv preprint arXiv:2211.13225. 2022.
[29] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills, ACM Transactions On Graphics (TOG), 37(4):1-14, 2018.
[30] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang. R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. 2022.
[31] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. In International Conference on Learning Representations (ICLR), 2025.
[32] J. Ren, P. Sundaresan, D. Sadigh, S. Choudhury, and J. Bohg. Motion tracks: A unified repre- sentation for human-robot transfer few-shot imitation learning. In International Conference on Robotics and Automation (ICRA), 2025.
[33] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[34] J. L. Schonberger and J.-M. Frahm. on Computer Vision and Pattern Recognition (CVPR), 2016
revisited. In IEEE/CVF Conference
[35] D. Seita, Y. Wang, S. J. Shetty, E. Y. Li, Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot Learning (CoRI.), 2023.
[36] K. Shaw, S. Bahl, A. Sivakumar, A. Kannan, and D. Pathak. Learning dexterity from human hand motion in internet videos. The International Journal of Robotics Research, 533455333-532, 2024.
[37] H. G. Singh. A. Loquercio, C. Sferrazza, J. Wu, H. Qi, P. Abbecl, and J. Malik. Hand- object interaction pretraining from videos. In International Conference on Robotics and Automation (ICRA), 2025.
[38] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.
[39] T. Tao, M. K. Srirama, J. Liu, K. Shaw, and D. Pathak. Dexwild: Dexterous human interactions for in-the-wild robot policies. In Robotics: Science and Systems (RSS), 2025.
[40] Z. Teed and Deng. Raft-3d: Scene flow using rigid-motion embeddings. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8888,, 2021.
19

---

[41] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar. G. Zhou, R. Hadsell, L. Agapito, and
J. Scholz. Robotup: Tracking arbitrary points for few-shot visual imitation. In International Conference Robotics and Automation (ICRA), 2024.
[42] S. Vedula, S. Baker, Rander, R. Collins, and T. Kanade. Three-dimensional scene flow. In IEEE/CVF International Conference on Computer Vision (ICCV), 1999.
[43] Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust fea- tures with denoising autoencoders. In International Conference on Machine Learning (ICML), 2008.
[44] C. Wang, H. Shi, W. Wang, R. Zhang, I.. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.0778, 2024.
[45] Q. Wang. V. Ye, H. Gao, J. Austin, Z. Li, and A. Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint arXiv:2407.1376 13764, 2024.
[46] Q. Wang. Y. Zhang, A. Holynski, A. Efros, and A. Kanazawa. Continuous 3d percep- tion model with persistent state. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[47] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[48] B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield. Foundationstereo: Zero-shot stereo matching. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[49] C. Wen, X. Lin, So, K. Chen, Q. Dou, Y. Gao, and Abbeel. Any-point trajectory modeling for policy learning. In Robotics: Science and Systems (RSS), 2024.
[50] Y. Xiao, Q. Wang. S. Zhang, N. Xue, S. Peng, Y. Shen, and X. Zhou. Spatialtracker: Track- ing any 2d pixels in 3d space. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[51] H. Xiong, Q Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In International Conference on Intelligent Robots and Systems (IROS), 2021.
[52] M. Xu, Z. Xu, Y. Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song. Flow as the cross-domain manipulation interface. In Conference on Robot Learning (CoRL), 2024.
[53] L.. Yang. B. Kang, 7. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. In Neural Information Processing Systems (NeurIPS), 2024.
[54] M. Yang. Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel. Learning inter- active real-world simulators. In International Conference on Learning Representations (ICLR), 2024.
[55] 7. Yin and P. Abbeel. Offline imitation learning through graph search and retrieval. In Robotics: Science and Systems (RSS), 2024.
[56] Z. Yin and P. Abbeel. Lightnnig grasp: High performance procedural grasp synthesis with contact fields. arXiv preprint arXiv:2511.07418, 2025.
[57] Z. Yin, C. Wang, L. Pineda, F. Hogan, K. Bodduluri, A. Sharma, P. Lancaster, I. Prasad, M. Kalakrishnan, J. Malik, et al. Dexteritygen: Foundation controller for unprecedented dexterity. In Robotics: Science and Systems (RSS), 2025.
[58] Youeube. You know what's cool? a billion hours, February 2017.
[59] C. Yuan, C. Wen, T. Zhang, and Y. Gao. General flow as foundation affordance for scalable robot learning. In Conference on Robot Learning (CoRL), 2024.
[60] J. Zhang, C. Herrmann, J. Hur, V. Jampani, T. Darrell, F. Cole, D. Sun, and M.-H. Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. In International Conference Learning Representations (ICLR), 2025.
20

---

[61] J. Zhang, H. Liu, D. X. Yu, H. Geng, Y. Ding, J. Chen, and H. Wang. Dexgraspnet 2.0: Learning generative dexterous grasping large-scale synthetic cluttered scenes. In Conference on Robot Learning (CoRL.), 2024.
[62] Y. Zhu, Z. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. In Conference on Robot Learning (CoRL), 2023.
21


Reviews:
[{"review_id": "nUhpso8fw2", "reviewer": "Reviewer_tpQo", "summary": "This paper proposes an object-centric pipeline for learning robot policies from human videos. One contribution of this paper is a fine object 3D motion estimator, and the other contribution is the prediction algorithm of object-centric 3D motion (the scene after the target is manipulated). The algorithm mostly operates in 3D, with compact usage of classic techniques such as ICP and RANSAC. Overall, this is a good work. Please see my comments below.", "questions": "please see my comments above.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. As the authors do not opensource the model, can the human video lists be available? Otherwise the results are not reproducible. \n\n2. The paper does not mention how the manipulator is controlled. Through inverse kinematics? \n\n3. Did the authors test the algorithm in complex background scenes? The paper says that the sensor is mounted 40cm~50cm away from the target. Therefore, are the figures in Fig.7 cropped images? not the raw data? \n\n4. I assume that the hole structure shown in Fig.7 is fixed on the table.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work! Our responses to your questions are as below.\n\n**Q1.** Thank you for the suggestion. We will release the synthetic data generator, pretrained model, and data labeling pipeline to ensure reproducibility and enable researchers to apply our method in their own task setups.\n\n**Q2.** Yes, the manipulator is controlled via inverse kinematics after computing the next target pose.\n\n**Q3.** The image in Figure 7 was recorded from a different viewpoint solely for visualization purposes and does not reflect the policy's actual observation. In practice, the object appears significantly smaller in the policy's visual input.\n\n**Q4.** Due to the difficulty of the precise insertion task, we opted not to rigidly fix the base structure. This simplification helps avoid damage that can occur when the stiff robot controller applies excessive force. However, it's important to note that this is still very hard as our control policy is purely vision-based (from 40-50cm away) without any force or tactile information, and accurately inferring the insertion angle from vision alone is inherently challenging (and this may result in a slight base movement). For the success trials, the base may be moved by around 1-2mm based on our measurement on the base corners. We hypothesize that in the future, the use of higher-resolution depth cameras or multi-view sensing setups (e.g. fusing the 3D motion flow prediction) could help improve performance in such tasks."}, {"review_id": "hPHaIvNV6F", "reviewer": "Reviewer_e4mh", "summary": "This paper introduces an object-centric 3D motion field as the action representation for learning robot control policies from human videos. The motion field is defined as a 4-channel image, where the first channel is depth and the next three channels are 3D movements of each pixel from the current image frame to the next. The paper proposes a two-phase learning framework. In phase 1, a denoising 3D motion field estimator (with extra intrinsic map features as input) is pretrained in simulated environments. In phase 2, object instances are segmented (SAM2) from human videos. A policy network is then trained on lifted (CoTracker3) and denoised (phase 1) 3D motion field extracted from human videos. The proposed 3D motion field-based policy is evaluated on 5 real-world scenarios, in which *Tool Use II: Wrench* and *Insertion* are difficult manipulation tasks. The experiment results demonstrate that the proposed method succeeds even when no robot demonstrations are provided, and the evaluation performance is better than the baseline.", "questions": "1. (Weakness 1) How to address the 3D motion field difference (the pixels' 3D movement part) caused by the speed difference between humans and robots?\n2. (Weakness 2) What is the exact frame interval used to compute the 3D movement used in the extraction of 3D motion field?\n3. (Weakness 3) Is the depth channel duplicated in the phase 2 model's input? If yes, why is this redundancy introduced?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes 3D motion field. This representation is aligned to observation frames (unlike pointcloud tracking methods) and can be integrated with input image observations.\n2. This paper introduces the intrinsic map as an extra feature input to the model, and the authors explain the motivation and insights for introducing it.\n3. This paper is clearly written.\n\nWeaknesses:\n1. (Motion field movement difference) The proposed motion field covers the 3D movement of each pixel between the current frame and the next frame. However, the amount of the movement is determined by the motion speed of the human/robot between the frames. Typically, humans move the object at a much faster speed, leading to a larger movement between the frames. The paper does not provide the explanation of how the 3D Motion Field addresses the differences in 3D motion fields arising from the speed differences between humans and robots when performing tasks.\n2. (Motion field frame interval) In addition to 1., the exact value of the frame interval between the current frame and the next frame used in the extraction of 3D motion fields is not directly provided. There is information on the fps of the camera used in deployment, but it is unclear whether that fps aligns with 3D motion field extraction.\n3. (Duplicated input channel) In phase 2, the input contains both the 3D motion field and the segmented RGBD. As the 3D motion field already contains the depth information, the depth channel in the input seems to be duplicated. The paper provides no explanation or analysis of this potentially redundant input format.", "was_revised": true, "final_justification": "The authors' response has addressed my concerns, including how to handle the gap between human demonstration and robot deployment, the frame interval for the extracted motion, and the specific form of the representation. The rating is kept as Accept.", "rebuttal": "Thanks for the review! Our responses to your questions are as below.\n\nQ1. Thank you for the question. Our method directly generates the next target positions for all object points, which are then used to compute the corresponding robot poses. The robot proceeds to the next prediction cycle only after successfully reaching the current target. We can command the robot to the target within the desired time interval.\n\nQ2. We use a control frequency of 10 Hz in our experiments, corresponding to a 0.1-second interval between predictions.\n\nQ3. In the second phase, the depth in the camera input is still very noisy, and our policy can output a cleaner depth map for motion recovery."}, {"review_id": "eKNM9Dzsce", "reviewer": "Reviewer_gWxp", "summary": "The paper demonstrates that the motion field is key to learning from human videos for robotic policy learning. It first analyzes the drawbacks of current action representation and points out that the proposed object-centric motion field has clear advantages. Then, it argues that the learning objective from the human video is object 3D motions and demonstrates the necessity of using RGBD video. After that, it raises the challenges when learning 3D motions from videos, which is the accumulated noise. It finally designs the 3D motion field estimator to reduce noise, which makes it possible to train video policy models from the accurate motion fields. It also discusses the deployment issues. Experiments on the real-world setups demonstrate the effectiveness of the proposed method.", "questions": "The paper is well-written and acceptable. The motivation is clear, the proposed method is effective, and the discussion of each part of the proposed model is adequate. I only have a few concerns (see above).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n- The writing is quite clear and fluent. The motivation of the proposed method is well established and discussed.\n- The proposed model that learns motion field from video is interesting and has clear advantages compared to previous learning strategies.\n- Experiments show that the proposed learning strategy performs well in real-world scenarios.\n\n## Weakness\n\n- In Phase 2, the model only takes object information as input, which is pre-processed by SAM2. Is it suitable for real-world settings? We may always have to manually instruct SAM2 to obtain the segmentation of the target object.\n\n- I found that when training the policy model in Phrase 2, it only uses RGBD image to infer the 3D motion field. This confuses me because object motion in one image can correspond to multiple potential motion fields. How can the predicted motion field be ensured is the desired motion field? For example, given a bottle image, the policy model could predict arbitrary motion trajectories of that bottle. It seems that it cannot control the predicted motions. Do I miss something?", "was_revised": true, "final_justification": "I keep my original rating.", "rebuttal": "Thanks for the review! Our responses to your questions are as below.\n\nQ1. Thank you for the question. In most of the scenarios, we may want to prompt the robot with language during deployment. There are many existing vision-language models (VLMs) and open-vocabulary object detectors (e.g. Grounding-DINOv2) capable of grounding target objects in human language instructions to a box/point in the current scene automatically. We believe that this will not be an issue in the near future due to rapid advances in VLM and vision research. In our setup, the object only needs to be labeled once at the beginning. Its mask is then robustly tracked throughout execution using SAM2. We have experimented with Grounding-DINOv2 + GPT and it works well in our setups. \n\nQ2. Thank you for the question. The human videos we use are task-relevant—for example, if the task involves picking up a bottle and pouring it, the predicted motion field consistently reflects that intention (e.g., the bottle moves upward and then tilts).\nWe acknowledge that motion in human demonstrations can be multi-modal—for instance, the bottle might be lifted slowly in one video and quickly in another. Our method addresses this by using an image-based generative model that learns to produce plausible motion trajectories observed in the training data. In other words, it can generate either a slow or fast lifting motion based on the current visual context, but the motion itself always remains valid and task-relevant. \nFurthermore, the predicted 3D object motion field defines how each visible point on the object should move in 3D space, relative to the camera frame. This makes the motion representation unambiguous—unlike 2D motion fields, which can be affected by depth ambiguities or projection artifacts."}, {"review_id": "yzFUOuEbPo", "reviewer": "Reviewer_xCja", "summary": "### Summary\n\nThis paper introduces a novel approach for robot learning from human videos, leveraging an object-centric 3D motion field to represent actions. The core of their framework is a 3D Motion Field Estimator, which is initially trained using simulated data to refine noisy motion fields. This estimator is then applied to improve the quality of human manipulation data, which is subsequently used to train a robot policy.", "questions": "Broadly speaking, the core idea of this paper—predicting the **point cloud flow (or motion)** of manipulated objects and then converting that into robot control signals—bears a strong resemblance to prior work like Track2Act and GFlow.\n\nWhile the paper undeniably presents a more refined workflow, incorporating elements such as dataset augmentation, deployment-time optimization, and leveraging SAM2 for object-centric understanding, it feels more like a highly engineered and detailed implementation of existing concepts rather than a breakthrough in innovation or a source of new insights. For instance, some of the \"new\" capabilities, such as **distractor generalization**, seem to stem from the integration of SAM2, and **depth robustness** appears to be a benefit of introducing the **3D Motion Estimator**.\n\nTherefore, I'd like the authors to elaborate further on the **innovative aspects and core insights** of your proposed method. Specifically:\n\n* Are there any capabilities that **only the proposed method can achieve** and that previous methods simply cannot?\n* Or is the proposed method primarily a **better-designed, more deployment-ready version** of existing approaches?\n\nI'm hoping for a discussion that highlights the unique contributions and fundamental advancements that set this work apart, beyond just improved engineering and integration of other state-of-the-art components.", "limitations": "Please see Questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n* The paper is well-structured, making it easy to follow with a clear and logical flow.\n* The proposed method is simple, direct, and easy to understand. Its workflow demonstrates a rigorous and effective engineering design.\n\n---\n### Weaknesses\n\n* The paper's method bears a resemblance to previous approaches, and some of its novel capabilities appear to stem from integrating existing models. I'll elaborate on this specific point in the \"Questions\" section.", "was_revised": true, "final_justification": "In my initial and current assessment, I found that the proposed method bears a resemblance to previous approaches, and some of its novel capabilities appear to stem from integrating existing models.\n\nHowever, it's undeniable that the **workflow is meticulously designed**, eliminating the need for manual dataset processing and enabling scalability. Insights from other reviewers and the authors' rebuttal have shifted my perspective; the distinction from previous methods now seems less critical. The proposed method successfully enables video-based policy learning to achieve fine-grained operational tasks, a capability that many advanced VLA models currently lack.\n\nTherefore, I agree that this work contributes significantly to the community, and the paper provides valuable insights into pipeline design.", "rebuttal": "Thank you for the thoughtful review! We would like to highlight several key novel contributions of our work:\n\n**1. More challenging data setups:**\nUnlike existing systems such as Track2Act, which require in-domain robot data, our method operates solely on human demonstration videos. In contrast to methods like GFlow, which rely on extensive point cloud filtering and heuristic, hand-engineered preprocessing (see their appendix), our approach avoids such manual steps by leveraging a learned motion denoiser. A detailed comparison with other systems can be found in Table 2. We made strictly less assumptions over data in this paper.\n\n**2. Scalability:**\nMany prior works, such as ATM and General Flow, use point-based prediction architecture, which makes it difficult to integrate them with standard generative modeling frameworks. Our method uses an image-based motion representation that aligns naturally with image generative models, facilitating greater scalability.\n\n**3. More precise tasks:**\nPrevious works have mostly focused on coarse manipulation tasks (e.g., mostly pick-and-place motion of large objects) and have not explored fine-grained scenarios involving small objects or tool use. Our paper demonstrates that such challenging tasks can, in fact, be addressed using a purely video-based learning pipeline.\n\nWe acknowledge that, at a high level, our method may resemble existing works in predicting \"flow\"-like representations for control. However, we argue that the specific choice of representation and the way to extract it are critical factors that determine the system’s capability. In summary, our contributions are twofold: 1. We propose a scalable pipeline that relies on significantly fewer data assumptions than existing methods. 2. We show that this pipeline can handle fine-grained manipulation tasks that prior approaches struggle with.\n\n\nTogether, these results provide the first strong evidence that simple, pure video-based policy learning is a viable and promising direction for solving complex, high-precision tasks—and now we are confident in its potential to scale further."}]

Author rebuttal:

Meta-review: The paper proposes object-centric 3d motion fields as a new action representation for robot learning from human videos, addressing limitations of pixel flow and point-cloud flow. A denoising motion field estimator is introduced to robustly reconstruct fine 3D object motions from noisy depth, along with a dense prediction architecture that supports cross-embodiment transfer and background 3d. Policies trained on these motion fields achieve over 50% lower estimation error and ~55% zero-shot success rates in diverse real-world tasks, including fine-grained manipulations such as insertion, where prior methods fail. This establishes a scalable framework for learning robot skills directly from human video data.

In the original round of reviews the committee appreciated the clear motivation and strong real-world validation of the proposed framework (R#gWxp, R#e4mh), the simple yet effective engg. design and workflow (R=xCja,tpQo), and the clarity and logical presentation of the method (reviewers gWxp and e4mh). The overall technical solidity and potential impact on video-based robot learning was also recognized.

The rebuttal effectively clarified key technical questions -- human–robot speed mismatch and frame intervals (reviewer e4mh), SAM2 reliance and motion ambiguity (reviewer gWxp), and novelty beyond prior flow-based methods (R=xCja) -- with reviewers confirming their concerns were resolved. R#e4mh kept their accept rating after being fully satisfied; gWxp reaffirmed their positive assessment; reviewer #xCja explicitly raised their score, acknowledging that the clarified design choices strengthen novelty and enable fine-grained video-based policy learning. The AC concurs with the unanimous positive consensus of hte committee’s reviews and post-rebuttal discussions.
Decision: Accept (spotlight)