Geometry-aware 4D Video Generation for Robot Manipulation
Authors: Zeyi Liu
Shuang Li
Eric Cousineau
Siyuan Feng
Benjamin Burchfiel
Shuran Song
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Robot Manipulation
3D Perception
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a 4D video generation model that enforces geometric consistency across views to generate spatio-temporally aligned RGB-D sequences, enabling downstream applications of robot manipulation tasks via pose tracking.

Abstract: Understanding and predicting dynamics of the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains a significant challenge. To address this, we propose a 4D video generation model that enforces multi-view 3D consistency of generated videos by supervising the model with cross-view pointmap alignment during training. Through this geometric supervision, the model learns a shared 3D scene representation, enabling it to generate spatio-temporally aligned future video sequences from novel viewpoints given a single RGB-D image per view, and without relying on camera poses as input. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, yielding robot manipulation policies that generalize well to novel camera viewpoints.

GEOMETRY-AWARE 4D VIDEO GENERATION
FOR ROBOT MANIPULATION
Zeyi Liu'>
Shuang Li'
Eric Cousineau"
Siyuan Feng*
Benjamin Burchfiel
Shuran Song'
Stanford University
Toyota Research Institute
https//rroboffbgen.githbbbb io/
ABSTRACT
Understanding and predicting dynamics the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains significant challenge. To address this, we propose 4D video generation model that enforces multi-view 3D consistency of generated videos by supervising the model with cross-view pointmap alignment during training. Through this geometric supervision, the model learns shared 3D scene representation, enabling it to generate spatio-temporally aligned future video sequences from novel viewpoints given single RGB-D image per view, and without relying on camera poses input. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shee shelf 6DoF pose tracker, yielding robot manipulation policies that generalize well to novel camera viewpoints.
INTRODUCTION
Cam Ve observations Cam V. coordinate
Output: 4D pointmaps with color in Cam vn's coordinate
Input
Video Generation
Cam Va, observations in Cam V., coordinate
t=1
t=
Time
Test Unknown Unseen Camera Poses
Figure 1: Geometry-aware 4D Video Generation. Our model takes RGB-D observations from two camera
views and predicts future 4D pointmaps in the coordinate frame of the reference view The blue pointmap predicted from camera tx., while the red pointmap shows the prediction from camera Uhe projected into the coordinate frame of v.. RGB videos are predicted separately each view. Together, the model enables geometry-consiseet consistent 4D video generation.
Understanding how the visual world changes with interactions is key capability for intelligent robotic systems. For robot manipulation tasks, the robots need to anticipate how the environment changes by taking into account object motions or occlusions upon interactions over extended time
"wok partially done internship

---

horizons. Recent advancements in video generation models offer powerful paradigm for learning such dynamic visual models: by forecasting future observations, robots can simulate possible outcomes, plan actions, and adapt to new scenarios.
However, it remains a challenge to generate realistic and physically plausible videos from which smooth and precise robot policies can be extracted from. There are two challenges: temporal coherence, ensuring smooth, causally consistent motion over time; and 3D consistency, preserving object geometry and spatial correspondences across different viewpoints. Most existing video generation models capture one at the expense of the other. Pixel-based models (Yan et al. 2021] Ho al,, ) trained on RGB videos often excel at short-term motion but lack an understanding 3D structures, which leads to artifacts like flickering, deformation, or object disappearance. In contrast, 3D-aware approaches enforce geometric constraints but are limited to simple, static backgrounds and struggle to scale to realistic, multi-object manipulation scenarios (Xie et al. 2024 Li et al. 2024a Zhang et al. 2024).
In this work, we present a video generation framework that bridges this gap by unifying strong temporal modeling with robust 3D geometric consistency. Our method produces 4D videos that can be rendered into RGB-D sequences that are both coherent over time and spatially consistent across camera views. To achieve this, we introduce geometry-consistent supervision mechanism inspired by DUS:3R (Wang et al. 2024b) and adapt it for the video generation task. Specifically, the model is trained to predict a pair of 3D pointmap sequences: one for reference view and one for a second view projected into the reference view camera coordinate frame. By minimizing the difference between reference and projected 3D points over time, the model learns a shared scene representation across views. This enables robust generalization to novel viewpoints inference, which is particularly useful in robotic applications where even small camera view shifts can push visuomotor policies out of distribution and lead failures.
Pretrained video diffusion models provide strong visual and motion priors learned from large-scale video datasets. To enhance temporal coherence, we initialize our model with pretrained weights and extend it to jointly generate future RGB frames and pointmaps. The RGB frames are trained using the original video generation loss, while the pointmaps are supervised using the proposed geometry- consistent loss. This combination enables the model to leverage the temporal priors of pretrained models while enforcing spatial and cross-view consistency through pointmap alignment, resulting in peatio-temporalyy consistent RGB-D video generation. We evaluate the 4D video generation quality on both simulated and real- world tasks, and our approach outperforms baselines in both video quality and cross-view consistency.
We further demonstrate that the predicted multi-view RGB-D videos can be directly used to extract robot end effector trajectories using off-the-shelf 6DoF pose tracker, such FoundationPose (Wen) et al. 2024). We evaluate this approach on three simulated robot manipulation tasks where the camera views are unseen during training, achieving good success rates all tasks and outperforms the baseline method. Additionally, our model generates future pointmaps in the reference view's 's coordinate frame solely based on the initial RGB-D observation in each view, without camera poses as inputs, bypassing extrinsics calibration for novel camera poses.
In summary, we propose a 4D video generation framework that achieves both 3D geometric consis- tency and temporal coherence. To achieve this, we first introduce geometry-consistee mechanism that enforces cross-view alignment of generated videos over time. Second, we develop a benchmark for video generation in robotic manipulation, comprising both simulation and real-world tasks. Each task is recorded from diverse camera viewpoints, enabling comprehensive evaluation of 4D generation quality and generalization to unseen views. Finally, we demonstrate that the generated 4D videos can be directly used to extract robot trajectories using an off-the-shelf 6DoF pose tracker, enabling applications to manipulation tasks.
supervision
2 RELATED WORK
Video Generation has been long-standing task in computer vision. Early works use recurrent networks (Srivastava et al. 2015 Chiappa et al.) 2017) or generative adversarial networks (Vondrick) et al. 2016) to learn temporal dynamics from video data. With the recent success of image diffusion models, many works have extended diffusion models for the video prediction task by adding additional
temporal layers (Yan et al. 2021 Ho et al. 2022) and using latent diffusion techniques (Blattmann
2

---

Input
Latent Video Diffusion UNet
Robot Gripper Pose Tracking from generated pointmaps
Robot Execution
RCB VAE
Eneeddr
Paintmep
RGe-o from Carn Ke
Shared weight
Poiromap Cross Attoniin
RCR VAE
Encoder
Pantmap WAE
RGB--D from Carn
Figure 2: 4D Video Generation for Robot Manipulation. Our model takes RGB-D observations from two camera views, and predicts future pointmaps and RGB videos. To ensure cross-view consistency, we apply cross-attention in the U-Net decoders for pointmap prediction. The resulting 4D video can be used to extract the 6DoF pose of the robot end-effector using pose tracking methods, enabling downstream manipulation tasks.
et al. 2023 Liu et al. 2024 Zheng et al., 2024 Yang al.) 2024b). In addition RGB videos, video diffusion models have also been shown to generate other modalities such as high-fide high-fidelity depth videos (Saxena et al. 2023 Hu et al. 2024 Shao et al. (024) and pointmaps (Nguyen et al. 2025). In this work, we adapt a latent video diffusion model (SVD) to jointly learn from RGB and depth videos with improved temporal and spatial consistency.
Multi- View and 4D Video Generation. Recent advances in camera-conditioned video generation improve spatial consistency by associating camera poses with multi-view videos (Van Hoorick et al.. 2024 He et al. 2024: Kuang et al. 2024 Cheong et al. (024). However, 4D video generation requires joint reasoning about both object geometry across views and motion over time. Prior 4D generation works separately optimize for temporal consistency with video model and spatial consistency with novel view synthesis model (Sun al.
[0244 Wang et al. [224c] Yang et al., 2024a). In this work, we present a 4D video generation framework unifies the two objectives. In particular, our method introduces gcometry-consistent supervision mechanism through cross-view pointmap alignment, inspired by DUS(3R (Wang et al. 2024b), on top of the standard video diffusion process. This approach enforces spatial consistency across camera views while ensuring temporal coherence. Our goal aligns with recent 4D generation works that jointly optimize spatial and temporal consistency (Xie et al.] 2024 Li et al. 2024a), but while they primarily focus on single-object videos with white backgrounds, target objti- dynamic robot manipulation scenes.
Generative Models for Robot Planning. With the recent success of video generation models and their strong generalization capabilities across diverse visual scenes, many works have explored their potential as dynamics models for robotic tasks (Yang et al. 2025b). Specifially, robot actions can be extracted from predicted future frames using learned inverse dynamics model (Du et al. 2023 Tian et al. 2024 Zhen et al., 2025), a behavior cloning policy conditioned on generated outputs (Xu et al. 2024 Bharadhwaj et al. 2024: Huang et al., (2025)), an BGB-based pose tracking model (Liang
et al. 2024). To more tightly couple future state prediction with action inference, recent methods have proposed unified models that simultaneously predict both future video frames and robot actions (Zhu et al., 2025 Li et al. 2025). Yet it remains challenge to generate spatially consistent predictions across views and capture accurate 3D geometry needed for precise robot manipulation. To bridge this gap, we propose model that learns spatial correspondences across camera views, enabling better RGBD video generation quality on novel views and improved end effector pose tracking accuracy for better robot task performance.
3 METHOD
In this section, present 4D video generation algorithm that unifies temporal coherence with 3D geometric consistency, enabling spatio-temporally consistent predictions over time and across varying viewpoints.
Problem Statement. Traditional video generation models predict future RGB frames
(Ot+1, O1+h} } from past observations (O.-w41, O.) captured from single view, where and are the history and future horizon respectively. However, single-view prediction lacks geometric grounding and often results in temporally plausible but spatially inconsistent outputs. To
3

---

address this, we introduce stereo-basee video generation network with additional 3D supervision to enforce consistent scene geometry across time and views. For pair of video frames O and O taken at time t from camera views Um and Un, each pixel coordinate (ł.j) E {1,... W x {1.-.. HH corresponds to a 3D point computed from its depth. This results in pointmaps X and X where Xς E R H x3 represents the per-pixel 3D coordinates for view Un and similarly for X Here, W and H denote the image width and height, respectively.
We first describe our video diffusion backbone for generating RGB videos ($ 3.1). next we intro- duce our geometry-consisten supervision mechanism on pointmaps across views to enforce spatial aligmment ($ 3.2p. We integrate temporal dynamics with 3D geometric consistency through joint optimization (S (.3 Finally, we demonstrate how 4D video predictions can be used to recover robot
end effector poses T2 E SE(3) at each time step using off-the-shelf trackers ($ 3.4).
3.1
VIDEO GENERATION
We adopt the Stable Video Diffusion (Blattmann et al. 2023) framework which has demonstrated
strong performance in generating short, temporally coherent video sequences. It first projects histori- cal video frames {O, (VAE) (Kingma et al., 2013) encoder. A diffusion model, implemented as a U-Net with an encoder- decoder structure, then predicts future latent representations (zi+1,'. Z.4hh, which are decoded back into RGB frames [0,+1. ... O++A} } using VAE decoder. The diffusion model fa is trained using an alternative of the standard DDPM (Ho et al., 2020) method, which directly predicts the original clean data from the noisy input at each diffusion step. The training objective for predicting future latent ž at timestep t' is to minimize:
O.3 into latent space using pretrained Variational Autoencoder
(tn( Eee z,)0). (0).k [lzz(0) folze (k),, (1)) k)] where Z (k) vag Z. (0) + /I as Ep' Er' (0,
Here ey denotes Gaussian noise, Ze (0) denotes the un-noised latent, and ze' (k) is the noised latent at diffusion step k. During inference, videos are generated by progressively denoising random Gaussian noise using the trained diffusion model.
3.2 CROMETRY-CONSISTEE
SUPERVISION
To enforce 3D consistency across views, we adopt the cross-view pointmap supervision strategy from DUSt3R (Wang et a 2024b), adapted to the video generation setting. As shown in Figure given the history pointmaps {X" A-++11"" X;) from camera view Un, we first encode them using a Pointmap VAE, which is initialized from the pretrained RGB VAE from SVD (Blattmann et al. 2023) and fine-tuned on pointmap data. This produces the latent representation (zi+1.'.. Z J We then apply the same latent diffusion method used for RGB video prediction to forecast future pointmaps in the latent space. The predicted latents are subsequently decoded by the Pointmap VAE X
decoder to obtain future pointmaps
In parallel, the model also predicts future pointmaps from second camera view Um, but instead of generating them in their native frame, it expresses them in the coordinate frame of view o.. This results in a sequence of projected pointmaps (Xm--" ...., X ). Each of these predictions are encoded into latent representations that are aligned with view Un. enabling supervision through cross-view consistency.
During training, we supervise the model fe at each future time step t' using diffusion losses applied to both the native view Un and the projected view Um + Un:
LAD ain (t') E.g, 23 (0). || (0) (2)2()),, k, c²]
+ Eeg. x,. (0(0),k || (0) (2 (m (k), k; |12
(2)
where (k) = Vox zミ (0) + vI ag ENr
2 (1:) = Vas प,u (0) + c7,
where 2, (k) denotes the noised latent of the pointmap from view v, at diffusion step k. and z ((k) the noised latent for the pointmap from view U.. projected into the coordinate frame of view Un.
4

---

Similar to Equation (1 ep and ep Gaussian noise added the pointmap latents from view Uh and Um respectively. The conditioning latents c" and cm are derived from the historical pointmaps of their respective views. See Appendix for model architecture details.
While camera poses are required during training to define the projection from view Um to the coordinate frame of Un. key advantage emerges at inference. Given observation from novel view Um the model can directly predict pointmaps the coordinate frame of Un eliminating the need for camera poses as inputs during testing.
Multi-Vies Cross-Attention for 3D consistency. Unlike RGB video prediction, where each view predicts future frames independently in its own coordinate system and single shared U-Net diffusion model can be used across views, pointmap prediction requires enforcing 3D aligmment across views. The native view Un predicts pointmaps in its own frame, while the second view Um predicts pointmaps projected into the coordinate frame of Un- To reflect this asymmetry, we use two separate decoders in the U-Net diffusion model (with identical architecture but independent weights) and introduce cross- attention layers between the decoders to enable information transfer. Specifically, the intermediate features from the decoder of view Un are passed to the corresponding stage in the decoder view Um through cross-attention. This allows the decoder of Um attend to and incorporate geometric cues from Un facilitating accurate pointmap prediction in the reference coordinate frame. This mechanism enhances cross- view geometric consistency, particularly under viewpoint variations.
3.3 JOINT TEMPORAL AND 3D CONSISTENCY OPTIMIZATION
The pretrained video diffusion model provides strong temporal priors for predicting scene dynamics, while the 3D pointmap supervision enforces 3D geometric consistency across views. We leverage the pretrained video model and optimize it with both the KGB-based video diffusion loss and the pointmap-based 3D consistency loss. The full training objective is defined as the sum of losses across all predicted time steps l' E (++1, 1+....., / + h and both camera views, Un and Um-
t+h
C =
(ti((t) + (t) +X- (t))
pointmap
where À balances the contribution of the geometric supervision. We set X = 1 in our experiments. This joint objective encourages both temporal coherence and cross-view 3D consistency.
3.4 ROBOT POSE ESTIMATION FROM 4D VIDEOS
We leverage the predicted 4D video to extract robot trajectories using off-the-shelf 6DoF pose tracking model, FoundationPose (Wen et al. 2024). The model takes as input RGB-D frames from a single view, binary mask of the target object in the initial frame (generated using SAM2 (Ravi et al. 2024)), camera intrinsics, and the gripper CAD model. At each pose estimation timestep, the model outputs the estimated pose of the CAD model, T, E SE(3), along with confidence score for the prediction, and tracks the object for subsequent frames.
Pose estimation is performed independently both views, and the result with the highest confidence score is selected. With the camera extrinsics of the reference view, the tracked poses are transformed into the global frame and used command the robot.
To infer the gripper open/close state, we segment the left and right gripper fingers and project their pixels into 3D space based on the predicted RGB-D sequences. The distance between the centroids of the two finger point clouds is measured: if it falls below a threshold 8 (0.10m for StoreCerealBox UnderShellf,,, 0.06m for PutSpatulaOnTatt Table. for 0.12m PutAppleFromBBwInntBii the gripper is considered closed; otherwise, it is considered open. The recovered trajectories are directly used to control the robot execute downstream tasks.
5

---

Method
Cross-view Consist.
RGB
Depth
mloU (1)
FVD.n FVD..n (
AhsRel-n (L) AhsRel-m 61-m (1) mm (1)
Task I: StoreCerealBox(ndderiiee
0.95
OURS w/o MV
497.43
607.73
0.15
0.31
0.75
0.66
4D Gaussiun (Wang al. 2024a
0.39
1208.00
1094.98
0.20
0.31
0.74
0.63
SVD Blattmann al. 2025
977.06
743.25
SVD W7 MV att
941.73
653.44
Task 2: PutSpatulaOnTable
OURS
0.69
377.68
257.70
0.03
0.07
0.98
OURS w/o MV atin
0,44
451.54
302.29
0.10
0.33
0.89
4D Gaussian (Wang al. 2024a
0.46
1241.13
815.77
0.33
0.30
0.43
0.37
SVI> Blattmann al. 2025
370.92
417.56
SVD W7 MV atm
536.02
445.68
Task 3: PlaceAppicrrommooltnoiin
OURS
0.64
490.88
36698
0.06
0.07
0.95
OURS w/o MV alin
0.26
597.05
573.73
0.14
0.49
0.76
0.30
4D Gaussian (Wang al.. 2024a
0.44
1396.10
1191.40
0.18
0.16
0.80
0.81
SVD Blattmann al 2025 SVD w/ MV alin
659.52
628.01
812.94
766.52
Multi-task Real World Dataset
OURS
0.56
384.26
331,00
0.10
0.20
0.93
0.89
OURS w/o MV altn
0.32
694,23
601.93
0.14
0.34
0.90
0.81
4D Gaussian WWnn et al. 2024a
0.00
2102.52
2608.94
0.32
1.80
0.78
0.22
SVD (Hlattmann et al. 2025
6055. 10
648.41
SVD W7 MV altn
1151.05
931.33
Table I: Multi-view 4D Video Generation Results. We compare method with baselines in terms of view consistency, RGB video generation quality, and depth generation quality. Our method consistently enables high-quality video depth generation while maintaining strong cross-view consistency on both simulated and real-world datasets.
EXPERIMENTS
4.1
TASKS
We evaluate video generation results in both the LBM simulation and real world. The LBM simulation (Institute 2025) is physics-based simulator built on Drake (Tedrake & the Drake Develop ment Team 2019) that provides realistic rendering and demonstrations for robot manipulation tasks.
Simulation Task: We evaluate three table-top manipulation tasks with both object and motion diversity (Figure 6): Unw.CrrreaaBooUnderSSSieei PurSpatulaOnTable and PlaceAppleFromi FromBBll....lll.r toBin. In StoreCereaBBoxUnnderiS and inserts it into the shelf. This task requires multi-view perception due to occlusions. PutSpatu- laOnTable involves grasping a spatula its handle from utensil crock and placing it the table. is long- horizon bimanual task where one arm picks up an apple from bowl and places it on the shelf, and then the other arm picks up the apple from the shelf and places it into bin. The simulation dataset contains 25 demonstrations per task with varying initial object poses. Each demo includes RGB-D observations from 16 camera poses, yielding 400 videos in total task. We use 12 views for training and for testing, yielding a total of 300 camera views for training and 100 camera views for testing per task. All camera poses are sampled from the upper hemisphere positioned above the workstation. More details can be found in Section A.2
robot picks up cereal box from the top of the shelf
Real-world Tasks; The dataset contains 4 tasks (Figure collected using two Franka Panda robot arms AddOrangeeSieee ToBool, PutCupOnSaucer, TwistCapOffBottle, and PutSpatulaOnTable Each task contains 20 robot teleoperation demonstrations. For each demonstration, we use two FRAMOS D415e cameras capture synchronized RGB-D sequences from different viewpoints.
4.2 4D VIDEO GENERATION RESULTS
Evaluation Metrics. We evaluate the proposed method and baselines across three key aspects: RGB video generation quality, depth generation quality, and cross-view 3D point consistency.

---

view Vn
RGB view V
Depth view Vn
Depth view Vm
Consistency
GT
OURS
OURS w/o
MV attn
4D Gaussian
SVD w/ MV attn
No Pointmap Generation
Figure Qualitative Results and Comparisons under Novel Camera Views. Our method generates gcometrically consistent videos across camera views. In contrast, baseline results often exhibit significant cross-view inconsistencies or contain noticeable artifacts in the RGB or depth predictions. Video results can be found on the project website
Video prediction: To evaluate RGB video generation quality, we compute the commonly used Fréchet Video Distance (Unterthiner et al. 2018) (FVD) between the generated video and ground-truth video. FVD-n evaluates the prediction from the reference view Un and FVD-m evaluates the predcition from view Um.
Depth prediction: To evaluate the quality of the generated depth, we extract the z-axis values from the predicted pointmaps and compare them with the ground truth depth images. We depth evaluation metrics: absolute relative eтгоr (AbsRel ly 51/3) and threshold accuracy (81 max( // y. v/ū) 1.25), where y the ground truth depth and ÿ the predicted depth.
standard
Cross-Vies 3D Consistency: To evaluate the 3D consistency of the generated pointmaps across views. we compute the mean Intersection-owe Union (mloU) on object masks. We use SAM2 (Ravi et al.. 2024) to track the robot gripper and obtain binary masks from the generated videos for both views. For each frame, we lift the gripper mask in view v2 to 3D space and then re-project it to view Un. We then compute the IoU between the bounding boxes of the projected and original gripper masks. The mloU averaged over time steps, and higher values indicate stronger 3D alignment across views.
We compare our method with prior 4D generation approaches and variants of our model evaluate generation quality and multi-view consistency. All models are trained on the same multi-view RGB-D video dataset and tested novel viewpoints sampled from 100 unseen views per task. More details can be found in Section A.2.2
Baselines.
OURS w/o MV atin: We remove the multi cross-attention mechanism in the U-Net decoder; each view is instead assigned separate decoder with no information sharing between them.
4D Gaussian Wang et al. 2024a): A baseline method that predicts single-view RGB video using a finetuned SVD model [Blattmann et al. (023) on our dataset, and then use a 4D Gaussian method, Shape of Motion (Wang al. 2024a), reconstruct dynamic 4D scene from the video.
7

---

Figure 4: Real World Multi-Task 4D Generation Results. By fine-tuning the model trained in simulation, model can generate high-fidelit, fidelity real- world RGB-D D sequences from multiple views that accurately capture robot gripper motion over time. Video results can be found on the project website
SVD (Blattmann et al. 2023); Stable Video Diffusion is state-of state-of-the-arr thetthe-rt video generation model. We finetune it dataset to predict stereo RGB video sequences.
SVD w/ MV atin: We finetune SVD on our dataset to predict stereo RGB video sequences, with additional multi-view cross-attention layers added in the U-Net decoder; similar to our method, each view is assigned a separate decoder.
Results on Simulation Tasks. Results are reported in Table I and Figure 3 compares the predictions generated by different methods for the StoreCerealBoxUndeer comparisons for other tasks can be found in Section A.4 Our method mostly achieves the best results across all tasks. For multi-view RGB video generation, it produces lower FVD scores than all baselines in most cases, indicating high temporal coherence and visual fidelity. Additionally, our method achieves the best depth prediction and cross-view consistency scores, demonstrating that the geometry-consisten supervision effectively guides the model to generate spatially aligned and geometrically coherent videos.
task. Qualitative results and
We show that multi-view cross-attention is a crucial design choice for helping the model learn 3D geometric correspondences across camera views. The variant without multi-view cross-attention OURS w/o MV attn exhibits significantly lower 3D consistency, as measured by the mloU metric. The video and depth generation quality also degrades both views. In particular, the model performs poorly on the AbsRel-m and 8, -m metrics, indicating that it fails to learn the transformation needed to generate pointmaps view v.," camera coordinate frame from view um without the support of cross-attention layers. As shown in Figure the gripper pose of OURS w/o MV alln is inconsistent across the two RGB views, and the projected gripper mask significantly misaligns with the actual gripper mask, shown in the last column.
4D Gaussian performs worse video and depth generation quality, as well as cross-view consistency, shown in Table I In Figure the generated RGB frames appear blurry from unseen viewpoints, and the predicted depth is inaccurate. This is because the model is optimized on one single generated RGB video with fixed camera view, making it difficult synthesize novel views.
For the other baselines, SVD and SVD w/ MV alin, depth is not predicted and no geometric supervision applied. While multi-view cross-attention is added SVI) w/ MV atin enable information transfer between camera views, it operates over RGB features rather than pointmaps (which naturally encode 3D structure). As result, the generated videos lack 3D geometric consistency across views. As shown in Figure 3 SVD w/ MV atin produces a noticeable gripper mismatch between the two views and the RGB generation quality is significantly worse than our method.
Results on Real-world Tasks. By fine-tuning the simulation trained model multi-tass tast dataset for around 15k steps, our method able generate high-fidelity multi-view RGB-D sequences for several real world manipulation tasks. As shown in Table [ and Figure 4 the generation results accurately capture both visual appearance and depth over time and consistently outperform baselines.
8

---

4.3 ROBOT POLICY RESULTS
We evaluate the robot policy's suc- cess rate and generalization to novel camera views across three simulation tasks. Each task tested in 30 roll- outs with unseen initial object poses and camera
Method
Task
Task
Task 3 Avg
Dreamitate (Liang et al.
2024
0.10
0.17
0.10
0.12
DP (Chi et al. 2023 DP3 et 2024 OURS
0.10
0.27
0.00
0.12
0.23
0.27
0.00
0.25
camera viewpoints sam- pled from the test dataset described in Section 4.1 The success rate of task completion is reported in Table
0.73
0.67
0.53
0.64
Table 2: Task Success Rate for Manipulation Tasks.
The video generation model takes RGB-D observations from two novel camera views as input and predicts future observations. The generated 4D video is then passed to the pose tracking model to extract 6DoF gripper poses for both robot arms. Gripper openness is inferred using the method described in Section 3.4 The robot executes actions in an open-loop manner: after each execution, the next inference is performed using the updated RGB-D observations. Each inference takes approximately 30 seconds to generate 10 future frames NVIDIA GeForee RTX 4090 GPU.
Baselines. We train baselines on the same dataset described in Section 4.1 as our method, and test on novel camera views during deployment.
Dreamitate (Liang et al. 2024): a state-of-the-art video generation method for visuomotor policy. Dreamitate uses a pretrained SVD model and finetunes it robotic task videos generate stereo RGB video predictions. Since it does not predict depth, it employs MegaPose (Labbé et al. (022) to extract the 6DoF pose of end effectors from the generated videos.
Diffusion Policy (DP) (Chi el al. 2023); a UNet diffusion model that predicts future robot end- effector trajectories, conditioned on history RGB image observations from two camera views. We randomly sample two camera views in the training dataset and encode their corresponding RGB observations using CLIP-prerrained ViT model (Radford et al., 2021), following the same encoder setup as in (Chi et al. 2023). The diffusion model outputs robot end effector trajectories in the next 16 steps of which the first actions are executed. The model evaluated on two unseen camera views sampled from the test dataset.
DP3 (Ze el al. 2024): To compare our method with a behavior cloning method that takes RGB-D information, we implement DP3, UNet-based diffusion model that predicts future robot end-effector trajectories conditioned on global 3D point clouds with colors. Following the DP baseline setup, randomly sample RGB-D images from two training camera views and aggregate their projected global point clouds as input condition. The evaluation setting is same as the DP baseline.
As shown in Table 2 Dreamitate consistently underperforms across all tasks. The lack of depth prediction and geometric consistency supervision results in lower-quality, view- niew-inconsistent video outputs, which degrade the accuracy of the extracted poses. Consequently, the downstream robot policy suffers from a high failure rate. In contrast, our method sequences and enforces 3D consistency- achieves over 50% higher task success rate on average.
jointly predicts RGB-D
In addition, Diffusion Policy struggles to generalize to unseen viewpoints, even though it is trained on demonstrations from multiple views. This is because the model does not explicitly model geometric correspondences across views and it's challenging for the model to learn view-invare niew-invariant actions by simply conditioning on features extracted from multi-view images. Using global point clouds as input, DP3 achieves slightly better performance on the CooreCerealBoUUnnerSSteee cereal box grasp accuracy with the added depth information. However, no gains are observed on the other two tasks, likely because the objects being manipulated are too small and the benefit of end-to-end learning from point clouds is not obvious. Most failures remain grasping failures on the spatula apple. On the other hand, direct gripper pose tracking on generated RGB-D videos yields higher action accuracy and improves grasping success.
task, improving
5 LIMITATION
First, our method requires RGB-D video dataset with varying camera viewpoints for training. While such datasets are easy generate in simulation, collecting them in the real world is challenging due

---

to hardware constraints and camera calibration requirements. Additionally, obtaining high-quality depth in real-world settings is often difficult. Recent advances in depth estimation from RGB images (Wen et al. 2025 Wang et al. 2025) show promising results and can be leveraged to support high quality data curation in real-world environments in future work. Second, the inference speed of the current video generation model is relatively slow compared to end-to-end behavior cloning policy, making closed-loop planning difficult. Recent flow matching (Davtyan et al.
2023 Jin et al. 2024)
or autoregressive transformers (Yin et al. 2024 Deng et 2024 Li et al.
2024b Gu 2025
have demonstrated faster video generation speed, which could lead to more reactive robot policies.
6 CONCLUSION
We present 4D video generation model that produces spatio-temporally consistent RGB-D sequences. Our method introduces geometrie-consiseer supervision during training by projecting pointmaps from one camera view into another to enforce cross-view consistency. By learning shared geometric space, the model can generate future RGB-D videos from novel viewpoints without requiring camera poses at inference time. We demonstrate improved video generation quality and 3D consistency compared to baseline methods. Additionally, the generated 4D videos can be directly used to extract robot actions using off offlth 6DoF pose tracking model, enabling higher success rate several robot manipulation tasks.
10

---

7 ETHICS STATEMENT
This work focuses on developing models for multi-view video generation and their application to robot manipulation. The primary goal is to advance general-purpose robotic capabilities in houschold and assistive settings. The datasets used in this work include synthetic data generated in simulation and real-world robot demonstrations collected in controlled lab environments; no human subjects or sensitive personal data are involved. The proposed method does not generate or rely on human biometric information. Potential societal benefits include enabling robots to better assist with daily tasks, elder care, and other beneficial applications. At the same time, we acknowledge potential misuse risks, such as manipulative content generation. To mitigate these risks, we limit the scope of this work to robotic data and will release code and models under a license that restricts harmful use.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of work. Details of the model architecture, training objectives, and hyperparameters are provided in Section the main paper and Section A.T Section A.3 Section A.4 in Appendix. The datasets used in simulation and real-world experiments. including task design and camera sampling procedure, are described in Appendix Section A.2 Evaluation metries and baseline implementations are specified in Section in the main paper. We also provide project website (https: / robot 4dgen. gi thub. To/l with qualitative results and visualizations, and we will release code, datasets, and pre-trained models upon acceptance to further support reproducibility.
ACKNOWLEDGMENTS
The authors would love thank Basile Van Hoorick and Kyle Sargent for sharing their codebase for SVD fine-tuning; Ruoshi Liu and Yinghao Xu for braintorming and technical discussions. In addition, would like thank all REAL lab members: Yifan Hou, Hojung Choi, Mengda Xu, Huy Ha, Mandi Zhao, Xiaomeng Xu, Austin Patel, Yihuai Gao, Chuer Pan, Haochen Shi, Haoyu Xiong, Maximilian Du, Zhanyi Sun, et al. for fruitful research discussions, feedback on paper writing, and emotional support. We would like to thank interns and collaborators at Toyota Research Institute: Alan Zhao, Chuning Zhu, Zhutian Yang. Naveen Kuppuswamy, Patrick Tree Miller, Blake Wulfe, and more, for project discussions and technical support. This work was supported in part by the Toyota Research Institute, NSF Award #2143601, #2037101, and #2132519, Sloan Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
REFERENCES
Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16288 2024.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512 2023.
Soon Yau Cheong, Duygu Ceylan, Armin Mustafa, Andrew Gilbert, and Chun-Hao Paul Huang. Boosting camera motion control for video diffusion transformers, arXiv preprint arXiv:2410.10800 2024.
Cheng Chi, Zhenjia Xu. Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel. Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241173666 2023.
11

---

Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint 1TXiv:/70..02554 2017.
Aram Davtyan,
Sepehr Sameni, and Paolo Favaro. Efficient video prediction via sparsely conditioned flow matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23263-23274, 2023.
Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14110 2024.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156 9172, 2023.
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503. /9325, 2025.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai. Hongsheng Li, and Ceyuan Yang. Cam- eractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 2022.
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint M.XXi:22499.22995. 2024.
Siyuan Huang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Zhengkai Jiang, Yue Hu, Peng Gao, Hongsheng Li, Maoqing Yao, and Guanghui Ren. Enerverse: Envisioning embodied future space for robotics manipulation. arXiv preprint arXiv:2501.01895 2025a.
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint S.kiv:25066000000 2025b.
Toyota Research Institute. Lbm eval: Drake-based Ibm simulation evaluation suite. https
[aareellmm__lllall__________ 2025.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song. Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprini urXiv:2410.0595. 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 555:6565--257777 2022.
Diederik Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas J Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. Advances in Neural Information Processing Systems, 37:16240-16271, 2024.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivie. Megapose: 6d pose estimation of novel objects via render & compare. arXiv preprint 2rXiv:2212.06870) 2022.
Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai. Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid-zoo: Multi-view video generation with diffusion model. Advances in Neural Information Processing Systems, 37:621 62222,, 2024a.
12

---

Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. arXiv preprint arXiv:2503.00200, 2025.
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, and Furu Wei. Arlon: Boosting diffusion transformers with autoregressive models for long video generation. arXiv preprint ar arXiv:2410.20500 2024b.
Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video generation. arXiv preprint arXiv:2406. 16862, 2024.
Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation. arXiv preprint arXiv:2501.0833 02.00118836,, 2025.
Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint arXiv:2410.20288 2024.
Thang-Anh-QUA Quan Nguyen, Nathan Piasco, Luis Roldão, Moussab Bennehar, Dzmitry Tsishkou, Laurent Caraffa, Yean-Philippee Tarel, and Roland Brémond. Pointmap conttmaap-ooniiionedd diffusion for consistent novel view synthesis. arXiv preprint arXiv:2501.0293 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PmLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan- Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/4 2024. URL
https: htttp:://arivv.or org/ abs 2408 00714
Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. Monocular depth estimation using diffusion models. arXiv preprint 288. 2023.
Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Vitor Guizilini, Yue Wang, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406. 01493, 2024.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852. PMLR, 2015.
Qi Sun, Zhiyang Guo, Ziyu Wan, Jing Nathan Yan, Shengming Yin, Wengang Zhou, Jing Liao, and Houqiang Li. Eg4d: Explicit generation of 4d object without score distillation. arXiv preprint 2XX.:2405518132. 2024.
Russ Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL/kttts://drake.mmit
Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Pre- dictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15119, 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint 1rxiv: 2018
Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. In European Conference on Computer Vision pp. 313-331. Springer, 2024.
13

---

Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny, Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294-5306, 2025.
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint arXiv:2407. 13764, 2024a.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709, 2024b.
Yikai Wang, Xinzhou Wang, Zilong Chen, Zhengyi Wang, Fuchun Sun, and Jun Zhu. Vidu4d: Single generated video to high-fidelity 4d reconstruction with dynamic gaussian surfels. arXiv preprint arXiv:2405.16822, 2024c.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Founda- tionstereo: Zero-shot stereo matching. arXiv preprint (tt::25011098995 2025.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprini urXiv:2401.00025, 2023.
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dynamic 3d content generation with multi-frame and multi- view consistency. arXiv preprint arXiv:2407. 17470, 2024.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208 2024.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint 100i 10157, 2021.
Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 21924-21935, 2025a.
Xiuyu Yang, Bohan Li. Shaocong Xu, Nan Wang, Chongjie Ye, Chen Zhaoxi, Minghan Qin, Ding Yikang, Xin Jin, Hang Zhao and Hao Zhao. Orv: 4d occupancy-centrr robot video generation. arXiv preprint arXiv:2506.03300
2025b.
Zeyu Yang, Zijie Pan, Chun Gu, and Li Zhang. Diffusion 2: Dynamic 3d content generation via score composition of video and multi-viww diffusion models, arXiv preprint arXiv:2404.021443 2024a.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408. 060.02 2024b.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. arXiv preprint arXiv:2412.07772. 2024.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXh5:2403.03954, 2024.
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-views video diffusion model for 4d generation. Advances in Neural Information Processing Systems, 37:15222-15295, 2024.
14

---

Sijie Zhao, Yong Zhang. Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. Advances in Neural Information Processing Systems, 37: 17:12847-1287, 2024.
Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprinl arXi::250..0999. 2025.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412. 204044 2024.
Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. arXiv preprint ar 2rXi:2554.002992 2025.
15

---

A APPENDIX
In Section A. we provide more details of our 4D generation model architecture. In Section A.2 we describe the camera sampling method used to generate the multi-view RGBD video dataset for training 4D generation model. In Section A.3] we provide details such as compute resources requirements and hyperparameter choices for model training and inference. In Section A.4 we provide more
quantitative and qualitative results of our method and additional basclines. In Section A.5 the LLM usage of our work.
we discuss
A.1 MODEL DETAILS
In 3.1 and 3.2, we discussed that the model takes in historical video frames {0 O.3 and historical pointmaps (X,-h+1,' X,} from both the native view and the second view Um- In practice, we use the latest observation, and repeat it h = 10 times to match the number frames that needs to be predicted, following the implementation in SVD (Blattmann et al. 2023).
Each pair of RGB image and pointmap condition Oj Xy, where E {n, m). is independently encoded using separate VAE encoders for images and pointmaps, detailed in S The image VAE encodes each RGB frame into latent feature of shape C x w' x h', where h = 10 the temporal horizon, c=4 is the latent channel size, w'=32 and ='=40 spatial dimensions of the latent feature maps. Similarly, the pointmap VAE encodes pointmap into shape c x w' x h These encoded image and pointmap features are then concatenated along the channel axis with the corresponding noisy latents of future images and pointmaps, yielding combined input tensor of shape 4c w' h (h X 16 x 32 40), which is fed into the U-Net diffusion model.
To allow information sharing between the two diffusion branches as shown in Figure add one cross-attention layer after each decoder block in the U-Net diffusion model for the branch corresponding to view This results added cross attention layers. As illustrated in 5 the query to the cross-attention layer are feature map tokens (feature at each pixel in the feature map) output by the decoder block in view Ums where ç' is the feature dimension, h' and w' are spatial dimensions of the feature map; the key and value are feature map tokens output by the corresponding decoder block in the native view U, ' branch. The updated fea- tures are passed the next decoder block in view Um- The cross-attention layers capture spatial correspondences between the views through our geometric-consistent supervision mechanism.
view
view Vin
Decoder block
Decoder block 1
h' X
h' W
k.v
Crose-eltention leyer
Decoder block
Decoder block 2
Figure 5: Multi View Cross-Attention. We insert a cross attention layer after each decoder block in the U- Net diffusion model for view Tm- By cross-attending features in the native view un the cross-aticntion layers allow information sharing between view branches.
We use pre-trained weights in SVD to initialize the denoising U-Net model and find that using pre-trained weights helps the model converge faster. To get better prediction quality around the robot gripper, which is important for action extraction later, we apply a re-weighting mechanism in the diffusion loss. Concretely, we use binary masks (in simulation, the object segmentations are provided; in real world, we use SAM2 (Ravi et al. 2024)) of the robot gripper region and downsample it by factor of 8 to match the resolution the latent space while still keeping the spatial correspondence. The resulting downsampled masks provide spatial weight map at each timestep t', denoted wy (t'), which is incorporated into the joint diffusion loss mentioned in $3.3 as follows:
I++h Σ
1 (ww(()))))
Cair("")
(iir(1") +A
LaD
M=t+1
RGB lnss
pointmap loss
where 1 (m)) is an indicator function that activates the pixel value on the spatial weight map is 1 and 0 otherwise. We add this indicator base weight of 1, effectively doubling the contribution of
16

---

loss terms gripper regions. This weighting encourages higher prediction accuracy in areas eritical for gripper pose estimation in the policy extraction phase.
A.2
DATASET DETAILS
A.2.1
SIMULATION TASKS
Task 1: StoreCerrelBoxxndddSheeffr
Task PutSpatulaOnTable
Pick cereal box
Insert cereal box into shelf
Pick up spatula
Place spatula table
Task PlaceApleeFronBoollntdiin
Pick up apple from bowl
Place apple on shelf
Pick up apple from shelf
Put apple
Figure 6: Simulation Tasks Evaluation.
A.2.2 CAMERA SAMPLING
To sample cameras for rendering multi-view RGBD videos, first sample camera positions within a half-sphere shell defined by an inner radius (r1 = 0.7m) and outer radius (r2 = 1.2m), with the center being the origin of the world coordinate system (center of the table). We restrict the range of the camera positions within the area between 0.2m I x < 0.6m, -0.5m S S 0.5m, and 0.7 < 1.2m, shown in Figure (a-c). The world coordinate system is shown in Figure (d).
StrvOanze BulistaStel
0d Plazsdgsleiic Bowil duCln
Sinkkd Cuoddmate
Figure 7: Camera Sampling Visualization. We randomly sample 16 camera poses per episode using our proposed technique. (a)-(c show example camera poses for each task, with green cameras used for training and red for evaluation. (d) shows the simulation world coordinate frame.
17

---

A.2.3 REAL WORLD TASKS
Task 1: TdderrangeelliessccBoow
Pick orange slices
Transfer orange slices to
Place bowl table
Task 2: PutCupOnSaucer
Pick up saucer
Move saucer to cup
Pick up cup
Put cup saucer
Task 3: TwistCapOfiBottte
Pick up bottle
Twist cap open
Remove cap
Place bottle on table
Task 4. PutSpatulaOnTable
Pick up spatula
Place spatula on table
Figure Tasks in Real-World Dataset.
A.3 TRAINING DETAILS
The 4D generation model described in 3 trained separately for each task in $ $ approximately 60 epochs using 4 NVIDIA RTX A6000 GPUs (48GB memory each). We fine-tune the full U-Net backbone of the SVD (Blattaann et al. 2023) model with learning rate of 1 10 using the Adam W optimizer (8₁ = 0.95, B2 = 0.999, 10-8, weight decay x 10-6) and a batch size of 4. The image and pointmap VAE encoders are frozen during diffusion model training.
At inference time, we apply the standard EulerEDMSanpler (Karras et al. 2022) with 25 denoising steps. For robot policy deployment, both the generation model and the pose tracking model are run on single NVIDIA GeForce RTX 4090 GPU.
Method OURS
Inference Time
Training Memory of Trainable Parameters
30.0
47 G
2.4 B
OURS w/o MV attn
29.3
46.5 G
2.38 B
4D Gaussian
2
2813 M
856,774
SVD (Dreamitate)
13.4 s
45.8 G
1.54 B
SVD w/ MV attn
15.1
46.3 G
2.4 B
Table Quantitative comparisons with baselines terms of inference time, GPU memory compute resource consumption and parameter count.
We report quantitative comparisons with baselines in terms of inference latency, GPU memory / compute resource consumption, and parameter count. Our method requires 30.0s inference time with 47G training memory and 2.4B parameters: adding multi-view cross attention layers only slightly increases each metric yet yields more cooneeryy-coniiteer contrast, 4D Gaussian is significantly faster (2s) and lighter (2813M memory, 856,774 parameters), but produces poor quality results for novel view rendering and also requires generated single-view video input, while method directly predicts future frames based single RGB-D image per view.
and high-quality generation results. In
18

---

We acknowledge that the higher inference time reflects the cost of predicting both RGB and depth modalities, but this trade-off results in stronger video fidelity and geometric consistency. Looking ahead, improvements in efficiency can be explored by applying the geometry consistency objective to faster generative models, such as leveraging 3D VAFs (Zhao et al. 2024) to reduce temporal dimensions of the letent tokens, adopting one-step few-step diffusion models (Lin al. 2025). or employing autoregressive transformers with lower latency (Huang et al. 2025b). On the policy side, a hierarchical structure (Black al. 2023 Wen et al., (023) could further mitigate the cost of dense video inference: instead of extracting and executing all poses from generated RGB-D frames, the model can act high-level planner where sparse predicted frames serve as subgoals, while low-level controller or learned policy fills in reactive actions. This design would reduce the number of generated frames required for action planning, thereby accelerating inference while still preserving the advantages of geometry-ave video generation for downstream robotic tasks.
A.4 ADDITIONAL RESULTS
view un
vilow
Viee Mn
vireo
Viewe
view om
Figure 9: Qualitative Multi-View Video Generation Results in Simulation. We show temporal results generated by our 4D video generation model across three robot manipulation tasks. With consistetn supervision and joint temporal 3D consistency optimization, our model able to output pempooltymporally consistent videos across camera views with high visual fidelity in unseen views.
In Figure 9 and Figure TO we show generated RGB video sequences on both simulation and real world tasks using our proposed 4D generation model. With ccometry-consistee supervision and joint temporal and 3D consistency optimization, our model able to output spatio-temporally consistent videos across camera views with high visual fidelity. We also show baseline comparison results on the ViaceAAplleioonBBollinte method consistently achieves the best RGB video and depth generation quality, with high multi-view consistency. Baseline results often exhibit significant cross-view inconsistencies (marked in red) contain noticeable artifacts in the RGB depth predictions.
task in Figure II and PutSpatulaOnTable task in Figure 12 Our
19

---

view Vn
view Vm
view vn
view vm
view Vn
view Vm
view Vi
view vm
Figure 10: Qualitative Multi View Video Generation Results Real World. We show temporal results generated by our 4D video generation model across four real-world robot manipulation tasks. By fine-tuning the simulation-traiin trained model for around 15k steps, our model is able to output temporllyyyy consistent videos across camera views with high visual fidelity.
A.5 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LL.Ms), specifically OpenAl's ChatGPT 5, to assist with writing and editing the manuscript. The models were used primarily for improving the clarity, grammar, and readability of the text, including rephrasing sentences, polishing explanations, and standardizing style. Anthropic's Claude was used to assist in generating visualization-related code, such as for camera pose and depthmap visualization the figures. All technical content, experimental design, analysis, and conclusions were conceived, implemented, and validated by the authors without the use of LLMs. The authors have carefully reviewed and verified all content produced with the assistance of LLMs to ensure accuracy and originality.
20

---

RGB view Vn
RGB view Vin
Depth view Vn
Depth view Vm
Consistency
GT
OURS
SURS WM MV attn
4D Gaussian
SVD w/ MV attn
No Pointmap Generation
SVD
Figure 11: Qualitative Results of Pronooioid task. Our method achieves the best RGB video and depth generation quality, with high multi-view consistency. Baseline results often exhibit significant cross-view inconsistencies (marked in red) or contain noticeable artifacts in the RGB or depth predictions.
A.6 POSE TRACKING ERROR ANALYSIS
The pose tracking error arises from two sources: (1) the RGB-D generation quality of the video model, and (2) the detection accuracy of FoundationPose. To disentangle these factors, we additionally evalu- ate Foundation Pose directly on ground-truth RGB-D images to isolate its performance independently of any generation errors. As shown in Figure [3] we visualize one representative trajectory from each task and report the per-action MSE between the detected pose and the ground-trute pose throughout the task. We also include the frames with the top five highest pose-tracking errors to illustrate the characteristic failure cases.
We observe that the MSE enror of pose tracking is generally around or below 0.001 for both grippers across tasks, indicating good pose-trad tracking accuracy. The only notable exception is the right gripper in the PutSpatulaon Table task and the left gripper in CrorCerralBoxUnddeShhee towards the end of the task, where the occasionally peaks around 0.003.
21

---

RGB view Vn
RGB view Vm
Depth view Vn
Depth view Vm
Consistency
GT
OURS
OURS w/O MV attn
40 Gaussian
SYD WM MV
No Pointmap Generation
SVD
Figure 12: Qualitative Results PutSpatulaOniTable task. method achieves the best RGB video and depth generation quality, with high multi-view consistency. Baseline results often exhibit significant cross-view inconsistencies (marked in red) or contain noticcable artifacts in the RGB or depth predictions.
A.7 SCALING TO MORE INPUT VIEWS
Our main evaluation focuses on simultaneously generating two views. We additionally study how the proposed framework scales when incorporating
input views inference time.
Without retraining, the model can incorporate third (or more) view(s) at inference time by performing an additional forward pass between the reference view and each new view. This approach increases inference time approximately linearly with the number of views, but does not require architectural modifications or additional training. Designing more efficient mechanisms for large-scale multi -vew inference is an interesting direction for future work, similar in spirit to how Fast3R (Yang et al. 2025a) extends DUSt3R to handle thousands input images.
View
mIoU
FVD
AbsRel
51
View
515.53
0.06
0.95
View 2 (w/ View 1) View 3 (w/ View 1) 0.54
0.62
566.86
0.09
0.95
662.83
0.12
0.89
Table 4: Performance generating three views simultaneously on StoreCerealBounddeSShelll
22

---

Figure 13: Analysis of 6-DoF pose-tracking errors. Left: per-action MSE curves for left and right grippers across one representative trajectory from each of the three tasks. Right: visual examples of the five highest-error frames for each task.
Three-view Evaluation. We evaluated the performance of incorporating a third predicted view on the StoreCerealBox UnderShelf task. In this setting, the model simultaneously generates videos in three randomly sampled, unseen viewpoints, and the reported metrics are averaged across different initial RGB-D conditions.
Table 4 summarizes the results. Although performance degrades slightly, the model is still able to produce RGB-D videos with good visual quality and cross-view consistency.
23


Reviews:
[{"review_id": "LTrbF0Aq9x", "reviewer": "Reviewer_QvcZ", "summary": "This paper proposes a robot control strategy based on image-conditioned 4D video generation.\n\nA 4D video diffusion model is conditioned on an RGB-D image pair; this generates a pair of pointmap videos (eg, that can be deprojected into a dynamic point cloud). The authors then propose to run FoundationPose-based pose tracking on these generated videos to extract gripper poses, which can then be executed on a robot.\n\nThis is similar to prior work in using RGB video for robot control (eg, Dreamitate), but includes extra geometry supervision for the video model.\n\nExperimental results show that this results in robot control policies that generalize better to unseen initial object poses and novel camera viewpoints than methods like Dreamitate and Diffusion Policy.\n\nOverall: the core idea of the paper seems simple but valuable. I'm suggesting a weak accept.", "questions": "How robust is FoundationPose on generated 4D video? Are any of your task failures caused by perception failures, for example if you cannot successfully track the gripper pose or open/close state?\n\nThe paper currently only presents task success rates for unseen object poses/novel camera viewpoints. It makes sense to me that methods with less 3D inductive bias would fail here. Do you have success rates for *seen* object poses and/or *fixed* camera viewpoints? It would be helpful, for example, to know how well the 4D video-based approach does compared to RGB video or vanilla diffusion policy when there's less of a train/test gap.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I found this to be a  well-written paper, with a clear formulation and convincing empirical results. There's reasonably thorough evaluation across stages: for both 4D generation and for learned robot policies.\n\nI think the robot learning community would benefit from these results; it's a nice existence proof for one way video/world models might be useful for robot learning.", "weaknesses": "While the baselines are reasonable, they do feel slightly \"set up to fail\". For video generation the baselines are RGB only while the proposed method is trained on RGB-D; same for policy rollout. The core message of the paper, however, is in part that the geometry component is critical so perhaps this is fine.\n\nWhile the approach is elegant and the results are strong, the system depends heavily on multi-view RGB-D data and an external pose tracker (FoundationPose). This raises questions about scalability to real-world deployment, especially in less instrumented or monocular setups.\n\nThe reliance on diffusion-based generation also makes inference relatively slow, which (for now) limits applicability to closed-loop control or online replanning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Prh85nEsOu", "reviewer": "Reviewer_ctzc", "summary": "This paper proposes a novel 4D video generation framework that jointly enforces temporal coherence and multi-view 3D geometric consistency for robotic manipulation tasks. The core idea is to supervise a diffusion-based video generation model using cross-view pointmap alignment, inspired by DUSt3R, to ensure that predicted RGB-D sequences from different camera views correspond to a shared 3D scene representation. The method does not require camera poses at inference time, yet it generalizes to novel viewpoints. The authors demonstrate that the generated 4D videos can be used with off-the-shelf 6DoF pose trackers (e.g., FoundationPose) to extract accurate robot end-effector trajectories, enabling successful execution of manipulation policies in both simulation and real-world settings. Experiments across three simulated and four real-world tasks show consistent improvements over strong baselines in video quality, depth accuracy, and cross-view consistency.", "questions": "1. Camera Pose at Training: The method requires camera extrinsics during training to project pointmaps into a shared frame. How sensitive is performance to errors in these extrinsics? Could the system be made fully self-supervised (e.g., via structure-from-motion)?\n\n2. Generalization Beyond Grippers: The pose tracking focuses on the robot end-effector. Can the same 4D video be used to track object poses (e.g., cereal box, apple) for object-centric manipulation? This would broaden applicability.\n\n3. Failure Modes: In Table 2, Task 3 has the lowest success rate (~53%). What are the dominant failure modes? Are they due to video generation errors, pose tracking inaccuracies, or open-loop execution?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Originality: The integration of cross-view pointmap alignment into a video diffusion model for 4D generation is novel. Unlike prior 4D methods that assume known camera poses or operate on static scenes, this work handles dynamic, multi-object manipulation without pose inputs at test time.\n2. Quality: Experiments are thorough, covering both simulation (LBM) and real-world domains, with ablations, multiple baselines (SVD variants, 4D Gaussian), and downstream policy evaluation.\n3. Clarity: The problem formulation, method description, and results are presented with exceptional clarity. Figures and tables are informative and well-designed.\n4. Significance: Enables pose-free novel-view video generation for robotics—a practical advance for real-world deployment where extrinsic calibration is often unavailable or unreliable. The demonstrated success in policy extraction via off-the-shelf trackers lowers the barrier for applying generative models in robotics.", "weaknesses": "1. Computational Cost: Inference takes ~30 seconds per 10-frame rollout (Table 3), which limits real-time or closed-loop use. While acknowledged in §5, more discussion on latency-accuracy trade-offs or potential optimizations (e.g., sparse prediction, distillation) would strengthen practical impact.\n\n2. Real-World Data Scale: The real-world dataset includes only 20 demos per task. While fine-tuning from simulation helps, it’s unclear how performance scales with more diverse real data or more complex scenes (e.g., clutter, deformables).\n\n3. Dependency on FoundationPose: The policy pipeline assumes access to a high-quality 6DoF tracker and a CAD model of the gripper. Generalization to arbitrary tools or unmodeled objects is not explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVrvBJHQbv", "reviewer": "Reviewer_b1Za", "summary": "This paper proposes a geometry-aware 4D video generation model for robot manipulation. The method introduces cross-view pointmap alignment as a geometric supervision during training. The geometric constraint allows the model to learn a shared 3D scene representation, enabling the joint optimization of spatial and temporal consistency.", "questions": "1. The current method uses geometric supervision via feature passing and cross-attention from view $v_n$ to view $v_m$. Have the authors investigated the effect of $v_m \\to v_n$ or $v_n \\rightleftarrows v_m$ on geometric consistency and final generation quality? Additionally, a sensitivity analysis on $\\lambda$ should be included to determine if there is a trade-off between FVD and mIoU performance.\n2. Since the current evaluation is primarily based on two views, what is the performance of this framework when the number of input views is increased (e.g., three or four views)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method achieves geometric consistency in the generated videos across multiple views through cross-view geometric supervision and cross-attention mechanism.\n2. The model shows a significant improvement in task success rate for manipulation tasks compared to baseline methods.\n3. The generated 4D videos can be directly combined with an off-the-shelf 6DoF pose tracker to extract robot end-effector trajectories.", "weaknesses": "1. The authors need to fully explain the difference between their proposed method and the joint spatio-temporal consistency optimization methods mentioned in references [3, 4]. The authors should specifically show how the proposed approach is better suited for the multi-object, dynamic robot manipulation scenes.\n2. In Table 1, the $FVD-{n}$ scores for Task 2 and Task 4 are not significantly different from the SVD finetuning baseline. More results from additional tasks are needed to verify the advantage of the proposed method over SVD finetuning.\n3. The current experiments mainly focus on rigid object manipulation, lack experimental evaluation on non-rigid objects (e.g., cloth).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lYVHdHz24a", "reviewer": "Reviewer_RXK1", "summary": "This paper proposes a geometry-aware 4D video generation model tailored for robot manipulation, which enforces multi-view 3D consistency during training via cross-view pointmap alignment and leverages pretrained video diffusion models for temporal coherence, enabling the generation of spatio-temporally aligned RGB-D sequences from novel viewpoints without camera pose inputs. It evaluates the model on both simulated (e.g., StoreCerealBoxUnderShelf) and real-world (e.g., TwistCapOffBottle) robotic tasks. Additionally, the predicted 4D videos can be used with an off-the-shelf 6DoF pose tracker (e.g., FoundationPose) to extract robot end-effector trajectories.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Enables geometry-consistent 4D video generation for multi-object dynamic robot manipulation scenes (a gap in prior 3D-aware methods limited to single objects and static backgrounds) by enforcing cross-view pointmap alignment during training, ensuring spatio-temporal consistency across novel camera viewpoints without relying on camera poses at inference.\n2. Integrates pretrained video diffusion models’ temporal priors with a dedicated geometry-consistent loss for pointmaps, achieving joint optimization of RGB video quality, depth accuracy, and multi-view 3D consistency.\n3. Bridges 4D video generation to practical robot manipulation: predicted RGB-D sequences can extract robot end-effector trajectories via an off-the-shelf 6DoF pose tracker (e.g., FoundationPose) and infer gripper open/close states, leading to higher task success rates than visuomotor policy baselines.", "weaknesses": "1. Relies on multi-view RGB-D datasets with varied camera viewpoints for training, which are challenging to collect in real-world settings due to hardware constraints and calibration requirements, and high-quality depth data acquisition in real scenarios remains difficult.\n2. The inference speed of the video generation model is relatively slow, making closed-loop planning for robot manipulation impractical compared to end-to-end behavior cloning policies.\n3. The baseline comparisons lack direct competition with state-of-the-art methods specifically designed for multi-object dynamic robot manipulation scenarios; selected baselines (e.g., 4D Gaussian, SVD) either have different task scopes or lack 3D consistency modeling, weakening the persuasiveness of performance superiority.\n4. Key implementation details for reproducibility are insufficiently disclosed, such as the specific camera sampling parameters (e.g., exact pitch/yaw ranges), training hyperparameter schedules (e.g., learning rate decay strategy), and threshold values for gripper state inference (e.g., distance threshold δ), which may hinder result validation.\n5. The multi-view cross-attention mechanism, a core component for 3D consistency, lacks unique design details; it is not clarified how it adapts to pointmap geometric features (e.g., whether attention weights correlate with 3D distances), making it indistinguishable from generic cross-attention modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)