Title: Building 3D Representations and Generating Motions From a Single Image via Video-Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scene Representation

Motion Generation
Abstract: Autonomous robots typically need to construct representations of their surroundings and adapt their motions to the geometry of their environment. Here, we tackle the problem of constructing a policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from a single image often involves monocular depth estimation. Developments in depth estimation have given rise to large pre-trained models such as \emph{DepthAnything}. However, using outputs of these models for downstream motion generation is challenging due to frustum-shaped errors that arise. Instead, we propose a framework known as Video-Generation Environment Representation (VGER), which leverages the advances of large-scale video generation models to generate a moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into a pre-trained 3D foundation model to produce a dense point cloud. We then introduce a multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the representation. We extensively evaluate VGER over a diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of a scene, all from a single RGB input image.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pipeline for generating robust collision-free motion policies from single images. Their framework consists of three parts: lifting an image into a complete 3D scene, training an implicit unsigned distance field (UDF) representation of the scene, and incorporating the implicit metric field into the motion planning for collision avoidance. To lift an image into a 3D scene, they observe that the simplest implementation of unprojected depth map suffers from two artifacts — noisy depth points along surface boundaries and incomplete surfaces — and use a camera-conditioned video generation method to generate unseen views and a feedforward SfM method (dust3r) to lift it into a 3D point cloud. They then train an implicit field on the resulting 3D point cloud and incorporate the UDF into the motion planning for collision-avoidance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Good direction of using 3D foundation models in robotics motion planning with minimal/no training overhead\n- Paper writing is clear and easy to follow\n\nWeaknesses:\n\n- Engineering focused — while I agree the framework is useful, I do think it is a bit plug-and-play and it’s worth noting there perhaps isn’t much technical novelty\n- Clunky and expensive — the framework has a few distinct moving pieces without interaction between them: the video generation model, feedforward SfM model, per-scene UDF model. Perhaps most clunky is that the UDF needs to be trained per scene. Either inferring this feedforward or avoiding with some heuristic geometry could simplify the pipeline and make it more practical\n- Experiments and ablations and pipeline could be improved — currently the authors compare their motion plans vs those from the unprojected depth anything results. I personally would be interested in the following ablations:\n    - what about heuristic geometries like meshes or using the raw point cloud from DUST3R\n    - heuristically cleaning the pointcloud with a depth-gradient thresholded mask like e.g. MoGe does in their visualizations\n    - using the pointcloud from an image-to-pointmap representation like single-frame VGG/DUST3R\n    - using some unseen-geometry heuristic like don’t cross any part of space unobserved (implemented as some geometry proxy if feasible)\n\nQUESTIONS:\n- Is there a simpler setup you can use? I would like to see more ablations in-between unprojected depth estimate and your full model (mentioned more in-depth in weaknesses section)\n- Why not just use resulting DUST3R point cloud in motion planning? (mentioned above)\n- How long does training the per-scene MLP take? Those details should be mentioned\n\nLIMITATIONS:\n- Method requires per-scene training of an MLP before motion plan can be generated, which is impractical for most robotics settings\n- Hallucinated content is used with no uncertainty measuring (one could e.g. render multiple videos and check variance or generated videos as proxy for uncertainty)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VGER, a novel pipeline that generates environment-aware motion policies from a single RGB image. The key idea is to leverage large video generation models (e.g., Seva) to synthesize a multiview video from the input image, and then reconstruct a dense 3D scene using a 3D foundation model (e.g., DUSt3R). An implicit distance field is trained using multi-scale noise contrastive supervision, and motion policies are then derived using a Riemannian metric field constructed from this geometry. The method is evaluated on diverse indoor/outdoor scenes, demonstrating improved reconstruction quality and smoother, collision-free motion trajectories compared to monocular depth baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: (1) The integration of pretrained video generators with 3D foundation models to create multiview depth from a single image is an innovative and elegant approach. (2) The modular design (video synthesis → point cloud → implicit field → motion policy) is clean and extensible. (2) The qualitative results is compelling. Visuals across several scenes show smoother, safer motion paths and less noisy geometry compared to strong baselines like DepthAnything-V2. (4) The construction of the implicit field via multi-scale noise denoising is well-grounded and connected to score-based modeling.\n\n**Weaknesses**: The authors acknowledge hallucinations from video generation but provide no mechanism to handle or flag unreliable geometry during motion planning. Also, no real-world to sim-to-real validation is conducted, all experiments are in synthetic environments. Demonstration on it would work on actual robots or noisy RGB images can be helpful. The Riemannian field used to guide motion is hand-designed. There’s no ablation or learning-based alternative explored. The benchmark scenes are mostly static and small-scale. The method is not evaluated in dynamic or large outdoor settings.\n\nQUESTIONS:\nHow robust is VGER to real-world image conditions (e.g., noise, lens distortion)? Could you provide an ablation or example on natural images? Have you considered learning the Riemannian metric instead of hardcoding it? This could help generalize better in cluttered or ambiguous scenes. Could confidence maps from the 3D foundation model be propagated into the motion planning stage to avoid unreliable areas? Do you plan to test this on a real robot or photorealistic simulator?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of collision-free motion generation. It leverages large-scale video generation models to generate multiple views for the static scene, which is used to construct the 3D dense point cloud through 3D foundation models. Then, it generates collision motion with motion models.\n\nSTRENGTHS AND WEAKNESSES:\n* Strength\n* The method demonstrates strong performance in scene construction. The approach achieves impressive results in building dense 3D point clouds by combining video generation models with 3D reconstruction models.\n* The paper is well-written and good presentation, making the methodology and results accessible and understandable.\n* Weakness\n* The proposed 3D construction approach is closely similar to existing work, particularly ViewCrafter [1], which similarly integrates video diffusion models with 3D reconstruction for novel view synthesis. This similarity raises concerns about the novelty of the contribution.\n[1] ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis.\n* The baseline comparisons are relatively simplistic. The paper mainly contrasts against 3D reconstruction methods such as Dust3R and VGGT applied to single images, which may not reflect the improvement of the methods.\n* And there is no comparison of motion generation methods with other works. \n* The actual impact of motion generation on downstream robotics tasks is not thoroughly demonstrated. The paper lacks a clear explanation or empirical evidence on how the generated motions contribute to improving performance or capabilities in practical robotic applications.\n\nQUESTIONS:\n* I'm not familiar with the related works about motion generation. Is it the first work to generate motion based on a 3D dense point cloud?\n* What's the efficiency of the multi-scale noise motion generation?\n* What's the hyperparameter of the motion generation models? What's the number of the point? If it's a sparse point cloud, how does the motion generation model perform?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages the additional information provided by a video generation model to perform 3D reconstruction from a single image using Dust3R, and subsequently builds an implicit representation to enable a collision-free motion generation algorithm. Specifically, after generating a video to obtain multi-view images, it merges the point clouds estimated by Dust3R (filtered by confidence) across views to reconstruct the 3D scene. It then trains an implicit distance field with multi-scale noise, and finally generates a motion plan based on this representation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The method effectively leverages a video generation model to fill in the missing information from single-image Dust3R point clouds, achieving impressive 3D reconstruction results.\n2. This is the first work to address the task of generating robot collision-free trajectories from single images by integrating perception and motion planning processes.\n3. The paper introduces a method to train an implicit distance model and continuous metric field that can efficiently generate motions based on the environment’s representation.\n\n**Weaknesses:**\n\n1. There are already many existing frameworks that use video generation (or multi-view diffusion) for 3D reconstruction [1, 2, 3], so the novelty of this contribution is questionable. At minimum, the paper should include a thorough literature review and provide a clear justification for why this approach offers advantages over prior work.\n2. As noted in the limitations, video generation models fail to produce accurate representations of parts of the scene that are not visible in the input image. The success of the method thus heavily depends on the input image capturing most of the scene.\n3. The reliability of the approach is uncertain. For robot motion planning, the required reliability is higher than for typical vision tasks. For example, as shown in Figure 5, even a human observer cannot confidently infer what lies behind the stone—there are many possible variations. Relying on the “imagination” of a video generation model may not provide sufficient safety. Active perception strategies might be necessary.\n4. A final concern is the computational cost and speed of the method. Running foundation models can be time-consuming and resource-intensive. In many cases, using a simpler solution, such as a depth sensor, might be more practical.\n\n[1] Gao, Ruiqi, et al. \"Cat3d: Create anything in 3d with multi-view diffusion models.\" arXiv preprint arXiv:2405.10314 (2024).\n\n[2] Yu, Wangbo, et al. \"Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.\" arXiv preprint arXiv:2409.02048 (2024).\n\n[3] Ren, Xuanchi, et al. \"Gen3c: 3d-informed world-consistent video generation with precise camera control.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nQUESTIONS:\n1. **Implicit scene description**\n   Are there prior works using implicit scene representations for collision avoidance? How does this method compare in terms of accuracy or efficiency?\n2. **Failure cases / initial image impact**\n   Can you show more failure cases or analyze how the initial image content affects reconstruction and motion planning?\n3. **Reliability**\n   Are there ways to improve reliability — for example, by incorporating active perception, sensor fusion, or uncertainty estimation?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}