Building 3D Representations and Generating Motions From a Single Image via Video-Generation
Authors: Weiming Zhi
Ziyong Ma
Tianyi Zhang
Matthew Johnson-Roberson
Venue: NeurIPS 2025 poster ()
Keywords: Scene Representation
Motion Generation
Primary area: applications

Abstract: Autonomous robots typically need to construct representations of their surroundings and adapt their motions to the geometry of their environment. Here, we tackle the problem of constructing a policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from a single image often involves monocular depth estimation. Developments in depth estimation have given rise to large pre-trained models such as \emph{DepthAnything}. However, using outputs of these models for downstream motion generation is challenging due to frustum-shaped errors that arise. Instead, we propose a framework known as Video-Generation Environment Representation (VGER), which leverages the advances of large-scale video generation models to generate a moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into a pre-trained 3D foundation model to produce a dense point cloud. We then introduce a multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the representation. We extensively evaluate VGER over a diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of a scene, all from a single RGB input image.

Building 3D Representations and Generating Motions From a Single Image via Video-Generation
Weiming Zhi 1,2,3
Ziyong Ma 3
Tianyi Zhang
Matthew Johnson-Roberee 2,3
School of Computer Science, The University of Sydney, Australia. 2 College of Connected Computing, Vanderbilt University, TN, USA. Robotics Institute, Carnegie Mellon University, PA, USA. 4 Aurora, USA.
Correspondence to W. Zhi (Weiming. ze @sydney eeayyddey.dd.... edu. au)
Abstract
Autonomous robots typically need to construct representations of their surround- ings and adapt their motions to the geometry their environment. Here, we tackle the problem of constructing policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from single image often involves monocular depth estimation. De- velopments in depth estimation have given rise to large pre-trained models such DepthAnything. However, using outputs of these models for downstream tion generation is challenging due to frustum-shaped errors that arise. Instead, we propose framework known (VGER), which leverages the advances of large-scale video generation models to generate moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into pre-trained 3D foun- dation model produce dense point cloud. We then introduce multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the represen- tation. We extensively evaluate VGER over diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of scene, all from single RGB input image.
Iideo-Generation Environment Representation
Introduction
Autonomous robots operating in unstructured environments maintain a representation of their sur- roundings to enable reliable motion planning. Building complex environment representations gener- ally requires large sets of images [T3, and often depth measurements 5] In this work, we push the boundaries of data efficiency by constructing both scene representation for collision avoid- ance and a downstream motion generation model from a single RGB image, by leveraging recent advances in rre-trained video generation and 3D foundation models.
common strategy for recovering 3D structure from a single image is estimate per-pixel depth. Recent advances have yielded powerful monocular depth estimators, most notably DepthAnything: However, the depth maps they produce often exhibit crustum-shaped artifacts object boundaries, rendering them unsuitable for collision avoidance. To overcome these limitations, we introduce Video-generation Environment Representation (VGER), which sidesteps direct monocular depth prediction by conditioning a large, pre-trained video model on the input image to generate a se-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Input image
(b) DepthAnything V2 image (1) and structure (r)
(c) VGER
Figure 1: (a): Single original image used construct the environment; (b): Predictions by DepthAnything-V2 [9]. Predicted depth image of the left, and extracted structure on the right. We observed errors that are frustum-shaped "tails" in the extracted structure; (c): Our proposed VGER does not suffer from these errors, and completes regions blocked from the original view. These arti- facts free space make it impossible to use the representation for motion generation.
quence of consistent novel views. We then apply a 3D foundation model such as DUSt3R (6 to fuse these views into dense representation that faithfully captures scene geometry.
Beyond representation, our primary contribution lies in coupling the structure extracted from the sin- gle input image with a structured aenienngeneration framework. We integrate our reconstruction into motion generation by deriving an implicit unsigned distance field from the 3D foundation model's output. We do so efficiently via multi-scale perturbations. By sampling perturbations of point posi- tions at varying noise scales and contrasting them against the dense output, we recover an implicit, unsigned distance field that defines continuous distance function over the workspace. From this, we construct smoothly varying metric field that encodes proximity obstacles. Finally, we em- bed this environment wevironment-dependent metric field into a motion policy framework, in a similar manner to Riemannian Motion Policies and Diffcomorphic Templates [8] in which the environment- induced metric modulates nominal dynamical system. The result is policy that generates motion and is modelled by non-linear dynamical system that adapts fluidly to the reconstructed geometry, producing smooth, collision-fre free trajectories directly from a single input image.
Concretely, our technical contributions within VGER include,
1. A pipeline that harnesses pretrained video synthesis and 3D foundation models to generate multi-view videos from single RGB image, producing dense, artifact-free reconstructions that overcome the limitations of monocular depth estimators;
2. A multi-scale unsigned distance field directly from the generated output;
denoising approach to extract a globally detailed implicit
3. A novel metric-modulated motion generation framework that embeds the implicit repre- sentation into a metric field that is dependent on the environment. This ensures that any nominal dynamical system coupled with the lbstacle-induced curvature metric field will yield smooth, collision-fii frree trajectories in real time.
2 Related Work
Pre-trained Models to Recover 3D Structure RGB Images: Estimating the structure within single RGB image typically requires the estimation of depth, i.e. monocular depth estimation. Ad- vances in deep learning have lead to large pre-trained models capable of predicting depth im- age given an RGB input. Well-known models include Marigold [10). DepthAnything [T]. and its follow-up model DepthAnything-V. 19]. However, extracting structure from learned depth esti- mates can often lead to artifacts. Our proposed VGER differs in that seeks to generate an image sequence from the single input image, and extract the structure that jointly appears.
Recovering 3D structure from a set of images is another well-known problem known as structure- from-motion [12]. Classical methods, such as COLMAP [13], produce relatively sparse represen- tations. Modern advances in deep learning have led to transformer models capable of efficiently constructing highly dense 3D representations in single feed-forward pass, including DUSt3R (6) MASt3R [14 and Visual Geometry Grounded Transformers (VGGT) [5). These models typically act as 3D foundatio models, which are large pre- pre-trained models, intended as plug-and-pla, modules

---

Input Image
Frames of generated input-conditioned video
3D structure
Figure 3: Examples of extracting 3D structure of outdoor bench (top) and indoor office envi- ronment (bottom) from imput images. Input images are shown in subfigure (a). We leverage a video generator, conditional on the images, to generate videos, with frames shown in subfigure (b). These are then subsequently used to construct 3D structures via foundation models, without any frustum- shaped artifacts, shown in subfigure (c).
for downstream tasks. These 3D foundation models have found use in variety of robot perception and calibration problems [16].
Video Generation Models: A recent wave of deep learning-based general video generators spans unconditional, text-to-video, and image-to-video synthesis. Early GAN-based methods (StyleGAN- V (17)) gave way to diffusion and transformer systems: Video Diffusion Models [18) produce high- quality clips, and MagicVideo
[19] extends diffusion priors for text-driven generation. Many advances in this field have focused on the controllability of the camera motions in the video gen- erator, these include GEN3C (20) and ViewCrafter211] and SEVA [22]. Particularly, in this work,
we leverage recent video generator Stable Virtual Camera (Seva) [22 which can condition on an image as well as a camera trajectory to generate a video that follows the trajectory the scene of the image.
Reactive Motion Generation: Our work is the first to apply structures and representations produced by 3D foundation models for reactive motion generation. Relevant frameworks, including Rieman- generate motion creatively generally approach the generation from planning perspective and require the assumption of a pre-defined sur- face, or completely known environment constructed from simple shape primitives. Other robotics focused representations may require depth [25) or velocity measurements (26). There has been some effort extend from representing environments to meshes instead of pre-designed primitives (27). However, there is clear disconnect between the literature on scene representation and motion gen- eration. To bridge this gap, Our method is unique in extracting both representation of the structure and subsequent motion generation model from single RGB image.
nian Motion Policies 7 and Geometric Fabrics 23 24
3 Video generation Environment Representation: Structure Extraction to Motion Generation From a Single Image
Here, we elaborate on the technical de- tails of the VGER method. This includes: leveraging pre-trained video generators to extract structure conditional on the in- put image (section 331];; the construction of an implicit model via multi-scal noise contrastive samples (section 3.2): generat- ing motion from an environment--ependeen metric field (section 3.3). giving collision- free motion. An overview of the workflow is shown in fig. 2
Inpurt Image Careera Trajectory
Video Generation Model
Collision-free Motion
Structure
3D
Distance Modell
Figure 2: Pipeline of VGER.

---

3.1 Structure Extraction from Single Image with Pre-trained Video Generators
We circumvent the direct prediction of any depth information from the imput image, thereby avoiding frustum-shapp shaped artifacts that are characteristic. Instead, we leverage modern video generators to synthesize a sequence of images of the camera moving, conditional on the input image. These images are then passed into a 3D) foundation model, such as DUS(3R [6]. to produce a dense point cloud. We use the pre-trained video generation model, Stable Virtual Camera (Seva) 22 Seva 1.3 billion parameter diffusion model, using stable diffusion 2.1 backbone (28]. It produces sequence of output images {T, .2....: In camera poses {Ti.... i....... Tn J. where n is pre pre-determined image sequence length. Despite the video generator not constructing an explicit 3D representation, Seva is trained on videos that maintain both photometric and geometric consistency across frames. As result, the generated image sequences maintain geometry and temporal consistency.
conditional on the input image To and trajectory of
The generated images, along with the conditioned input image, are then inputted into the 3D foun- dation model, DUS(3R [6. DUS(3R is fully data- driven system built around large vision transformer [29]. It produces dense 3D point-maps along with per-pixel confidence and depth estimates, and then uses these to recover relative camera poses and fully consistent 3D reconstruction. Because it learns correspondences end-to-end rather than relying on hand- cand-crafted features, it can accurately estimate pose even when the two views have very little visual over- lap. After filtering away the points with low confi- dence, recover dataset of dense cloud of N 3DD points (pi, G; ) where E R gives the 3D coordinates and c gives the RGB colors of the point. Next, seek to construct a continuous implicit representation of the environment from the point cloud. This implicit model enables smooth geometry reconstruction and supports downstream tasks such collision avoidance and motion generation.
Imgel
Confmmp
3D Foundaaion Model
(DUS(3R)
Roonestapial TD some with comere poszs
Ing?
Conf
Llepth may
Figure 4: We leverage DUSt3R [6], which can produce 3D structures from sets 2D images, and filter based on confidence maps.
3.2 Multi-scale Noise Contrastive Implicit Model
Reactive motions generated in the environment are influenced by distances to the nearest surface in the environment. Here, VGER converts the point-based output from the 3D foundation model into smoothly varying function that captures the distance to the nearest surface the scene. This implicit function is modelled by neural network surface captured in our point cloud p {pi J
R - R that approximates the unsigned distance to
1 Our training procedure optimizes a loss:
L Lat + Quuur Courf + Qeikk Ceik.
This loss combines: (i) regression to ground-trutte truth distances at query points, (ii) surface consistency term enforcing a zero level-set at the surface, and (iii) an Eikonal regularizer to encourage unit gradient norm. Linear weightings are denoted as asurf and Creik Next, elaborate on this construct of each loss term.
To train a model to recover distances both on and near the surface, we generate negative samples by perturbing points drawn from the surface with Gaussian noise at multiple scales. Concretely, at each iteration we first sample batch of raw surface points (pr N uniformly from P. Around each pt, we add a zero-mean Gaussian noise e whose standard deviation oj is itself drawn from a log-uniform distribution:
Multi-Scale Noise Negative Querying:
Uj Uniform (In(amin), In((max)); oj = exp(u,)) Ej ~ oJ Pi.j = Pt + Ej-
Here, Pa.j denotes query point that is perturbed off the surface point p; with noise Ej. and aiin and maax give boundaries of the standard deviation of the noise. Sampling or, on log scale ensures that the network sees perturbations ranging from extremely fine to coarse (one-tenth of the bounding radius) distances. This multi-scale scheme serves two purposes: Local surface fidelity: Small oj focuses the loss on points extremely close to the true surface, improving pointwise accuracy and helping the zero level set align precisely with P: Robust gradient learning: Larger o, forces the model to estimate distances further from the surface, ensuring that the predicted gradient field V

---

remains informative and prevents collapse to trivial solutions. Next, we compute the true unsigned distance efficiently via a KD-tree, and construct a mean squared eтror loss learn continuous distance representation:
d = min I pll
Cat = MSS((f(P.))
PEP
The implicit model fe is represented by a neural network, that incorporates SIREN activation layers That is, each hidden layer applies a periodic activation:
30
1000 swn (W (1-1) b)))), 1_
Here, f21-1) is the input to layer I, W E 1 and (tD) E R are the learned weights and biases, and Wo > is a fixed frequency scale controlling the periodicity. By fitting parameters in the frequency domain, these activation layers enable fo to capture finer details with higher frequency.
Surface Consistency and Eikonal Regularization: To ensure that our representation has a priori and fox))) 1, we provide additional regularization. For each training
structure of fo(p,) iteration, let B, denote the minibatch of surface points and B, the full minibatch of query points. To align the network's zero level-set with the input surface, and to encourage [[V]( almost everywhere. We add surface loss Laurt and Eikonal loss Leiks given by
1 Σ B
Laurf
[a(P))
Laik
(IV=/660)12211)
(2)
B
PEB,
BEB,
Together, these terms ensure exact surface interpolation and a well-conditioned gradient field, which promotes accurate normal estimation and stable mesh extraction.
Connection Diffusion Models: Multi-scale denoising has proven crucial for sharpening implicit energy-based representations by teaching models to recover fine details at varying noise levels. In particular, frameworks such as denoising score matching (3 and score-based diffusion models 33
denoise data corrupted at multiple noise scales to learn noise-conditional score
so a) (r)
where Pe is the data distribution corrupted by Gaussian noise of scale σ. By sampling a log- uniformly over several orders of magnitude, these methods capture both gigh-freuency surface details (small σ) and global structure (large σ). Analogously, the multi-scale noise in VGER trains the implicit distance field So to denoise points at various distances from the surface, effectively learning landscape whose gradient aligns with the true distance gradient. This mirrors the reverse diffusion process, where successive denoising steps refine sample towards the data manifold. Our incorporation of an Fikonal term Les further ensures that [|Vlell = 1, enforcing the learned field into a distance metric and yielding sharp, artifact-free reconstructions.
3.3 Motion Generation in Constructed Environment via Continuous Metric Fields
Generating motion is central enabling robots interact with their surroundings. Here we outline how we can leverage the environment representations extracted from a single image to shape robot motions. We model robot policies as a dynamical system that is influenced by an environment- dependent metric field.
We take reactive dynamical systems approach, enabling motion at arbitrary initial positions in the environment. Similar to many of the reactive methods introduced in [), model the task-space motion of a robot an autonomous dynamical system, where trajectories can be obtained via numerical integration,
Robot Policies as Dynamical System:
x(t) =x0 xo I ((x(s))ds.
xt() gx((t))
(4)
where x(1) give the task-space coordinates the motion, g R3 R3 is the dynamics, Xo the initial condition for the trajectory, integrated for time l. For robot manipulators, where the Jacobian of the robot forward kinematics is known, we can further pull the dynamical system into the robot's configuration space by multiplying with the pseudoinverse of the Jacobian, as is done in (
5

---

(c) Our reconstructed 3D representation and trajectories Figure 5: With single example imput image, shown in (a), of stone model in indoor environ- ment, VGER can build a 3D representation of the seene. does not suffer from incomplete surfaces like the results from DepthAnything- V2, shown in (b)). It can also facilitate downstream motion trajectory generation. Two trajectories, colored in red and blue, smoothly avoiding the obstacle, are shown in (c).
(a) input
(b) DA-V2
Environment Inftuence as a Metric Field: We propose principled framework for shaping motion generation based on task-driven Riemannian metrics derived from the upstream distance function fe. such that the reactive behaviours like collision-avoidance and surface following emerge. Given an nominal base dynamical system Gbase (x). we define a metric-modulatee motion as
x = G(x) Ibose (x)s,(x),
and G(x) E S*+
(5)
Here, Pbase is the dynamics that governs the initial motion pattern without considering the environ- ment. This could be defined as a simple attractor to goal coordinate or dynamical system learned from data 33 34 G(x) Riemannian metric that positive definite (S++) and varies smoothly throughout the environment, constructed from fe- This metric-based modulation can be interpreted solving, every point x, local quadratic program,
\ G(x)v - v gbaee (x)
x = arg min VERS
(6)
Thus, the metric shaping procedure can be viewed equivalently solving a structured QP at every point in state space, where the metric Gixx defines the local cost geometry. Importantly, by modu- lating G(x), we influence both the preferred directions of motion and the associated effort, enabling smooth reactive collision avoidance.
Our goal is now to construct a Riemannian metric field induce smooth collision avoidance behavior around surfaces represented by learned unsigned distance function fe 3 + R20- where fe(x) approximates the distance from a point æ E RR3 to the closest point on the surface. To penalize motion toward the surface, we construct the collision avoidance metric as
Constructed metric field stretches and warps a base dynamical sys- tem (base(x) with flows (left), to produce the flows G(x) gbese (x) which avoid colliding into the black surface (right).
Figure 6:
Gxx)
fNlww (x) u(x)u(x)),
where is the 3 x 3 identity matrix. Here, define blow-up scaling factor folow and denote unit surface normal u(x), where,
k folow (x) (fe(x) + e)a exp( (op(-)f((x))..
fe( u(x) = V fo())]
(8)
Here, > O controls the penalty strength,
ensures numerical stability near the surface, and >> O induces exponential decay away from the surface. Direct inversion of the metric G at every control step would incur full x 3 matrix solve. Instead, we exploit the fact that
rank-one
update of the identity and apply the Sherman-Morrison formula [35] to eq. (5)
(+ (T+auu = 1 uu -
a
folow (x)
u(x)u(x)7 Ibase (x))
1 + folow (x)
the trajectory approaches surface, that is fo(x) - O, the metric sharply penalizes motion aligned with the surface normal, naturally steering trajectories tangentially to the surface without

---

Imput Image
Depth.nyythnn--22
DUS(3R Single Image
VGGT Single Image
Owrs
Figure 7: Qualitative evaluations of complex 3D environment scenes (from top to bottom: gar- den, bench, indoor, table respectively) each extracted from single image with our VGER relative to the baseline, DeythAnything-- V2 and DUS(3 VGGT models with a single image input. We ob- serve that not only does VGER fill in gaps occluded in the single image, but also altogether avoids the frustum-shaped noise artifacts, which are particularly prominent in the garden and bench images.
requiring hard constraints. The construction smooth and fully differentiable, making well-suited for reactive online motion generation. We illustrate the effect of metric field steering the vector field away from the normal of the surface in fig. 6 on a toy problem. Here, we observe that the illustrated base vector field (shown in the left), after applying the effect of the metric field, smoothly warps around the black surface.
Empirical Evaluations
To evaluate the robustness and quality of our proposed Video-gent rideo-generation Environment Represen- tation (VGER) along with the produced motion trajectories, we collect sets of 10 images in both outdoor and indoor scenes. These include the complex multi-object scenes Garden, Bench, Table, and Indoor, along with the more object-centric Stone and Cabinet scenes. For the evaluation of the quality of the performance of VGER and benchmark models, extract the structure from single image. We use the entire set of ten images, passed to DUSt3 to construct representation that then consider be the ground truth. Implementation details in experiments are provided in section Al
4.1 VGER Avoids Frustum Noise and Extracts Structure Accurately
We compare VGER with single input image against a suite of strong baselines. In our setup, we uniformly sample 10 frames from the generated video to extract out our structure for VGER. These include: DepthAnything-22 [9]: state-of-the-art monocular depth estimation foundation model; VGGT with a single image [15]: a recent 3D foundation model which can handle a single input image. We only pass single image to generate the 3D representation; DUS(3R with single image 66: a widely used 3D foundation model, we again only input single image to generate the 3D representation; GEN3C 20 and ViewCrafter (21]: These contemporary controllable video generators, which we then extract 3D structure from to obtain the representation.
To evaluate the quality of the extracted scene representation, we rescale the environments to the interval [-1, 1] and align the extracted scene representations with the ground truth representation built with the entire set of images using iterative closest point (36). Then, compute the Chamfer

---

Garden
Bench
Table
Office
Stone
Cabinet
DepthAnythingV2
0.222
0.033
0.065
0.065
0.259
0.148
VGGT (single image)
0.109
0.068
0.094
0.062
0.296
0.121
DUSI3 (single image)
0.266
0.049
0.045
0.054
0.283
0.090
GEN3C (20
0.060
0.016
0.060
0.049
0.104
0.066
ViewCrafter 21
0.086
0.028
0.066
0.057
0.178
0.075
VGER (Ours)
0.075
0.029
0.030
0.047
0.096
0.044
Figure 8: (Left) Before computing distances, structure from one image by VGER (green) vs. ground-
truth (red) are aligned. (Right) We report normalized Chamfer distances on different scene cate- gories. Lower is better.
Distance (37] between the point clouds of the ground truth and extracted structure. For two point clouds P1 and Pz, this is defined as,
Dhhamfer (P1,
Σ
min yErs
vll2 +
Σ yEP,
min DEP
y
(10)
Pi
PP
zEP
The qualitative results are presented in fig. Ad- ditionally, we illustrate an example alignment of the structure of the indoor environment extracted from single image with VGER overlaid with structure extracted via 3D foundation model using all of the images collected. The Cham- fer distances are subsequently computed between the aligned structures to obtain our quantitative results. We observe that VGER outperforms all the baseline methods consistently. Qualitatively, we observed that by using video generators, con- ditional on the input image, VGER subjects our structures to greater multi-view consistency. As result, the frustum-shaped noise artifacts that appear in free space no longer appear our ob- tained structure. Additionally, the video generator successfully estimates much of the geometry occluded from single image, resulting in more geometrically accurate representations. This is particularly prominent in the complex Bench and Table scenes shown in rows 2 and 4 in fig, 7
Figure Single-image extraction leaves oc- cluded regions incomplete, VGER reliably re- constructs them. Left: Top: DepthAnything-
V2; bottom: VGER.; Right: Trajectories ap- proaching the cabinet from the occluded back- side, avoiding collision reach the green goal.
Here, also highlight VGER's ability deal with obstructions in fig. 9. where VGER can reconstruct the occluded backside of cabinet. This enables motion trajectories and the correct collision-avoidance behaviour of even when ap- proaching the cabinet from the back, which was not observed from the input image. A more in- tricate level of detail in regions that are hidden from the input image can also seen in fig. [T Here, by leveraging the large amounts of video data used to train the video generator model, VGER is able to accurately capture the geometry of the reverse side of the table surface, well as the structure under the table, hidden from the input image view. We observe that this cannot be captured by the alternative comparison methods, and are left with gaps in obstructed regions.
Figure 10:
Left: Input image; Middle: Com- parisons do not reconstruct unseen areas; Right: VGER reconstructs geometrically-coniisi de- tails in unseen regions, such under the table.
4.2 Trajectories from Motion Policies Built on Constructed Representations
Next, we assess how VGER improves downstream motion generation. Using the approach out- lined in section 3.3 we generate trajectories with implicit distance models derived from VGER, motion policy
DepthAnything- V2 (DA-V2), and our ground-truth geometry. Our yields smooth trajectories, exemplified in the table scene (fig. [T], and we can compute per-step velocities in under 1ms. To quantify alignment with true motion, we integrate collision-free paths

---

Figure 11: VGER enables smooth collision -ree motion. Here, two trajectories (red and blue) warp over benches and around the leg of the table to reach the goal (green).
pics
pics
pies
pics
pics
Figure 12: The extracted structure provided by VGER, sampling different numbers of frames from the generated video.
a designated goal and measure the Discrete Fréchet Distance (DFD) 388, normalized by the Euclidean distance from start to goal. As shown in fig. 13c lower DFD values indicate closer agree- ment with the ground-truth trajectories. Across all scenes, VGER-based structures consistently outperform DA-V2, with particularly large gains in indoor, cabinet, and stone environments. This advantage arises because alternative methods introduce shastu noise artifacts free space, which "funnel" the motion into occluded regions (see fig. 13a). By contrast, VGER completes ob- structed areas without spurious artifacts, guiding trajectories correctly around the backside of the cabinet. Further examples in the complex indoor and garden scenes (fig. 13bb demonstrate that VGER-driven policies smoothly avoid collisions with the table surface and elegantly skirt around the plant and curb, respectively.
4.3 Ablation: Frames of Generated Video Used for Structure Extraction
The key to VGER's performance is its ability to lever- age frames of videos generated from large pre-trained model. When extracting the structure from the video, it can be unnecessarily resource-intensin to use every frame in the video, and we can uniformly subsample frames instead. In our results reported, we subsample 10 images. Here, conduct an ablation study over the number images sampled from the video. We report results on the table scene using 3, 5, 7, 10, 15 images, computing the normalized Cham- fer distance between the extracted structure and the ground truth after alignment. We observe that, beyond 5 images, the performance of VGER is robust to the number of frames extracted from the generated video. Images of the structures extracted for number of images sampled are in fig. 12
Table I: Performance of VGER with var- ious images sampled on the table scene.
# Images
3
5
7 10
15
Dist. (x10 3) 4.2
3.1
3.3
3.0
2.9
Limitations
The VGER pipeline exploits pre-trained video generators as foundation models for reconstruc- tion and motion generation. In practice, these models synthesize geometry well within the cam- era's 's field of view but may "hallucinate" when ex- trapolating beyond the observed frustum. As the spatial distance from the original image capture region increases, so too does the uncertainty. An example of hallucinations in the generated video illustrated in fig. [4 Future avenues of research
Figure 14: Left: The input image; Right: Gen- erated frame contains hallucinated structure out- lined in red that does not match the setting.
to extend VGER can incopporate mechanisms to track extrapolated regions far outside the spatial re-

---

Ouss
DA-V2
Bench
0.087
0.098
Garden
0.232
0.245
Indeor
0.111
0.203
Table
0.211
0.245
Cabinet
0.322
Stone
0.109
0.190
(a) Cabinet, Left: DA-V2; Right: VGER. (b) VGER trajectory examples. (c) Normalized DFD.
Figure 13: (a) Frustum-shaped noise artifacts often introduce local minima in the motion objec- tive. We compare trajectories generated using DepthAnything-Ve (DA-V2) versus VGER: because VGER completes the cabinet's backside without spurious artifacts, successfully navigates to the goal. (b) Sample paths on VGER-derived structures smoothly traverse along the table surface (in- door) and curve around plant (garden), reaching their targets without collision.
(c) The normalized Discrete Fréchet distances between trajectories generated with VGER and DA-V2 structures.
gion captured by the input image, assign uncertainty estimates and integrate the confidence-levels within the downstream motion policies.
6 Conclusion
In this paper, tackle the problem of building environment representation that enables motion generation from a single image. Here, we have present Video-generation Environment Represen- tation (VGER), framework that leverages pre-trained video synthesis and 3D foundation models to recover dense scene geometry from a single RGB image, without frustum-shaped artifacts. By applying multi-scale noise contrastive sampling procedure to the 3D reconstruction, we extract an implicit unsigned distance field that smoothly encodes obstacle proximity across the workspace. Embedding this field into metric-moduuated motion framework yields a motion policy, modelled by non-linear dynamical system capable of producing collision-free trajectories directly from single viewpoint. Extensive evaluations demonstrate that VGER consistently outperforms state-of- the-art baselines, avoiding Trustum-shapee artifacts and smoothly navigating complex indoor and outdoor scenes. This drives the broader impact of future where robots can be actively deployed in open environments, enabling a safer workplace in the future.
References
[1] Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, and Richard Szeliski. Building rome in day. In 2009 IEEE 12th International Conference on Computer Vision, pages 72-79, 2009.
[2] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[3] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[4] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang. Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[5] Raúl Mrr-Artal and Juan D. Tardós. ORB-SLAME an open-source SLAM system for monoc- ular, stereo and RGB-D cameras. IEEE Transactions on Roboties, 33(5):1255-1266 2017.
[6] Shuzhe Wang, Vincent Leroy, Yohamn Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
[7] Nathan Ratliff, Jan Issac, Daniel Kappler, Stan Birchfield, and Dieter Fox. Riemannian motion policies, 2018
10

---

[8] Weiming Zhi, Tin Lai, Lionel Ott, and Fabio Ramos. Diffeomorphic transforms for generalised imitation learning. In Learning for Dynamies and Control Conference. L4DC, 2022.
[9] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Heng- shuang Zhao. Depth anything v2. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
[10] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[11] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[12] Shimon Ullman. The interpretation of structure from motion, Proceedings of the Royal Society of London. Series B. Biological Sciences, 1979.
[13] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Con- ference on Computer Vision and Pattern Recognition (CVPR), 2016
[14] Vincent Leroy, Yohann Cabon, and Jerome Revaud. Grounding image matching in 3d with mast3r, 2024.
[15] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2025.
[16] Weiming Zhi, Haozhan Tang. Tianyi Zhang, and Matthew Maanson-Robbrson Unifying repre- sentation and calibration with 3d foundation models. IEEE Robotics and Automation Letters, 2024.
[17] Nya Skorokhodov, Stefano Sabatini, and Guang Wang. Stylegan-v: continuous video gener- ator. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[18] Jonathan Ho. Jascha Sohl-Dickstein, and Pieter Abbeel. Video diffusion models, arXiv preprint arXiv:2209. 14792, 2022.
[19] Hu Zhou, Liang Lin, and Shih-Fu Chang. Magicvideo: Aligning latent diffusion with trans- formers for text-guided video generation. arXiv preprint 11Xv:2222..1236,, 2022.
[20] Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3e: 3d-informed world- consistent video generation with precise camera control. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[21] Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fideliyy novel view synthesis. IEEE transactions on pattern analysis and machine intelligence, 2025.
[22] Jinghao Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthesis with diffusion models. arXiv preprint arXiv:2503. 14489, 2025.
[23] Karl Van Wyk, Mandy Xic, Anqi Li, Muhammad Asif Rana, Buck Babich, Bryan Peele, Qian Wan, Iretiayo Akinola, Balakumar Sundaralingam, Dieter Fox, Byron Boots, and Nathan D. Ratliff. Geometric fabrics: Generalizing classical mechanics to capture the physics of behavior. IEEE Robotics and Automation Letters, 2022.
[24] Weiming Zhi, Iretiayo Akinola, Karl van Wyk, Nathan Ratliff, and Fabio Ramos. Global and reactive motion generation with geometric fabric command sequences. IEEE International Conference on Robotics and Automation, ICRA, 2023.
11

---

[25] Weiming Zhi, Lionel Ott, Ransalu Senanayake, and Fabio Ramos. Continuous occupancy map fusion with fast bayesian hilbert maps. In International Conference on Robotics and Automation (ICRA), 2019.
[26] Weiming Zhi, Ransalu Senanayake, Lionel Ott, and Fabio Ramos. Spatiotemporal learning of directional uncertainty urban environments with kernel recurrent mixture density networks. IEEE Robotics and Automation Letters, 2019.
[27] Michael Pantic, Lionel Ott, Cesar Cadena, Roland Siegwart, and Juan Nieto. Mesh manifold based riemannian motion planning for omnidirectional micro aerial vehicles. IEEE Robolics and Automation Letlers, 2021.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[29] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[30] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
[31] Pascal Vincent. connection between score matching and denoising autoencoders. Neural Computation, 2011.
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[33] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.
[34] Weiming Zhi, Tin Lai, Lionel Ott, Edwin V. Bonilla, and Fabio Ramos. Learning efficient and robust ordinary differential equations via invertible neural networks. In International Confer- ence on Machine Learning, ICML, 2022.
[35] Jack Sherman and Winifred J. Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. Annals of Mathematical Statistics, 21:124-127, 1950.
[36] K. Arun, T. S. Huang, and S. Blostein. Least-squares fitting of two 3-d point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1987.
[37] H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and H. C. Wolf. Parametric correspondence and chamfer matching: two new techniques for image matching. In Proceedings of the 5th International Joint Conference Artificial Intelligence, 1977.
[38] Thomas Eiter and Heikki Mannila. Computing discrete fréchet distance. Technical Report CD-TR 94/64, 1994.
12

---

Appendix
A1
Implementation Details
We run our experiments on a standard desktop with an Intel 19 CPU and an NVIDIA RTX 4090 GPU with 24GB VRAM. We use all the standard hyper-parameters in the Seva and DUSt3R foundation models used in our pipeline. Here list all the parammersss used for the reconstruction and motion policies construction of VGER method in table A1
Implicit Model Multi Scale Sampling
Network for Implicit Model
Motion Policy Blow-up
Qaurf
Layers Width
0.1
256
100
0.0025
25
10
0.1
Training Epochs per
2000
[P]
10000
Learning rates (I.s))
3e-4, 1e-4, Se-5, 1e-5
B.
5000
Table Al: VGER hyperparameters at glance.
A2 Additional Visualizations of Reconstructions
Here we provide additional visualizations of the structures extracted from VGER along with base- lines. These are provided in fig. A1 Additionally, here we provide visualizations for our ablation
Input Imss
VGER YOU RS) on
Dustr
Dopth Anythin-V2
VOGI
Groun Truit
Figure A1: The extracted structures from method, baselines, and ground truth, along with the single input image, for the table, bench, garden, indoor, cabinet, and stone environments.
experiment, and observe the structure extracted via VGER when the number of frames that we use from the generated video differ.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions of our proposed task and method are introduced in both abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have a limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No theoretical results is presented in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we detail the implementation details both the experimental section, and more in detail in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Though code has not been released during the review, the authors commit to releasing code with camera-ready version of this manuscript.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
http:/gudies/e/ee wew.edddeeeeoeuubbtt.. Comeeuumieeiiececcceeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All hyper-parameters used reproduce the results are provided in the appen- dices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
The metrics used are clearly defined in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g,, Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The resources used in our experiments are clearly stated in the implementa- tion details in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////..aaeeepcpcpppppt//ttaa
Answer: [Yes]
Justification:
The authors reviewed the code and have concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification: We highlight the application of VGER to robotics, and emphasize how this will lead a more automated future with safer workplaces.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The authors see no potential risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
They are properly cited and licenses are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use the pack- age should be provided. For popular datasets, wwwpppppppttteeeccccppppp thhoode com/ hom/aatsetts has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
If assets are released,
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
No new asset is released at the review stage.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: No crowdsourcing experiments and research with human subjects in- volved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to LLM policy lbttps:/neeurips. / for what should or should not be described.
20


Reviews:
[{"review_id": "KPG7SDyyAx", "reviewer": "Reviewer_V16n", "summary": "This paper leverages the additional information provided by a video generation model to perform 3D reconstruction from a single image using Dust3R, and subsequently builds an implicit representation to enable a collision-free motion generation algorithm. Specifically, after generating a video to obtain multi-view images, it merges the point clouds estimated by Dust3R (filtered by confidence) across views to reconstruct the 3D scene. It then trains an implicit distance field with multi-scale noise, and finally generates a motion plan based on this representation.", "questions": "1. **Implicit scene description**\n   Are there prior works using implicit scene representations for collision avoidance? How does this method compare in terms of accuracy or efficiency?\n2. **Failure cases / initial image impact**\n   Can you show more failure cases or analyze how the initial image content affects reconstruction and motion planning?\n3. **Reliability**\n   Are there ways to improve reliability — for example, by incorporating active perception, sensor fusion, or uncertainty estimation?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The method effectively leverages a video generation model to fill in the missing information from single-image Dust3R point clouds, achieving impressive 3D reconstruction results.\n2. This is the first work to address the task of generating robot collision-free trajectories from single images by integrating perception and motion planning processes.\n3. The paper introduces a method to train an implicit distance model and continuous metric field that can efficiently generate motions based on the environment’s representation.\n\n**Weaknesses:**\n\n1. There are already many existing frameworks that use video generation (or multi-view diffusion) for 3D reconstruction [1, 2, 3], so the novelty of this contribution is questionable. At minimum, the paper should include a thorough literature review and provide a clear justification for why this approach offers advantages over prior work.\n2. As noted in the limitations, video generation models fail to produce accurate representations of parts of the scene that are not visible in the input image. The success of the method thus heavily depends on the input image capturing most of the scene.\n3. The reliability of the approach is uncertain. For robot motion planning, the required reliability is higher than for typical vision tasks. For example, as shown in Figure 5, even a human observer cannot confidently infer what lies behind the stone—there are many possible variations. Relying on the “imagination” of a video generation model may not provide sufficient safety. Active perception strategies might be necessary.\n4. A final concern is the computational cost and speed of the method. Running foundation models can be time-consuming and resource-intensive. In many cases, using a simpler solution, such as a depth sensor, might be more practical.\n\n[1] Gao, Ruiqi, et al. \"Cat3d: Create anything in 3d with multi-view diffusion models.\" arXiv preprint arXiv:2405.10314 (2024).\n\n[2] Yu, Wangbo, et al. \"Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.\" arXiv preprint arXiv:2409.02048 (2024).\n\n[3] Ren, Xuanchi, et al. \"Gen3c: 3d-informed world-consistent video generation with precise camera control.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "was_revised": true, "final_justification": "Though the use case of robot path search seems a little weird to me given the time cost and VRAM cost, I still think the 3D reconstruction part of the method have enough contribution to be accepted, as it gets better reconstruction result with less time-cost comparing to prior works.", "rebuttal": "We thank the reviewer for their constructive comments and efforts in helping us improve the manuscript. Thank you for highlighting the methodological contributions and empirical results in our work.\n\nWe particularly appreciate the reviewer taking the time to outline the methods, Cat3D, Viewcrafter, and Gen3C. These are contemporary methods which share similarities with VGER. In particular, Viewcrafter similarly leverages the 3D foundation model DUSt3R, Gen3C integrates a video generator, while VGER makes use of both.\n\n| Method          | Tables and Benches | Indoor Office | Stone | Garden | Bench | cabinet |\n| --------------- | ------------------ | ------------- | ----- | ------ | ----- | ------- |\n| **VGER**        | **0.030**              | **0.047**         | **0.096** | 0.075  | 0.029 | **0.044**   |\n| **ViewCrafter** | 0.066              | 0.057         | 0.178 | 0.086  | 0.028 | 0.075   |\n| **Gen3C**       | 0.060              | 0.049         | 0.104 | **0.060**  | **0.016** | 0.066   |\n\nThe comparison results are tabulated above, we observe that all three of the methods have solid performance, as measured by chamfer distance. VGER generally outperforms the alternative methods, with Gen3C also having strong performance in the garden and bench scenes. \n\n| GPU             | VRAM  | Time      |\n| --------------- | ------- | --------- |\n| **GEN3C**       | 42.9 GB | \\~30 mins |\n| **ViewCrafter** | 22.6 GB | \\~4 mins  |\n| **VGER**        | 19.7 GB | \\~7 mins  |\n\nThe above table provides information of VRAM usage and runtime, we observe that VGER and ViewCrafter have relatively lower VRAM usage and runtime.\n\nImplicit representations, such as learned distance functions are typically used in trajectory optimization methods for motion generation. Our approach differs in that it is reactive, it does not require an entire trajectory to be found, and instead focuses on generating the immediate action. This enables reactive methods to be a lot faster and produces an entire policy (velocity field) rather than a single trajectory. This is conceptually the most similar to Riemannian Motion Policies (RMPs) (Ratliff, 2018), but unlike RMPs which require an environment representation constructed from primitive shapes, we allow for the learned implicit model to be used.\n\nWe also appreciate the point raised about reliability, potential hallucinations in unseen regions,  and agree with the opportunity to leverage active perception. We note that we can actually track which regions of the reconstruction are connected to the original input image, and which regions are connected to frames generated via video generation. Additionally, the produced reconstruction from VGER includes pointwise confidence maps, which are inherited from DUSt3R. These confidence values can indeed be used to guide robots to actively perceive and reduce the uncertainties in areas not covered by previous images. However, we believe that producing an estimate of the environment structure even in unseen regions remains a valuable prior. Accomplishing object-level shape completion, for occluded objects, is valuable in robotics, likewise VGER accomplishes the completion of scene level occlusions not captured in a single image in one go. Overall, we definitely agree that active perception would be a practical solution to potential hallucinations described in the limitations section, and we hope to incorporate active perception as a follow-up to this work."}, {"review_id": "mublArBNJQ", "reviewer": "Reviewer_xqDw", "summary": "This paper focuses on the problem of collision-free motion generation. It leverages large-scale video generation models to generate multiple views for the static scene, which is used to construct the 3D dense point cloud through 3D foundation models. Then, it generates collision motion with motion models.", "questions": "* I'm not familiar with the related works about motion generation. Is it the first work to generate motion based on a 3D dense point cloud?\n* What's the efficiency of the multi-scale noise motion generation?\n* What's the hyperparameter of the motion generation models? What's the number of the point? If it's a sparse point cloud, how does the motion generation model perform?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "* Strength\n* The method demonstrates strong performance in scene construction. The approach achieves impressive results in building dense 3D point clouds by combining video generation models with 3D reconstruction models.\n* The paper is well-written and good presentation, making the methodology and results accessible and understandable.\n* Weakness\n* The proposed 3D construction approach is closely similar to existing work, particularly ViewCrafter [1], which similarly integrates video diffusion models with 3D reconstruction for novel view synthesis. This similarity raises concerns about the novelty of the contribution.\n[1] ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis.\n* The baseline comparisons are relatively simplistic. The paper mainly contrasts against 3D reconstruction methods such as Dust3R and VGGT applied to single images, which may not reflect the improvement of the methods.\n* And there is no comparison of motion generation methods with other works. \n* The actual impact of motion generation on downstream robotics tasks is not thoroughly demonstrated. The paper lacks a clear explanation or empirical evidence on how the generated motions contribute to improving performance or capabilities in practical robotic applications.", "was_revised": true, "final_justification": "1. The method demonstrates strong performance in scene construction.\n2. The motion policies seem robust and novel, as claimed by the authors.\n3. The baseline comparisons are relatively simplistic.", "rebuttal": "We thank the reviewer for the thoughtful comments and for highlighting both the performance and presentation of the manuscript. \n\nWe thank the reviewer for pointing out additional baselines for experimental comparisons. Here we report normalized Chamfer distances of VGER as compared to the contemporary methods, ViewCrafter and GEN3C:\n\n| Method          | Tables and Benches | Indoor Office | Stone | Garden | Bench | cabinet |\n| --------------- | ------------------ | ------------- | ----- | ------ | ----- | ------- |\n| **VGER**        | **0.030**              | **0.047**         | **0.096** | 0.075  | 0.029 | **0.044**   |\n| **ViewCrafter** | 0.066              | 0.057         | 0.178 | 0.086  | 0.028 | 0.075   |\n| **Gen3C**       | 0.060              | 0.049         | 0.104 | **0.060**  | **0.016** | 0.066   |\n\nThe comparison results are tabulated above. We observe that all three of the methods have solid performance, as measured by chamfer distance. VGER generally outperforms the alternative methods, with GEN3C also having strong performance in the garden and bench scenes. Additionally, VGER and ViewCrafter have comparable VRAM and runtime costs while GEN3C requires more VRAM and time.\n\n| GPU             | VRAM  | Time      |\n| --------------- | ------- | --------- |\n| **GEN3C**       | 42.9 GB | \\~30 mins |\n| **ViewCrafter** | 22.6 GB | \\~4 mins  |\n| **VGER**        | 19.7 GB | \\~7 mins  |\n\nThe introduced method is the first reactive approach, which produces a policy (velocity field) in the 3D environment that leverages an implicit representation constructed from a dense point cloud. Alternative approaches such as motion planning via sampling-based motion planners and trajectory optimization approaches generally produce a single path or trajectory rather than a vector field which can be integrated. In that sense, the proposed approach is reactive, and is conceptually similar to Riemannian Motion Policies (RMPs) (Ratliff 2018) and Geometric Fabrics (Van Wyk, 2024). These approaches typically assume that the environment is constructed of a set of primitive shapes. The training of the implicit method with the multi-scale noise sampling is fairly efficient and can be achieved in minutes on consumer-grade GPUs. \n\nThe hyperparameters for the motion policies constructed are in Table A1 in the appendix under “Motion Policy Blow-up”, we will make sure to make this more prominent in the next version of this manuscript. Additionally, the number of points in the representation before implicit representation is constructed differs from scene to scene, depending on the confidence outputted by the 3D foundation model, but is generally in the millions of points. We also thank the reviewer for emphasizing the connection with robotics tasks. In this paper, we focused on relatively larger both indoor and outdoor scenes. An interesting scene that further highlights the benefits of our approach is table-top scenes to do motion generation for manipulators. This is an area where predicting complete geometries can prove to be very important, we will add this scene into the next version of our submission."}, {"review_id": "4XVMFXsEWW", "reviewer": "Reviewer_pvyM", "summary": "This paper proposes VGER, a novel pipeline that generates environment-aware motion policies from a single RGB image. The key idea is to leverage large video generation models (e.g., Seva) to synthesize a multiview video from the input image, and then reconstruct a dense 3D scene using a 3D foundation model (e.g., DUSt3R). An implicit distance field is trained using multi-scale noise contrastive supervision, and motion policies are then derived using a Riemannian metric field constructed from this geometry. The method is evaluated on diverse indoor/outdoor scenes, demonstrating improved reconstruction quality and smoother, collision-free motion trajectories compared to monocular depth baselines.", "questions": "How robust is VGER to real-world image conditions (e.g., noise, lens distortion)? Could you provide an ablation or example on natural images? Have you considered learning the Riemannian metric instead of hardcoding it? This could help generalize better in cluttered or ambiguous scenes. Could confidence maps from the 3D foundation model be propagated into the motion planning stage to avoid unreliable areas? Do you plan to test this on a real robot or photorealistic simulator?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**: (1) The integration of pretrained video generators with 3D foundation models to create multiview depth from a single image is an innovative and elegant approach. (2) The modular design (video synthesis → point cloud → implicit field → motion policy) is clean and extensible. (2) The qualitative results is compelling. Visuals across several scenes show smoother, safer motion paths and less noisy geometry compared to strong baselines like DepthAnything-V2. (4) The construction of the implicit field via multi-scale noise denoising is well-grounded and connected to score-based modeling.\n\n**Weaknesses**: The authors acknowledge hallucinations from video generation but provide no mechanism to handle or flag unreliable geometry during motion planning. Also, no real-world to sim-to-real validation is conducted, all experiments are in synthetic environments. Demonstration on it would work on actual robots or noisy RGB images can be helpful. The Riemannian field used to guide motion is hand-designed. There’s no ablation or learning-based alternative explored. The benchmark scenes are mostly static and small-scale. The method is not evaluated in dynamic or large outdoor settings.", "was_revised": true, "final_justification": "The authors' responses address key concerns effectively—real-world images confirm applicability beyond synthetics, and commitments to uncertainty handling, metric learning, and robot testing enhance potential impact. While ablations on noisy conditions and larger/dynamic scenes are still missing, the core innovation and results hold up well. Reasons to accept (elegant integration, strong visuals, robotics relevance) outweigh remaining limitations (e.g., hallucination handling, evaluation scope). I therefore keep my score for borderline accept", "rebuttal": "We thank the reviewer for their constructive comments and for highlighting the performance and innovation of our approach!\n \nWe wish to highlight that the pictures of both the indoor and outdoor scenes were images that we took in the real-world, so the performance seen is indicative of performance on real images. We agree with the suggestions here that the uncertainty from the 3D foundation model can be propagated to the VGER model to guide the motion generation, and potentially achieve active perception by enabling robots to actively perceive these regions. This shall be explored in subsequent work. Additionally, the Riemannian Metric here is constructed such that collision avoidance is achieved. However, you’re completely right that this metric can also be learned to, for example, imitate a set of provided demonstrations. Finally, we definitely plan to deploy the method onto a real robot and have identified table-top scenes to be particularly suitable, as VGER would be able to estimate regions of the scene which are occluded, which would be particularly valuable."}, {"review_id": "7ZxoPcF0sb", "reviewer": "Reviewer_NSXd", "summary": "This paper presents a pipeline for generating robust collision-free motion policies from single images. Their framework consists of three parts: lifting an image into a complete 3D scene, training an implicit unsigned distance field (UDF) representation of the scene, and incorporating the implicit metric field into the motion planning for collision avoidance. To lift an image into a 3D scene, they observe that the simplest implementation of unprojected depth map suffers from two artifacts — noisy depth points along surface boundaries and incomplete surfaces — and use a camera-conditioned video generation method to generate unseen views and a feedforward SfM method (dust3r) to lift it into a 3D point cloud. They then train an implicit field on the resulting 3D point cloud and incorporate the UDF into the motion planning for collision-avoidance.", "questions": "- Is there a simpler setup you can use? I would like to see more ablations in-between unprojected depth estimate and your full model (mentioned more in-depth in weaknesses section)\n- Why not just use resulting DUST3R point cloud in motion planning? (mentioned above)\n- How long does training the per-scene MLP take? Those details should be mentioned", "limitations": "- Method requires per-scene training of an MLP before motion plan can be generated, which is impractical for most robotics settings\n- Hallucinated content is used with no uncertainty measuring (one could e.g. render multiple videos and check variance or generated videos as proxy for uncertainty)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Good direction of using 3D foundation models in robotics motion planning with minimal/no training overhead\n- Paper writing is clear and easy to follow\n\nWeaknesses:\n\n- Engineering focused — while I agree the framework is useful, I do think it is a bit plug-and-play and it’s worth noting there perhaps isn’t much technical novelty\n- Clunky and expensive — the framework has a few distinct moving pieces without interaction between them: the video generation model, feedforward SfM model, per-scene UDF model. Perhaps most clunky is that the UDF needs to be trained per scene. Either inferring this feedforward or avoiding with some heuristic geometry could simplify the pipeline and make it more practical\n- Experiments and ablations and pipeline could be improved — currently the authors compare their motion plans vs those from the unprojected depth anything results. I personally would be interested in the following ablations:\n    - what about heuristic geometries like meshes or using the raw point cloud from DUST3R\n    - heuristically cleaning the pointcloud with a depth-gradient thresholded mask like e.g. MoGe does in their visualizations\n    - using the pointcloud from an image-to-pointmap representation like single-frame VGG/DUST3R\n    - using some unseen-geometry heuristic like don’t cross any part of space unobserved (implemented as some geometry proxy if feasible)", "was_revised": true, "final_justification": "I think this paper is definitely still in the borderline accept region and not a 'definite accept', but I also don’t have much expertise in motion planning (more so the 3d vision side), so I am not 100% sure. \nThe idea is interesting but the implementation is a bit impractical in my estimation (waiting several minutes for optimization rather before the robot can make a motion plan / act). The evaluations are also a bit weak in terms of simpler evaluations and baselines and no real-world demonstration. I think if it is accepted, they have to definitely address the speed of optimization in the paper as limitation and future work (or at the very least include the optimization times in the paper).", "rebuttal": "We thank the reviewer for their constructive and extremely helpful feedback. We appreciate the positive feedback of our methodology and the clarity and presentation of the manuscript.\n\nHere, we’d like to answer some of the questions presented. Our proposed VGER goes directly from a single RGB image to a scene, and also attempts to provide estimates in regions that are occluded. Generally, the reconstructed surfaces provided by 3D foundation models are of higher quality (less noise) than off-the-shelf RealSense cameras. With a good stationary depth image, I would expect it not to have the frustum-shaped errors that are common when trying to extract structure from monocular depth estimation models (e.g. DepthAnything). However, VGER still provides additional value by enabling the generation of regions that are occluded. \n\nWe agree that the resulting point cloud from the 3D foundation models could perhaps be used for collision checking in motion planners such as RRTs. The construction of the implicit model enables a smoothly varying distance to the surface to be learned, enabling the construction of the policy, given as a vector field in 3D space. This will enable trajectories to be efficiently integrated at different initial positions, whereas sampling-based motion planners, which do collision-checking based on the point cloud, generally return a single trajectory/path, rather than a policy. We acknowledge that training the implicit model from the point cloud representation needs to be done offline. However, this is not overbearingly slow, and the per-scene training time is approximately several minutes on consumer-grade GPUs."}]

Author rebuttal:

Meta-review: After the rebuttal period, this paper received only positive reviews (3 borderline accept, 1 accept).

On the positive side, the reviewers consider the task (the combination of 3D reconstruction as well as collision-free motion policy) to be  interesting and the visual results compelling. On the more critical side, several of the reviewers questioned the technical novelty of the paper, noting that the 3D reconstruction community has explored the use of video generation quite extensively. While I tend to agree with this criticism, I overall agree with the reviewers that this line of work, particularly the motion policy side, seems promising and interesting.

I advocate for acceptance but would strongly encourage the authors to add a related work section discussing 3D reconstruction using video generators (e.g. the papers note by Reviewer V16n) as well as some rewriting.
Decision: Accept (poster)