3D Scene Prompting for Scene-Consistent Camera-Controllable Video Generation

## Abstract

Abstract We present 3DScenePrompt , a framework that generates the next video chunk from arbitrary-length input while enabling precise camera control and preserving scene consistency. Unlike methods conditioned on a single image or a short clip, we employ dual spatio-temporal conditioning that reformulates context-view referencing across the input video. Our approach conditions on both temporally adjacent frames for motion continuity and spatially adjacent content for scene consistency. However, when generating beyond temporal boundaries, directly using spatially adjacent frames would incorrectly preserve dynamic elements from the past. We address this by introducing a 3D scene memory that represents exclusively the static geometry extracted from the entire input video. To construct this memory, we leverage dynamic SLAM with our newly introduced dynamic masking strategy that explicitly separates static scene geometry from moving elements. The static scene representation can then be projected to any target viewpoint, providing geometrically-consistent warped views that serve as strong 3D spatial prompts while allowing dynamic regions to evolve naturally from temporal context. This enables our model to maintain long-range spatial coherence and precise camera control without sacrificing computational efficiency or motion realism. Extensive experiments demonstrate that our framework significantly outperforms existing methods in scene consistency, camera controllability, and generation quality.

## 1 Introduction

Camera-controllable video generation (He et al., 2024; Wang et al., 2024b; Jin et al., 2025) aims to synthesize videos following user-specified camera trajectories while maintaining visual coherence and temporal consistency. Recent advances have progressed from generating entirely new videos with controllable viewpoints (Bahmani et al., 2025) to enabling users to extend a single image or short video clips along desired camera paths (He et al., 2024; Agarwal et al., 2025). Yet these methods share a fundamental limitation: they can only process extremely short conditioning sequences, typically just a few frames, which constrains their ability to understand longer videos and hence fails to preserve the rich scene context present in those longer videos. What if we could provide a model with arbitrary-length video sequences and generate continuations that not only follow precise camera controls but also maintain scene consistency with the entire input? Such task, which we refer to as scene-consistent camera-controllable video generation, has immediate applications in film production (Zhang et al., 2025), virtual reality (He et al., 2025), and synthetic data generation (Knapp & Bohacek, 2025).

Scene-consistent, camera-controllable video generation presents three coupled challenges: first, static elements must remain stable across time, while dynamic elements e.g., moving objects or people, should evolve naturally from their most recent states rather than replaying motions from the distant past.Second, effective camera control requires understanding the scene’s 3D geometry: generated content must obey physical constraints, handle occlusions correctly, seamlessly compose dynamic elements with static structure, and plausibly extrapolate previously unobserved regions. Third, these capabilities must remain computationally practical: naïvely conditioning on all input frames scales poorly and becomes intractable as sequence length grows.

Rather than building a bespoke architecture, we leverage strong pretrained video generators, retaining their learned priors and training efficiency, by redesigning how prior content is referenced. Our key insight is to fundamentally rethink this referencing via a dual spatio-temporal conditioning mechanism. Current image-to-video (Yang et al., 2024) and video-to-future-video models (^1^11Throughout our paper, video-to-future-video models refer to models that are capable of generating the subsequent frames of the given input video (e.g., cosmos-predict2 (Agarwal et al., 2025).) (Agarwal et al., 2025) achieve realistic generation by conditioning on temporally adjacent frames to maintain short-term consistency and motion continuity. However, adjacency in video is not purely temporal—it can also be spatial. When generating scene-consistent videos, the frames we synthesize may be spatially adjacent to frames from much earlier in the input sequence, particularly when the camera revisits similar viewpoints or explores nearby regions. This dual nature of adjacency suggests a new conditioning paradigm that leverages both temporal and spatial relationships.

Based on these motivations, we propose 3DScenePrompt, a novel video generation framework designed for scene-consistent camera-controllable video synthesis. It takes an arbitrary-length video as context and generates the future video that is consistent with the scene geometry of the context video. The key innovation lies in our dual spatio-temporal conditioning strategy: the model conditions on both temporally adjacent frames (for motion continuity) and spatially adjacent frames (for scene consistency). However, an important consideration for spatial conditioning for our task is that it must provide only the persistent static scene structure while excluding dynamic content, as directly conditioning on spatially adjacent frames from the past would incorrectly preserve dynamic elements. To enable this without temporal contradictions, we construct a 3D scene memory that exclusively represents the static geometry extracted from the entire input video.

To construct this 3D scene memory from dynamic videos, we leverage recent advances in dynamic SLAM frameworks (Zhang et al., 2022; 2024; Li et al., 2024) to estimate camera poses and 3D structure from the input video. To extract only the static regions from the estimated 3D structure, we introduce a dynamic masking strategy that explicitly separates static elements and moving objects. The static-only 3D representation can then be projected to target viewpoints, yielding geometrically-consistent warped views that serve as spatial prompts while allowing dynamic elements to evolve naturally from temporal context alone. Surprisingly, the integration of 3D scene memory provides an additional benefit: the geometrically-consistent warped views provide rich visual references that significantly reduce uncertainty in viewpoint manipulation, enabling precise camera control without any other explicit camera conditioning.

In summary, 3DScenePrompt enables both accurate camera control and long-range spatial consistency by treating the static scene representation as a persistent spatial prompt that guides generation across arbitrary timescales. Extensive experiments demonstrate that our framework significantly outperforms existing methods in maintaining scene consistency, achieving precise camera control, and generating high-quality videos from arbitrary-length inputs.

Figure: Figure 2: Comparison to existing architectures. (a) Camera-controllable methods condition on a single frame and camera trajectory (Wang et al., 2024b; He et al., 2024; Jin et al., 2025). (b) Video-to-future-video methods use the last few frames of the input video when generating the future video for temporal continuity (Song et al., 2025), but fail to maintain long-term spatial consistency when revisiting viewpoints unseen in the given few frames. (c) Our approach combines temporal conditioning (last few frames) with spatial conditioning (spatially adjacent frames) to achieve scene-consistent generation with precise camera control.
Refer to caption: x2.png

## 2 Related Work

#### Camera-controllable video generation.

Building upon the recent success of video diffusion models (Blattmann et al., 2023; Guo et al., 2023; Yang et al., 2024; Runway, 2024; Brooks et al., 2024), recent works (He et al., 2024; Wang et al., 2024b; Bahmani et al., 2024) have achieved camera-controllable video generation by introducing additional adapters into U-Net-based video diffusion models that accept camera trajectories. For instance, CameraCtrl and VD3D (Bahmani et al., 2024; He et al., 2024) incorporate spatiotemporal camera embeddings, such as Plücker coordinates, via ControlNet-like mechanisms (Zhang et al., 2023). While these methods enable precise trajectory following, they only condition on single starting images, lacking mechanisms to maintain consistency with extended video context. In contrast, our approach enables leveraging entire video sequences as spatial prompts through 3D memory construction, enabling scene-consistent generation that preserves the rich scene context within arbitrary-length inputs.

#### Geometry-grounded video generation.

Recent works (Ren et al., 2025; Yu et al., 2025; Seo et al., 2025) have integrated off-the-shelf geometry estimators into video generation pipelines to improve geometric accuracy. Gen3C (Ren et al., 2025), for instance, similarly adopts dynamic SLAM (Li et al., 2024; Han et al., 2025) to lift videos to 3D representations. However, these methods exclusively address dynamic novel view synthesis—generating new viewpoints within the same temporal window as the input. This constrained setting allows them to simply warp entire scenes without distinguishing static and dynamic elements. Our work fundamentally differs by generating content beyond temporal boundaries, requiring selective masking of dynamic regions during 3D construction—a critical challenge that emerges only when static geometry must persist while dynamics evolve naturally into the future.

#### Long-horizon scene-consistent generation.

Various approaches attempt scene-consistent long video generation through different strategies. ReCamMaster (Bai et al., 2025) and TrajectoryCrafter (Yu et al., 2025) interpolate frames or construct 3D representations but remain confined to the input’s spatiotemporal coverage, essentially performing dynamic novel view synthesis. StarGen (Zhai et al., 2025) scales to long trajectories but assumes static worlds, eliminating temporal dynamics entirely. DFoT (Song et al., 2025) most closely relates to our work, proposing guidance methods that condition on previous frames for scene consistency. However, DFoT also faces fundamental memory constraints when processing extended sequences, limiting its ability to maintain long-range spatial coherence. Our dual spatio-temporal strategy with SLAM-based spatial memory overcomes these limitations by selectively retrieving only the most relevant frames, both temporally and spatially, enabling computationally efficient processing of arbitrary-length videos while maintaining both motion continuity and scene consistency.

## 3 Methodology

### 3.1 Problem Formulation and Motivation

We address the task of scene-consistent camera-controllable video generation: given a dynamic video $\mathbf{V}_{\text{in}}\in\mathbb{R}^{L\times H\times W\times 3}$ of arbitrary length $L$ as context with height $H$ and width $W$, our goal is to generate $T$ subsequent frames $\mathbf{V}_{\text{out}}\in\mathbb{R}^{T\times H\times W\times 3}$ that follow a desired camera trajectory $\mathbf{C}=\{C_{t}\}_{t=1}^{T}$ while maintaining consistency with the scene captured in the context input:

$$ $\mathbf{V}_{\text{out}}=\mathcal{F}(\mathbf{V}_{\text{in}},\mathcal{T},\mathbf{C}),$ (1) $$

where $C_{t}\in\mathbb{SE}(3)$ represents camera extrinsic matrices and $\mathcal{T}$ is a text prompt when a video generator $\mathcal{F}(\cdot)$ is based on pretrained text-to-video priors (Hong et al., 2023; Yang et al., 2024; Bahmani et al., 2025).

#### Comparison to existing solutions.

This task fundamentally differs from existing video generation paradigms. Existing camera-controllable generation methods (He et al., 2024; Wang et al., 2024b; Bahmani et al., 2024) synthesize videos following user-specified trajectories but only condition on a single image $\mathbf{I}_{\text{ref}}$ or plain text $\mathcal{T}$ (Fig. [2](https://arxiv.org/html/2510.14945v1#S1.F2)-(a)):

$$ $\mathbf{V}_{\text{out}}=\mathcal{F}(\mathbf{I}_{\text{ref}},\mathcal{T},\mathbf{C}),\;\;\text{or}\;\;\mathbf{V}_{\text{out}}=\mathcal{F}(\mathcal{T},\mathbf{C}),$ (2) $$

which is insufficient for our task, where the entire underlying 3D scene of the context video should be considered. In contrast, video-to-future-video generation methods such as Cosmos-predict-2 (Agarwal et al., 2025) $\mathcal{G}(\cdot)$ employ temporal sliding windows to generate future frames (Fig. [2](https://arxiv.org/html/2510.14945v1#S1.F2)-(b)):

$$ $\mathbf{V}_{\text{out}}=\mathcal{G}(\mathbf{V}_{\text{in}}[L-w:L],\mathcal{T})$ (3) $$

where $\mathbf{V}_{\text{in}}[L-w:L]$ for $w\ll L$ represents a small overlap window, typically consisting of the last few frames of $\mathbf{V}_{\text{in}}$. Although this design encourages temporal smoothness by providing the last few frames when generating the future video, it often fails to preserve long-term spatial consistency when the camera revisits regions not covered by the small window $w$.

Figure: Figure 3: Overview of 3DScenePrompt framework. To generate the next chunk video that remains spatially consistent with the input video, we design a dual spatio-temporal conditioning pipeline to extract the most relevant information from the input video. The last few frames are utilized to provide temporal conditioning, ensuring motion continuity between conditioning inputs and the generated frames. In parallel, for the spatial conditioning, we first select the most representative frames from the input sequence, lift their static regions into a 3D point cloud using the dynamic mask, and render it along a user-specified camera trajectory to preserve scene geometry.
Refer to caption: x3.png

### 3.2 Towards Scene-Consistent Camera-Controllable Video Generation

The key challenge of scene-consistent camera-controllable video generation lies in reconciling two competing requirements: maintaining consistency with potentially distant frames that share spatial proximity (when the camera returns to similar viewpoints), while evolving dynamic content naturally from the recent temporal context. Ideally, extending the conditioning window of video-to-future-video frameworks to condition on all frames $\mathbf{V}_{\text{in}}$ would ensure optimal global spatial consistency and dynamic smoothness.
However, this quickly becomes impractical as the sequence grows, since standard self-attention incurs quadratic time/memory in the sequence length.

#### Dual spatio-temporal sliding window strategy.

Instead of increasing the temporal window size $w$ of the existing video-to-future-video generation methods, we introduce a dual sliding window strategy that conditions on frames selected along both temporal and spatial axes (Fig. [2](https://arxiv.org/html/2510.14945v1#S1.F2)-(c), and Fig. [3](https://arxiv.org/html/2510.14945v1#S3.F3)). Beyond the standard temporal window that captures recent motion dynamics, we add a spatial window that retrieves frames sharing similar 3D viewpoints, regardless of their temporal distance:

$$ $\mathbf{V}_{\text{out}}=\mathcal{F}(\tilde{\mathbf{V}}_{\text{in}},\mathcal{T},\mathbf{C}),\;\;\text{where}\;\ \tilde{\mathbf{V}}_{\text{in}}=\{\text{Temporal}(w)\}\cup\{\text{Spatial}(T)\},$ (4) $$

where the model $\mathcal{F}$ generates a future sequence $\mathbf{V}_{\text{out}}$ conditioned on $\text{Temporal}(w)$, last $w$ frames of the input video $\mathbf{V}_{\text{in}}[L-w:L]$, and $\text{Spatial}(T)$, the $T$ retrieved frames from the entire input sequence based on viewpoint similarity to the target viewpoint $\mathbf{C}$. This dual conditioning enables the model to reference distant frames that observe the same spatial regions, maintaining scene consistency without processing all $L$ input frames.

While this dual conditioning is conceptually appealing, naïvely retrieving and providing spatially adjacent frames directly would be problematic for our task. Since we aim to generate content beyond the input’s temporal boundary, directly conditioning on much earlier frames would freeze outdated dynamics—for example, a walking person seen many frames ago should not reappear at the same location when synthesizing far-future frames. The spatial conditioning should therefore provide only the persistent scene structure while excluding dynamic content. Rather than retrieving individual frames, we introduce a 3D scene memory $\mathcal{M}$ that represents exclusively the static geometry extracted from all spatially relevant frames.

### 3.3 3D Scene Memory Construction

Our 3D scene memory must efficiently encode spatial relationships across all $L$ frames while extracting only persistent static geometry. To construct the 3D scene memory, we leverage dynamic SLAM frameworks (Li et al., 2024; Zhang et al., 2024) to estimate camera poses and reconstruct 3D structure:

$$ $(\hat{\mathbf{C}},\mathbf{P})=\mathcal{D}_{\text{SLAM}}(V_{\text{in}}),$ (5) $$

where $\hat{\mathbf{C}}=\{\hat{C}_{i}\}_{i=1}^{L}$ are the estimated camera poses, $\mathbf{P}$ represents the aggregated 3D point cloud from the $L$ input frames, and $\mathcal{D}_{\text{SLAM}}(\cdot)$ represents the dynamic SLAM framework. This SLAM integration is effective in that it not only estimates the camera parameters of the input frames but also reconstructs the 3D structure of the scene, which can be further utilized to represent the 3D static geometry in a more efficient manner than other representations (Hong et al., 2024a; b).

While the camera poses $\hat{\mathbf{C}}$ enable efficient spatial retrieval by comparing viewpoint similarity with the target trajectory $\mathbf{C}$, the aggregated 3D point cloud $\mathbf{P}$ still contains both static and dynamic regions. Thus, we now explain our full pipeline on how to identify dynamic regions and only maintain the persistent static geometry of the input video.

#### Dynamic masking for static scene extraction.

Figure: Figure 4: Illustration of dynamic masking for static scene extraction. When aggregating 3D points across frames, moving objects create ghosting artifacts if not properly masked. (a) Without masking, dynamic elements (horses and riders) appear frozen at multiple positions, severely degrading the warped views. (b) With our dynamic masking pipeline, these elements are identified and excluded, resulting in clean static-only point clouds that can be reliably warped to new viewpoints.
Refer to caption: x4.png

Naïvely aggregating points across frames creates ghosting artifacts where moving objects appear frozen at multiple positions, as shown in Fig. [4](https://arxiv.org/html/2510.14945v1#S3.F4)-(a). We address this through a comprehensive three-stage masking pipeline that identifies and excludes all dynamic content as depicted in Fig. [5](https://arxiv.org/html/2510.14945v1#S3.F5).

We begin with pixel-level motion detection following existing works (Zhang et al., 2024; Han et al., 2025). For each frame pair, we compute optical flow using off-the-shelf models (Wang et al., 2024a; Teed & Deng, 2020; An et al., 2025) ($\text{Flow}_{\text{optical}}$) and compare it against the flow induced by camera motion alone ($\text{Flow}_{\text{warp}}$). Regions where the L1 difference exceeds a specific threshold $\tau$ are marked as potentially dynamic:

$$ $M_{i}^{\text{pixel}}=\mathds{1}\left[\|\text{Flow}_{\text{optical}}-\text{Flow}_{\text{warp}}\|_{1}>\tau\right],$ (6) $$

where $\mathds{1}[\cdot]$ denotes the element-wise indicator that returns 1 if the condition holds for a pixel and 0 otherwise, i.e., it flags pixels whose optical flow deviates from the warped flow by more than $\tau$.

However, pixel-level detection captures motion only at specific instants and misses complete object boundaries. We therefore propagate these sparse detections to full objects using SAM2 (Ravi et al., 2024), where we sample points from dynamic pixels in the first frame for prompts. Yet this approach still has limitations: static objects that begin moving in later frames may not be detected if they appear static initially.

Our solution employs backward tracking with CoTracker3 (Karaev et al., 2024) to aggregate motion evidence across the entire sequence. From the sampled points in each frame obtained from our pixel-level motion detection, we track these points from all frames back to $t=0$, capturing motions of objects that move at any point. These aggregated points are used to prompt the final SAM2 pass, producing complete object-level masks $M_{i}^{\text{obj}}$ that cleanly separate all dynamic content (Fig. [4](https://arxiv.org/html/2510.14945v1#S3.F4)-(b)). With the full dynamic mask, we can now obtain the static-only 3D geometry $\mathbf{P}_{\text{static}}$:

$$ $\mathbf{P}_{\text{static}}=\bigcup_{i=1}^{L}\mathbf{P}_{i}\odot(1-M_{i}^{\text{obj}}).$ (7) $$

From the constructed static-only 3D geometry $\mathbf{P}_{\text{static}}$ with our proposed dynamic masking strategy, we now obtain the 3D scene memory:

$$ $\mathcal{M}=(\hat{\mathbf{C}},\mathbf{P}_{\text{static}}),$ (8) $$

where we now explain how this 3D scene memory $\mathcal{M}$ can be used for scene-consistent camera-controllable video generation in the following section.

Figure: Figure 5: Dynamic masking. A three-stage pipeline refines dynamic region detection to produce complete object-level masks: (1) optical-flow differences detect pixel-level motion (Dynamic thresholding); (2) sample points from these regions for all frames and perform backward tracking (BW tracking) with CoTracker3 (Karaev et al., 2024) to aggregate motion evidence across all frames back to $t{=}0$ (dynamic aggregation), capturing objects that move at any time; (3) propagate aggregated points in the first frame to the entire video using SAM2 (Ravi et al., 2024). The resulting dynamic masks cleanly separate moving elements (people, objects) from the static background, enabling construction of the static-only point cloud $\mathbf{P}_{\text{static}}$.
Refer to caption: x5.png

### 3.4 3D Scene Prompting

Having constructed the static-only 3D representation $\mathbf{P}_{\text{static}}$, rather than naïvely retrieving $T$ frames from the input video based on viewpoint similarity, we synthesize static-only spatial frames through the projection of $\mathbf{P}_{\text{static}}$. For each target camera pose $C_{t}\in\mathbf{C}$, we generate the corresponding spatial frame by projecting the static points from the most spatially relevant input frames:

$$ $\text{Spatial}(t)=\Pi(K\cdot C_{t}\cdot\mathbf{P}_{\text{static}}^{(n)}),$ (9) $$

where $\mathbf{P}_{\text{static}}^{(n)}\subset\mathbf{P}_{\text{static}}$ contains points from the top-$n$ spatially adjacent input frames to $C_{t}$, $\Pi(\cdot)$ denotes perspective projection, and $K$ is the camera intrinsic matrix. The complete spatial conditioning becomes $\text{Spatial}(T)=\{\text{Spatial}(t)\}_{t=1}^{T}\in\mathbb{R}^{T\times H\times W\times 3}$, where spatial adjacency is calculated by field-of-view overlap.

This projection-based approach ensures only static content appears in conditioning while providing geometrically consistent views aligned to target poses. Notably, the static point cloud aggregates information from multiple viewpoints, potentially filling regions occluded by dynamic objects. These projected views serve as 3D scene prompts that provide explicit guidance about persistent scene structure, enabling precise camera control without additional encoding modules.

The projected views $\text{Spatial}(T)$ serve as what we term 3D scene prompts—they provide the model with explicit guidance about the persistent scene structure. By conditioning on both $\text{Temporal}(w)$ and $\text{Spatial}(T)$, our framework effectively enables scene-consistent camera-controllable video generation with computational efficiency while preserving the prior for high-quality video synthesis.

### 3.5 Training Dataset Construction

To train our model to effectively handle both static and dynamic scenes while leveraging spatial and temporal conditioning appropriately, we construct training data from diverse video sources with different preprocessing strategies based on scene characteristics.

#### Data processing.

Our training data comes from two primary sources: RealEstate10K (Zhou et al., 2018) containing static scenes, and OpenVid-1M (Nan et al., 2024), featuring dynamic content. For static videos from RealEstate10K, we directly extract 3D scene geometry without dynamic masking, as the scenes contain negligible motion. In contrast, for dynamic videos from OpenVid-1M, we employ a comprehensive dynamic masking pipeline explained in Section [3.3](https://arxiv.org/html/2510.14945v1#S3.SS3) to separate static and dynamic elements.

#### 3D scene memory construction.

For effective training, we require videos of sufficient length to construct meaningful 3D scene context. To enable the model to handle arbitrary-length videos as input, we select long videos as the input video with $L>100$. As our video generation framework $\mathcal{F}(\cdot)$ generates $T$ frames, we designate the last $T$ frames as the ground truth future video that the model should generate, while frames before this window serve as input for constructing the 3D scene memory.

Using MegaSAM (Li et al., 2024), we process the entire video to estimate camera poses $\{\hat{\mathbf{C}}_{i}\}_{i=1}^{L}$ for all $L$ frames and reconstruct the 3D structure. We use frames $V_{\text{in}}\in\mathbb{R}^{(L-T)\times H\times W\times 3}$ (i.e., frames before the last $T$ frames) to construct the 3D scene memory, while the camera poses for the last $T$ frames $\{\hat{\mathbf{C}}_{i}\}_{i=L-(T-1)}^{L}$ are used during training to project the scene memory and generate spatial conditioning.

For static scenes, we directly use the reconstructed geometry from $V_{\text{in}}$. For dynamic scenes, we apply our dynamic masks to extract static-only geometry:

$$ $\mathbf{P}_{\text{static}}=\bigcup_{i=1}^{L-T}\mathbf{P}_{i}\odot(1-M^{\text{obj}}_{i})$ (10) $$

The resulting 3D scene memory $\mathcal{M}=(\hat{\mathbf{C}},\mathbf{P}_{\text{static}})$ provides the spatial context for generating the future $T$ frames. During training, for each target frame $t$ in the last $T$ frames, we use its corresponding camera pose $\hat{\mathbf{C}}_{t}$ to project $\mathbf{P}_{\text{static}}$ from spatially adjacent viewpoints, creating the spatial conditioning Spatial$(T)$ that guides the generation process.

#### Spatial and temporal conditioning.

To train the model to utilize spatial conditioning effectively, it is crucial to incorporate both static and dynamic regions in our training data. The mixture of RealEstate10K and OpenVid-1M ensures the model tackles scenarios where spatial conditioning should dominate (static regions) and also maintains the prior of generating high-fidelity dynamic motions learned from large-scale pretraining.

For each training sample, we provide:

- •
Temporal conditioning: The last $w=9$ frames from the input sequence, capturing recent motion dynamics.
- •
Spatial conditioning: Projected views from the static 3D scene memory, synthesized by selecting the point clouds from the top-$n$ spatially adjacent viewpoints based on field-of-view overlap.

This dual conditioning strategy, trained on both static and dynamic content, enables the model to learn when to rely on spatial prompts for scene consistency versus temporal frames for motion continuity. The diversity in our training data—from completely static architectural scenes to videos with dynamic motion—ensures robust generalization to real-world scenarios where both static and dynamic elements coexist.

## 4 Experiments

### 4.1 Implementation Details

#### Model architecture.

We build upon CogVideoX-I2V-5B (Yang et al., 2024), extending its single-image conditioning to accept dual spatio-temporal inputs with minimal architectural changes. The key modification is repurposing the existing image conditioning channel to accept concatenated latents from both temporal frames and spatial projections. Specifically, we provide the last $w=9$ frames from $\mathbf{V}_{\text{in}}$ as temporal conditioning and $T$ projected views from the static point cloud as spatial conditioning. This enables the DiT backbone to remain entirely unchanged, preserving all pretrained video priors. Both conditions are encoded through the frozen 3D VAE and concatenated channel-wise such that $\mathbf{Z}_{\text{cond}}=\text{Concat}[\mathcal{E}(\text{Temporal}(w)),\mathcal{E}(\text{Spatial}(T))]$.

#### Fine-tuning.

We fully fine-tune the model for a total of 4K iterations with a batch size of 8 using 4 H100 GPUs, which required approximately 48 hours. We used the 16-bit Adam optimizer with a learning rate of $1\times 10^{-5}$, and adopted the same hyperparameter settings as those used in the training of CogVideoX (Yang et al., 2024). For the temporal sliding window, we provide the last 9 frames of the input video, setting $w=9$. For the projection of top-$n$ spatially adjacent views, we set $n=7$.

#### Experimental settings.

We evaluate our method across four key aspects: camera controllability, video quality, scene consistency, and geometric consistency. Since no prior work directly addresses scene-consistent camera-controllable video generation, we compare against two categories of baselines: (1) camera-controllable methods (CameraCtrl (He et al., 2024), MotionCtrl (Wang et al., 2024b), FloVD (Jin et al., 2025), AC3D (Bahmani et al., 2025)) for camera control and video quality metrics, and (2) DFoT (Song et al., 2025), which attempts scene-consistent camera-controllable generation, for spatial and geometric consistency metrics.

We primarily evaluate on 1,000 dynamic videos from DynPose-100K (Rockwell et al., 2025). For scene consistency evaluation, we additionally test on 1,000 static videos from RealEstate10K (Zhou et al., 2018), as static scenes provide clearer spatial consistency assessment.

**Table 1: Evaluation of spatial and geometric consistency. We compare DFoT and Ours on the RealEstate10K (Zhou et al., 2018) and DynPose-100K (Rockwell et al., 2025) datasets. For spatial consistency, we evaluate PSNR, SSIM, and LPIPS on revisited camera trajectories, while for geometric consistency, we report the MEt3R (Asim et al., 2025) metric.**
| Methods | RealEstate10K | DynPose-100K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | MEt3R$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | MEt3R$\downarrow$ |  |
| DFoT Song et al. (2025) | 18.3044 | 0.5960 | 0.3077 | 0.181164 | 12.1471 | 0.3040 | 0.4172 | 0.183202 |
| 3DScenePrompt (Ours) | 20.8932 | 0.7171 | 0.2120 | 0.040843 | 13.0468 | 0.3666 | 0.3812 | 0.124189 |

### 4.2 Scene-consistent video generation

#### Evaluation Protocol.

As mentioned in Section [3.1](https://arxiv.org/html/2510.14945v1#S3.SS1), one of the unique and key challenges in scene-consistent camera-controllable video generation is maintaining spatial consistency over extended durations. From a given input video, we evaluate spatial consistency by generating camera trajectories that revisit the viewpoints in the given video. By matching frames in the generated video and the input video that share the same viewpoint, we calculate PSNR, SSIM, and LPIPS. For RealEstate10K, we evaluate the whole image, whereas we only evaluate the static regions by masking out the dynamic regions for DynPose-100K. We also assess geometric consistency using Met3R (Asim et al., 2025), which measures multi-view alignment of generated frames under the recovered camera pose.

#### Results.

As shown in Tab. [1](https://arxiv.org/html/2510.14945v1#S4.T1), 3DScenePrompt significantly outperforms DFoT across all metrics for both static and dynamic scenes. Most notably, our MEt3R evaluation error drops 77% (0.041 vs 0.181), demonstrating superior multi-view geometric alignment. While DFoT similarly tackles scene-consistent camera-controllable video generation through history guidance, their approach fails to maintain scene-consistency for long sequences due to memory constraints. In contrast, our dual spatio-temporal conditioning enables long-term scene-consistency without causing significant computational overhead. The qualitative comparisons shown in Fig. [6](https://arxiv.org/html/2510.14945v1#S4.F6) also validate the effectiveness of our approach over DFoT.

Figure: Figure 6: Visualization of generated videos following trajectories that revisit early frames in the input video. We visualize and compare frames obtained from DFoT (Song et al., 2025) and Ours. We condition both DFoT and ours to generate a video that follows the camera trajectory shown in the top row (GT Future Video), which revisits a viewpoint in the input (Input Video). The comparison shows that ours shows much more consistent generation, whereas DFoT fails to generate scene-consistent frames mainly due to the limited number of frames it can condition on. Also, the comparison of the camera trajectory shows that our method more faithfully follows the given camera condition.
Refer to caption: x6.png

Figure: Figure 7: Visualization of scene-consistent camera-controllable video generation. Comparison of different methods (Wang et al., 2024b; He et al., 2024; Jin et al., 2025; Bahmani et al., 2025) for generating videos from the same input (shown in Input Video) that follow the camera trajectory shown in the rightmost column (Camera). Our method best preserves scene consistency with the input video. Note the red-boxed regions: while the input video shows a white wall, competing methods either lose scene detail or fail to maintain the original scene structure. In contrast, our approach accurately remembers the white wall and maintains consistent scene elements throughout generation. In addition, when compared with the GT Future Video, ours best follows the camera condition, effectively verifying the strength of our framework for scene-consistent camera-controllable video generation.
Refer to caption: x7.png

### 4.3 Camera-Controllable Video Generation

#### Evaluation Protocol.

We employ the evaluation protocol of previous methods (He et al., 2024; Zheng et al., 2024; Jin et al., 2025) for the camera controllability.
We provide an input image along with associated camera parameters for I2V models (He et al., 2024; Wang et al., 2024b; Jin et al., 2025) and solely provide camera parameters for the T2V model (Bahmani et al., 2025). For our model, we provide the last 9 frames of the input video together with the camera parameters. To evaluate how faithfully the generated video follows the camera condition, we estimate camera parameters from the synthesized video using MegaSAM (Li et al., 2024), and compare the estimated camera parameters against the condition camera trajectory $\mathbf{C}$.

As done in previous works (Hong et al., 2024b; a; Rockwell et al., 2025), the comparison between the estimated and input camera parameters is quantified using three metrics: mean rotation error (mRotErr), mean translation error (mTransErr), and mean error in the camera extrinsic matrices (mCamMC).

**Table 2: Camera controllability evaluation.**
| Methods | DynPose-100K |  |  |
| --- | --- | --- | --- |
| mRotErr (^∘)$\downarrow$ | mTransErr$\downarrow$ | mCamMC$\downarrow$ |  |
| MotionCtrl Wang et al. (2024b) | 3.5654 | 7.8231 | 9.7834 |
| CameraCtrl He et al. (2024) | 3.3273 | 9.5989 | 11.2122 |
| FloVD Jin et al. (2025) | 3.4811 | 11.0302 | 12.6202 |
| AC3D Bahmani et al. (2025) | 3.0675 | 9.7044 | 11.1634 |
| DFoT Song et al. (2025) | 2.3977 | 8.0866 | 9.2330 |
| 3DScenePrompt (Ours) | 2.3772 | 7.4174 | 8.6352 |

For the generated video, we also assess video synthesis performance using the Fréchet Video Distance (FVD) (Skorokhodov et al., 2022) and seven metrics from VBench++ (Huang et al., 2024): subject consistency, background consistency, aesthetic quality, imaging quality, temporal flickering, motion smoothness, and dynamic degree.

#### Results.

We first evaluate camera controllability and compare our method with competitive baselines. As shown in Tab. [2](https://arxiv.org/html/2510.14945v1#S4.T2), our approach consistently outperforms existing methods, indicating 3DScenePrompt is capable of generating videos with precise camera control. We then assess the overall video quality (Tab. [3](https://arxiv.org/html/2510.14945v1#S4.T3)) and provide qualitative comparisons (Fig. [7](https://arxiv.org/html/2510.14945v1#S4.F7)). As observed in Tab. [3](https://arxiv.org/html/2510.14945v1#S4.T3), our method achieves the best generation quality across all metrics for dynamic video generation, which is further supported by the visual results in Fig. [7](https://arxiv.org/html/2510.14945v1#S4.F7).

**Table 3: Evaluation of video generation quality. We assess the quality of generated videos using FVD and VBench++ scores. For FVD, lower values indicate higher video quality. For VBench++ scores, higher values indicate better performance for all metrics. All VBench++ scores are normalized.**
| Methods | DynPose-100K |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FVD | Overall<br>Score | Subject<br>Consist | Bg<br>Consist | Aesthetic<br>Quality | Imaging<br>Quality | Temporal<br>Flicker | Motion<br>Smooth | Dynamic<br>Degree |  |
| MotionCtrl Wang et al. (2024b) | 1017.4247 | 0.5625 | 0.5158 | 0.7093 | 0.3157 | 0.3149 | 0.8297 | 0.8432 | 0.7900 |
| CameraCtrl He et al. (2024) | 737.0506 | 0.6280 | 0.6775 | 0.8238 | 0.3736 | 0.3888 | 0.6837 | 0.6955 | 0.9900 |
| FloVD Jin et al. (2025) | 171.2697 | 0.7273 | 0.7964 | 0.8457 | 0.4722 | 0.5546 | 0.7842 | 0.8364 | 0.9900 |
| AC3D Bahmani et al. (2025) | 281.2140 | 0.7428 | 0.8360 | 0.8674 | 0.4766 | 0.5381 | 0.8020 | 0.8673 | 1.0000 |
| 3DScenePrompt (Ours) | 127.4758 | 0.7747 | 0.8669 | 0.8727 | 0.4990 | 0.5964 | 0.8551 | 0.9260 | 1.0000 |

### 4.4 Ablation studies

**Table 4: Ablation study on varying $n$.**
| Methods | Dynamic<br>mask $\mathcal{M}$ | DynPose-100K |  |  |  |
| --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | MEt3R$\downarrow$ |  |  |
| Ours ($n=1$) | ✓ | 13.0207 | 0.3732 | 0.3771 | 0.124773 |
| Ours ($n=4$) | ✓ | 13.0382 | 0.3733 | 0.3758 | 0.124893 |
| Ours ($n=L$) | ✓ | 13.0206 | 0.3631 | 0.3810 | 0.123507 |
| Ours ($n=7$) | ✗ | 12.2304 | 0.3063 | 0.3821 | 0.134885 |
| Ours ($n=7$) | ✓ | 13.0468 | 0.3666 | 0.3812 | 0.124189 |

We analyze two critical components of our framework: the dynamic masking strategy that separates static and dynamic elements, and the number of spatially adjacent frames $n$ retrieved for spatial conditioning.
Tab. [4](https://arxiv.org/html/2510.14945v1#S4.T4) demonstrates the impact of varying $n$ and the necessity of dynamic masking. Without dynamic masking (4th row), the model suffers significantly across all, showing a large drop of PSNR of approximately 0.8dB and also an increase of MEt3R error. This degradation occurs because unmasksed dynamic objects create ghosting artifacts when warped to new viewpoints, corrupting the spatial conditioning. Regarding the number of spatially adjacent frames, we find that performance stabilizes around $n=7$, with minimal improvements beyond this point, suggesting that 7 frames provide sufficient spatial context while maintaining computational efficiency.

## 5 Conclusion

In this work, we introduced 3DScenePrompt, a framework for scene-consistent camera-controllable video generation. By combining dual spatio-temporal conditioning with a static-only 3D scene memory constructed through dynamic SLAM and our dynamic masking strategy, we enable generating continuations from arbitrary-length videos while preserving scene geometry and allowing natural motion evolution. Extensive experiments demonstrate superior performance in camera controllability, scene consistency, and generation quality compared to existing methods. Our approach opens new possibilities for long-form video synthesis applications where maintaining both spatial consistency and precise camera control is essential.

## References

- Agarwal et al. (2025)
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al.
Cosmos world foundation model platform for physical ai.
*arXiv preprint arXiv:2501.03575*, 2025.
- An et al. (2025)
Honggyu An, Jin Hyeon Kim, Seonghoon Park, Jaewoo Jung, Jisang Han, Sunghwan Hong, and Seungryong Kim.
Cross-view completion models are zero-shot correspondence estimators.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 1103–1115, 2025.
- Asim et al. (2025)
Mohammad Asim, Christopher Wewer, Thomas Wimmer, Bernt Schiele, and Jan Eric Lenssen.
Met3r: Measuring multi-view consistency in generated images.
*arXiv preprint arXiv:2501.06336*, 2025.
- Bahmani et al. (2024)
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al.
Vd3d: Taming large video diffusion transformers for 3d camera control.
*arXiv preprint arXiv:2407.12781*, 2024.
- Bahmani et al. (2025)
Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B. Lindell, and Sergey Tulyakov.
Ac3d: Analyzing and improving 3d camera control in video diffusion transformers.
*Proc. CVPR*, 2025.
- Bai et al. (2025)
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al.
Recammaster: Camera-controlled generative rendering from a single video.
*arXiv preprint arXiv:2503.11647*, 2025.
- Blattmann et al. (2023)
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.
Stable video diffusion: Scaling latent video diffusion models to large datasets.
*arXiv preprint arXiv:2311.15127*, 2023.
- Brooks et al. (2024)
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.
Video generation models as world simulators, 2024.
URL [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators).
- Guo et al. (2023)
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai.
Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.
*arXiv preprint arXiv:2307.04725*, 2023.
- Han et al. (2025)
Jisang Han, Honggyu An, Jaewoo Jung, Takuya Narihira, Junyoung Seo, Kazumi Fukuda, Chaehyun Kim, Sunghwan Hong, Yuki Mitsufuji, and Seungryong Kim.
Dˆ 2ust3r: Enhancing 3d reconstruction with 4d pointmaps for dynamic scenes.
*arXiv preprint arXiv:2504.06264*, 2025.
- He et al. (2024)
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang.
Cameractrl: Enabling camera control for text-to-video generation.
*arXiv preprint arXiv:2404.02101*, 2024.
- He et al. (2025)
Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and Ling Pan.
Pre-trained video generative models as world simulators.
*arXiv preprint arXiv:2502.07825*, 2025.
- Hong et al. (2024a)
Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jisang Han, Jiaolong Yang, Chong Luo, and Seungryong Kim.
Pf3plat: Pose-free feed-forward 3d gaussian splatting.
*arXiv preprint arXiv:2410.22128*, 2024a.
- Hong et al. (2024b)
Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jiaolong Yang, Seungryong Kim, and Chong Luo.
Unifying correspondence pose and nerf for generalized pose-free novel view synthesis.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 20196–20206, 2024b.
- Hong et al. (2023)
Susung Hong, Junyoung Seo, Heeseong Shin, Sunghwan Hong, and Seungryong Kim.
Direct2v: Large language models are frame-level directors for zero-shot text-to-video generation.
*arXiv preprint arXiv:2305.14330*, 2023.
- Huang et al. (2024)
Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, et al.
Vbench++: Comprehensive and versatile benchmark suite for video generative models.
*arXiv preprint arXiv:2411.13503*, 2024.
- Jin et al. (2025)
Wonjoon Jin, Qi Dai, Chong Luo, Seung-Hwan Baek, and Sunghyun Cho.
Flovd: Optical flow meets video diffusion model for enhanced camera-controlled video synthesis.
*arXiv preprint arXiv:2502.08244*, 2025.
- Karaev et al. (2024)
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht.
Cotracker3: Simpler and better point tracking by pseudo-labelling real videos.
*arXiv preprint arXiv:2410.11831*, 2024.
- Knapp & Bohacek (2025)
Václav Knapp and Maty Bohacek.
Synthetic human action video data generation with pose transfer.
In *Synthetic Data for Computer Vision Workshop@ CVPR 2025*, 2025.
- Li et al. (2024)
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely.
Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos.
*arXiv preprint arXiv:2412.04463*, 2024.
- Nan et al. (2024)
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai.
Openvid-1m: A large-scale high-quality dataset for text-to-video generation.
*arXiv preprint arXiv:2407.02371*, 2024.
- Ravi et al. (2024)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al.
Sam 2: Segment anything in images and videos.
*arXiv preprint arXiv:2408.00714*, 2024.
- Ren et al. (2025)
Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao.
Gen3c: 3d-informed world-consistent video generation with precise camera control.
*arXiv preprint arXiv:2503.03751*, 2025.
- Rockwell et al. (2025)
Chris Rockwell, Joseph Tung, Tsung-Yi Lin, Ming-Yu Liu, David F Fouhey, and Chen-Hsuan Lin.
Dynamic camera poses and where to find them.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 12444–12455, 2025.
- Runway (2024)
Runway.
Runway.
[https://runwayml.com/](https://runwayml.com/), 2024.
Accessed: 2024-11-05.
- Seo et al. (2025)
Junyoung Seo, Jisang Han, Jaewoo Jung, Siyoon Jin, Joungbin Lee, Takuya Narihira, Kazumi Fukuda, Takashi Shibuya, Donghoon Ahn, Shoukang Hu, et al.
Vid-camedit: Video camera trajectory editing with generative rendering from estimated geometry.
*arXiv preprint arXiv:2506.13697*, 2025.
- Skorokhodov et al. (2022)
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny.
Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 3626–3636, 2022.
- Song et al. (2025)
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann.
History-guided video diffusion.
*arXiv preprint arXiv:2502.06764*, 2025.
- Teed & Deng (2020)
Zachary Teed and Jia Deng.
Raft: Recurrent all-pairs field transforms for optical flow.
In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16*, pp. 402–419. Springer, 2020.
- Wang et al. (2024a)
Yihan Wang, Lahav Lipson, and Jia Deng.
Sea-raft: Simple, efficient, accurate raft for optical flow.
In *European Conference on Computer Vision*, pp. 36–54. Springer, 2024a.
- Wang et al. (2024b)
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.
Motionctrl: A unified and flexible motion controller for video generation.
In *ACM SIGGRAPH 2024 Conference Papers*, pp. 1–11, 2024b.
- Yang et al. (2024)
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al.
Cogvideox: Text-to-video diffusion models with an expert transformer.
*arXiv preprint arXiv:2408.06072*, 2024.
- Yu et al. (2025)
Mark Yu, Wenbo Hu, Jinbo Xing, and Ying Shan.
Trajectorycrafter: Redirecting camera trajectory for monocular videos via diffusion models.
In *Proceedings of the IEEE/CVF international conference on computer vision*, 2025.
- Zhai et al. (2025)
Shangjin Zhai, Zhichao Ye, Jialin Liu, Weijian Xie, Jiaqi Hu, Zhen Peng, Hua Xue, Danpeng Chen, Xiaomeng Wang, Lei Yang, et al.
Stargen: A spatiotemporal autoregression framework with video diffusion model for scalable and controllable scene generation.
*arXiv preprint arXiv:2501.05763*, 2025.
- Zhang et al. (2024)
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang.
Monst3r: A simple approach for estimating geometry in the presence of motion.
*arXiv preprint arXiv:2410.03825*, 2024.
- Zhang et al. (2023)
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.
Adding conditional control to text-to-image diffusion models.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 3836–3847, 2023.
- Zhang et al. (2025)
Ruihan Zhang, Borou Yu, Jiajian Min, Yetong Xin, Zheng Wei, Juncheng Nemo Shi, Mingzhen Huang, Xianghao Kong, Nix Liu Xin, Shanshan Jiang, et al.
Generative ai for film creation: A survey of recent advances.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 6267–6279, 2025.
- Zhang et al. (2022)
Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman.
Structure and motion from casual videos.
In *European Conference on Computer Vision*, pp. 20–37. Springer, 2022.
- Zheng et al. (2024)
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li.
Cami2v: Camera-controlled image-to-video diffusion model.
*arXiv preprint arXiv:2410.15957*, 2024.
- Zhou et al. (2018)
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.
Stereo magnification: Learning view synthesis using multiplane images.
*arXiv preprint arXiv:1805.09817*, 2018.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: In addition to the weakness section listed above, I have the following questions:

- How exactly is the pose similarity is computed and what are the thresholds?

- The paper argues benefits for consistency, but doesn’t show what happens when the planned camera explores regions poorly covered by SLAM. What happens there?

- How are colors/features assigned when projecting? How do you handle holes or depth ordering?
soundness: 2
strengths: - Dual spatio-temporal conditioning: retrieving spatially adjacent content by pose similarity, not just the most recent frames.

- Static-only 3D scene memory: A practical masking pipeline (flow residuals -> backward point tracking -> SAM2 object masks) to construct a static-only 3D scene memory by excluding dynamics before projecting to target viewpoints.

- Experimental results show that the method performs favorably against baselines. There is also a nice spread of different metrics: geometric consistency, spatial revisitation metrics, camera-following errors, FVD, and VBench.

- Ablations isolate the value of dynamic masking and the number of spatial frames.
summary: The paper proposes 3DScenePrompt, a framework for scene-consistent, camera-controllable video generation from an arbitrary-length input video. The core idea is a dual spatio-temporal conditioning scheme: (i) a short temporal window of recent frames to preserve motion continuity, and (ii) spatial window obtained by projecting a static-only 3D scene memory built from the entire input via dynamic SLAM + dynamic masking. The static memory is rendered to the target viewpoints and concatenated with the temporal window to condition a CogVideoX-I2V backbone. Experiments on RealEstate10K and DynPose-100K report improved spatial/geometric consistency, camera controllability, and overall video quality over baselines including DFoT and recent camera-control methods.
weaknesses: - Fairness of controllability comparisons: Competing camera-control baselines are evaluated with their standard inputs (single image or text + camera), whereas your method consumes last 9 frames + spatial prompts, which is strictly richer conditioning. This can advantage camera trajectory adherence and quality metrics. Consider a variant that is restricted to the same conditioning budget (e.g., single frame + spatial prompts vs single frame) to isolate the benefit of spatial prompts rather than extra temporal frames.

- Running the entire pipeline seems computationally expensive. What is the end-to-end run time during inference compared to the baselines?

- The spatial retrieval is defined by FOV overlap to the planned camera. It’s less clear how the method performs when the planned trajectory explores previously unseen regions or when SLAM coverage is sparse/erroneous.

- How sensitive is the model w.r.t. the output errors/bias of the pretrained models? What are the failure modes? What happens when we use different pretrained models?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What happens when SLAM fails (texture-less scenes, fast motion, dynamic-dominated scenes)?
2. The paper shows only success cases. When does the method fail?
3. Table 4 only compares with/without the full mask. What's the contribution of each stage (pixel detection, SAM2, backward tracking)?
4. The temporal window size is fixed at 9 with no justification or ablation. 
5. There should be a baseline method, that the model does not condition on the projected views, instead the original chosen temporal frames.
5. Does the method work on other video diffusion models beyond CogVideoX?
soundness: 3
strengths: 1. The motivation of the paper is clear: when generating the videos consistent with previous frames, the static and dynamic element must be handled differently, and the underlying 3D geometry must be kept during the camera controlling.

2.  The dual spatio-temporal conditioning strategy is intuitive yet powerful. The recognition that spatial conditioning must provide only static geometry while temporal conditioning handles dynamics is key.

3. Building on frozen CogVideoX preserves pretrained priors and suggests the approach could generalize to other video models.
summary: This paper presents 3DScenePrompt, a framework for generating camera-controllable videos from arbitrary-length input videos while maintaining scene consistency. The key innovation is a dual spatio-temporal conditioning strategy: the model conditions on both temporally adjacent frames (last 9 frames) for motion continuity and spatially adjacent frames for scene consistency. To handle dynamic elements correctly, the authors construct a 3D scene memory using dynamic SLAM that represents only static geometry, extracted via a three-stage dynamic masking pipeline (pixel-level motion detection, SAM2 propagation, and backward tracking with CoTracker3). This static-only 3D representation is projected to target viewpoints to provide spatial prompts while allowing dynamics to evolve naturally. Experiments on RealEstate10K and DynPose-100K show improvements over baselines in scene consistency, camera control, and video quality.
weaknesses: 1. The ablation study is not sufficient, see the question parts.
2. There are some recent works try to address the scene-consistent camera controlled video generation (Line 336), for example, Cameractrl 2 and apt2.
3. The method only tested on CogVideoX.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Regarding weakness 1, can the authors explain why competing explicit 3D memory works are not mentioned and compared against? I think this is critical for the paper's acceptance.

2. Regarding weakness 2, can the author explain the relationship between the proposed dynamic object removal pipeline and existing works?

3. The paper's introduction section inspires the development of the proposed method for long-duration generation. Can the author clarify the maximum duration the model supports? And also, how long were the generated videos used for evaluation? The supplementary material only shows very short example videos, where the original videos are so short and can completely be fitted to the attention as a condition, which eliminates the need for such complex spatial prompting design, and the generated videos are also short, which does not showcase the enhancement using the method.
soundness: 2
strengths: The paper is clearly written. The idea of using explicit 3D point cloud for long-term video generation memory is sound.
summary: The paper proposes a method to maintain geometric and temporal consistency for camera-controlled video extension generation. For temporal consistency, the last 9 frames of the original video are given to the model as a condition. For spatial consistency, the method extracts a 3D point cloud representation of the static scene from the original video and projects the point cloud to corresponding views, and provides the model as a condition, hence the name ScenePrompting. The 3D point cloud only encodes the static scene by masking out the dynamic objects. The paper's evaluation shows improved performance compared to prior methods, and the ablation studies validate the effectiveness of the design choice.
weaknesses: 1. Missing comparisons to competing works [1, 2, 3]. The paper completely does not mention or compare with these prior works that also utilize explicit 3D memory for video generation.

2. Missing related works: CameraCtrl2 [4] explores improved camera conditioning and training on a large-scale foundational model. APT2 [5] explores long-duration and real-time camera-controlled generation. These works can be included in the related works section.

3. The paper proposes methods to remove dynamic objects from video. However, erasing and inpainting dynamic objects from video is an established research topic [5]. The paper does not mention such previous work.

4. The model in the paper can only generate short videos of a few seconds, which limits the actual use case of long-duration memory. The videos in the supplementary materials do not show significant improvement. Also, the method only compares against old works (CameraCtrl1 instead of CameraCtrl2).

[1] Video World Models with Long-term Spatial Memory

[2] Learning 3D Persistent Embodied World Models

[3] WORLDMEM: Long-term Consistent World Simulation with Memory

[4] CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models

[5] Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation

[6] Flow-edge Guided Video Completion

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: I have highlighted the major questions in the weaknesses already, and I am summing them up below(I have summarised them shortly so that it is easier for the reviewer to quote the exact question they are answering. Please refer to the Weakness for the detailed problem and question asked.
1. Why is the spatial structure not preserved in the first supplementary video where two chairs were supposed to be side-by-side, but instead a white pillar or cupboard appears?
2. Is there any error in camera pose estimation that, when back-projected to image space, causes inaccurate projections and leads to inconsistency?
3. Are the comparisons fair, given that your method takes both the initial few frames and the last few frames as input, while the other methods only use the last frame as prior?
4. Are the better qualitative results mainly due to conditioning on more input frames rather than the proposed architectural changes?
5. Why didn’t the authors train a baseline like CameraCtrl or another strong model with minimal modifications to take more input frames and verify whether improvements come from more inputs or architecture changes?
6. What are the limitations of the proposed work?
7. What are the potential future directions or improvements for this work?
8. Why are there so few video results in the supplementary, only a handful instead of 15–20 videos as expected for a video generation paper?
9. Are the generated videos coherent with each other when given the same initial video but different trajectories, especially in the static regions?
10. Why hasn’t the paper compared its method with ReCamMaster, given that its evaluation code has been available since before the ICLR deadline?
soundness: 3
strengths: 1. Proposed new problem statement which is relevant to Video Generation which the previous camera controllable methods which conditions only on the last few frames
2. Dual spatio‑temporal conditioning addresses both motion continuity and scene consistency, overcoming the short‑window limitation of prior future‑video models.
3. A static‑only 3D scene memory cleanly separates persistent geometry from dynamics, avoiding the leakage of outdated moving objects into future frames.
4. Three‑stage dynamic masking (optical‑vs‑warped flow differencing, backward point tracking, SAM2 propagation) yields clean static point clouds without ghosting.
summary: The method conditions a video diffusion backbone on both the last few temporal frames for motion continuity and projected views from a static‑only 3D scene memory for spatial consistency, where the memory is created by dynamic SLAM plus a masking pipeline that removes moving objects using optical‑vs‑warped flow differencing, backward point tracking, and SAM2 propagation to yield clean static point clouds. Static‑only point clouds are projected from spatially adjacent viewpoints to the target camera poses and concatenated with temporal latents through a frozen 3D VAE channel, enabling camera‑controllable generation without modifying the DiT backbone.
weaknesses: 1. In the first video in the supplementary video, I see that the spatial structure is not preserved. In the initial video, there were supposed to be 2 chairs side-by-side but I don’t see them in the generated video even though the 3D scene memory captures it. Instead a white pillor / cupboard seems to appear which is wrong. Why is it so? Is there any error in camera pose estimation that when back projected on image space, they are not projected accurately and may lead to inconsistency?
2. I feel like the comparisons are not fair. The video comparisons are all against methods which only do camera control video generation with only the last frame as prior. But your method takes in the few initial videos frames along with the last few frames for video generation and so automatically your method would be more consistent since it is taking more frames as input. This is an unfair comparison in my opinion. The good qualitative results maybe just because you are conditioning your model with more priors and hence more consistent results or is it because of your proposed new architectural changes that enabled better consistent results. I feel the paper didn’t differentiate these two effectively. I agree that it is a novel problem statement and hence no baselines exist but the authors could have trained CameraCtrl or any of the best model to take in more inputs and trained the model with very minimal changes to effectively clarify if the difference in clarity is from more data inputs or actual model architectural changes. 
3. There are no limitations mentioned in the main paper or the appendix. I request the authors to propose the limitations of the work and the future work. It is important for the reviewers and the community to know the limitations so that there could be more work/discussions in that direction to resolve it. 
4. Minimal video results. For a paper proposing video generation, I would expect a minimum of 15-20 videos to be shown in the supplementary. It is difficult for the reviewer to judge if the results are cherry picked or does it actually generalise well on many videos and different camera trajectory. 
5. There are no results shown in the case where given an initial video and 2-3 different trajectories, are all 2-3 videos coherent to the initial video atleast in the region of static part. Are the generated videos coherent with each other in the common regions?
6. What about a comparison against the newly released ReCamMaster? It is an ICCV paper I assume and hence the evaluation code has been released long before ICLR deadline. I would request the authors to compare this work with their method and show few comparison.


Decision: Accept (Poster)