SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency

## Abstract

Abstract We present Stable Video 4D (SV4D) — a latent video diffusion model for multi-frame and multi-view consistent dynamic 3D content generation.
Unlike previous methods that rely on separately trained generative models for video generation and novel view synthesis, we design a unified diffusion model to generate novel view videos of dynamic 3D objects.
Specifically, given a monocular reference video, SV4D generates novel views for each video frame that are temporally consistent.
We then use the generated novel view videos to optimize an implicit 4D representation (dynamic NeRF) efficiently, without the need for cumbersome SDS-based optimization used in most prior works.
To train our unified novel-view video generation model, we curated a dynamic 3D object dataset
from the existing Objaverse dataset.
Extensive experimental results on multiple datasets and user studies demonstrate SV4D’s state-of-the-art performance on novel-view video synthesis as well as 4D generation compared to prior works.
Project page: https://sv4d.github.io .

## 1 Introduction

The 3D world we live in is dynamic in nature with moving people, playing pets, bouncing balls, waving flags, etc.
Dynamic 3D object generation, also known as 4D generation, is the task of generating not just the 3D shape and appearance (texture) of a 3D object, but also its motion in 3D space. In this work, we tackle the problem of generating a 4D (dynamic 3D) object from a single monocular video of that object. 4D generation enables the effortless creation of realistic visual experiences, such as for video games, movies, AR/VR, etc.

4D generation from a single video is highly challenging as this involves simultaneously reasoning both the
object appearance and motion at unseen camera views around the object.
This is also an ill-posed problem as a multitude of 4D results can plausibly explain a single given video.
There are two main technical challenges in training a robust 4D generative model that can generalize
to different object types and motions.
First, there exists no large-scale dataset with 4D objects to train a robust generative model.
Second, the higher dimensional nature of the problem requires a large number of parameters to represent the 3D shape, appearance, and motion of an object.

Figure: Figure 1: Stable Video 4D (SV4D) framework overview and generated 4D assets. We adapt and train a video diffusion model to generate novel view videos, conditioned on a single-view video and a multi-view orbital video of the first frame. SV4D generated novel view videos are consistent across both view and motion axis, which we directly use to optimize dynamic 3D objects without the cumbersome SDS loss.
Refer to caption: x1.png

As a result,
several recent techniques [; ; ; ; ; ; ; ] optimize 4D content by leveraging priors in pre-trained video and multi-view generative models
via score-distillation sampling (SDS) loss [] and its variants. However, they tend to produce unsatisfactory results due to independent modeling of object motion using video models, and novel view synthesis using multi-view generative models. In addition, they tend to take hours to generate a single 4D object due to time-consuming SDS-based optimization.
A concurrent work [] tries to partially address these issues by jointly sampling novel view videos (along both video frame and view axes) using both video
and multi-view
generative models; and then using the resulting novel view videos for 4D optimization. This results in a better novel view video synthesis with multi-view dynamic consistency, but several inconsistencies still remain due to the use of separate video and multi-view generative models.

In this work, we propose Stable Video 4D (SV4D) model that takes as input a single video, such as of a dynamic object, along with a user-specified camera trajectory around the object, and outputs videos of the object along each of the specified camera views. That is, given a video with $F$ number of frames and a camera trajectory with $V$ number of camera views, SV4D outputs a $V{\mkern-2.0mu\times\mkern-2.0mu}F$ grid of images, as illustrated in Fig. 1. Given a reference video, we first obtain the reference multi-views for the first frame using an off-the-shelf multi-view generator (SV3D []). SV4D then jointly outputs the remaining grid of images as highlighted by a yellow box in Fig. 1.
In contrast to prior and concurrent works, SV4D jointly reasons along both view and motion axes, resulting in state-of-the-art multi-frame and multi-view consistency in the output novel view videos.

Specifically, we start with Stable Video Diffusion (SVD) [], a state-of-the-art video generator, and equip it with two more attention blocks: view attention and frame attention.
The view attention block aligns the multi-view images at each video frame, conditioning on the first view *i.e*, in the reference video.
Similarly, the frame attention block aligns the multi-frame images at each view, conditioning on the first frame at each view, *i.e*, reference multi-view.
This dual-attention design leads to significantly improved dynamic and multi-view consistency compared to prior art.

A key challenge here is that SV4D needs to simultaneously generate the $V{\mkern-2.0mu\times\mkern-2.0mu}F$ grid of images, which can quickly become large with long input videos; making it infeasible to fit into memory even on modern GPUs. As a remedy, we propose techniques to sequentially process an interleaved subset of input frames while also retaining consistency in the output image grid.
After generating the multiple novel view videos, we optimize a 4D representation of the dynamic 3D asset, as illustrated in Fig. 1.

Given the lack of large-scale 4D datasets, we carefully initialized SV4D weights with those from SVD [] and SV3D [] networks, thereby leveraging the priors learned in the existing video and multi-view diffusion models.
To further train SV4D, we carefully curated a subject of Objaverse [; ] dataset with dynamic 3D objects, resulting in the ObjaverseDy dataset.

We perform extensive comparisons of both novel view video synthesis and 4D generation results with respective state-of-the-art methods on datasets with synthetic (ObjaverseDy, Consistent4D []), and real-world (DAVIS [; ; ]) data.
We modify the FVD metric []
to evaluate both video frame and view consistency,
validating the effectiveness and design choices of our approach.
Fig. 1 shows some sample results of our approach.

To summarize, our contributions include:

- •
A novel SV4D network that can simultaneously reason across both frame and view axes. To our knowledge, this is the first work that trains a single novel view video synthesis network using 4D datasets, that can jointly perform novel view synthesis as well as video generation.
- •
A mixed sampling scheme that enables sequential processing of arbitrary long input videos while also retaining the multi-frame and multi-view consistency.
- •
State-of-the-art results on multiple benchmark datasets with both novel view video synthesis as well as 4D generation.

## 2 Related Work

3D Generation.
Here, we refer to the works that generate static 3D content as 3D generation.
DreamFusion [] first proposed to distill priors from the 2D diffusion model via SDS loss to optimize the 3D content from text or image.
Several subsequent works [; ; ; ; ; ; ; ; ; ; ; ; ] try to solve issues caused by the original SDS loss, such as multi-face Janus, slow generation speed, and over-saturated/smoothed generations.
Recent works [; ; ; ; ; ] try to directly predict the 3D model of an object via a large reconstruction model.
Another approach [; ; ; ; ; ; ; ; ; ; ; ] to 3D generation is generating dense multi-view images with sufficient 3D consistency.
3D content is reconstructed based on the dense multi-view images.
We follow this strategy, but generate consistent multi-view videos (instead of images) and then reconstruct the 4D object.

Video Generation.
Recent video generation models [; ; ; ; ; ; ] have shown very impressive performance with consistent geometry and realistic motions.
Video generation models have good generalization capabilities, as they are trained on large-scale image and video data that are easier to collect than large-scale 3D or 4D data.
Hence, they are commonly used as foundation models for various generation tasks.
Well-trained video generative models
have shown their potential to generate multi-view images as a 3D generator [; ; ; ; ].
SV3D [] adapts a latent video diffusion model (Stable Video Diffusion - SVD []) to generate novel multiple views.
In this work, we leverage the pre-trained video generation model for 4D generation by adding an additional view attention layer to align the multiview images.

4D Generation.
On one hand, recent optimization-based methods [; ; ; ; ; ; ; ] can generate 4D content by distilling pre-trained diffusion models in a 4D representation [; ; ] via SDS loss [].
MAV3D [] distills a text-to-video diffusion model and several following works [; ; ; ] combine both video and
multi-view diffusion models to leverage the motion and multi-view priors.
However, they tend to take hours to generate 4D content due to time-consuming optimization.
On the other hand, photogrammetry-based methods [; ] mimic a 3D object capture pipeline by directly generating multi-frame multi-view images of a 4D content with dynamic and multi-view consistency and then directly reconstructing 4D representations with them.
Although these methods have much faster speeds, inference-only pipelines are adopted due to the paucity of the 4D data, thus making the spatial-temporal consistency still unsatisfactory.
In this work, we proposed a monolithic model to generate a more consistent image grid, and we use a carefully curated 4D dataset to train the model.

## 3 Method

Our main idea is to build multi-frame and multi-view consistency in a 4D object by surgically combining the frame-consistency in a video diffusion model, with the multi-view consistency in a multi-view diffusion model. In our case, we choose SVD [] and SV3D [] as the video and multi-view diffusion models respectively, for the advantages reasoned below. However, it is to be noted that any choice of attention-based diffusion models should work.

### 3.1 Novel View Video Synthesis via SV4D

Problem Setting.
Formally, we begin with a monocular input video ${\bf J}\in\mathbb{R}^{F{\mkern-2.0mu\times\mkern-2.0mu}D}$ of a dynamic object of $F$ frames and $D\coloneq 3{\mkern-2.0mu\times\mkern-2.0mu}H{\mkern-2.0mu\times\mkern-2.0mu}W$ dimensions, where $H$ and $W$ are the height and width of each frame. Our goal is to generate an image matrix ${\bf M}\in\mathbb{R}^{V{\mkern-2.0mu\times\mkern-2.0mu}F{\mkern-2.0mu\times
\mkern-2.0mu}D}$ of the 4D object consisting of $F$ dynamic frames and $V$ camera views of each frame. Similar to SV3D [], the multi-view frames follow a camera pose trajectory $\bm{\pi}\in\mathbb{R}^{V{\mkern-2.0mu\times\mkern-2.0mu}2}=\{(e_{v},a_{v})\}_{
v=1}^{V}$ as a sequence of $V$ tuples of elevation $e$ and azimuth angles $a$.
We generate this image matrix by iteratively denoising samples from a learned conditional distribution $p({\bf M}|{\bf J},\bm{\pi})$, parameterized by a 4D diffusion model. Due to memory limitation in generating $V{\mkern-2.0mu\times\mkern-2.0mu}F$ images simultaneously, we break down the full generation process into multiple submatrix generation steps.

SV4D Network.
Our goal is to make the generated image matrix $\bf M$ dynamically consistent in the “frame” axis, and multi-view consistent in the “view” axis (see Fig. 2 left).
To achieve this, we condition the image matrix generation on the corresponding frames of the monocular input video $\{{\bf M}_{0,f}\}={\bf J}$ as well as the views from reference multi-view images of the first video frame $\{{\bf M}_{v,0}\}$, which provide motion and multi-view information of the 4D object, respectively.
Without loss of generality, we obtain the reference multi-view images of the first input frame by sampling from a pre-trained SV3D [] model, which can be expressed as $p(\{{\bf M}_{v,0}\}|\{{\bf M}_{0,0}\},\bm{\pi})$.
The overall SV4D novel view video synthesis can be rewritten as sampling from the distribution $p({\bf M}|\{{\bf M}_{0,f}\},\{{\bf M}_{v,0}\},\bm{\pi})$

Figure: Figure 2: SV4D Model Architecture. For camera conditioning, we feed the sinusoidal embedding of camera viewpoints to the convolutional blocks in the UNet, and use the input video for cross-attention conditioning in the spatial and view attention blocks. To improve temporal consistency, we introduce an additional motion attention block, conditioned on the corresponding views of the first frame.
Refer to caption: x2.png

We build the SV4D network based on SVD [] and SV3D [] models to combine the advantages of both video and multi-view diffusion models.
As illustrated in Fig. 2, SV4D consists of a UNet with multiple layers, where each layer contains a sequence of one residual block with Conv3D layers and three transformer blocks with attention layers: spatial, view, and frame attention.
Similar to SV3D, the residual Conv block takes in the noisy latents of flattened image matrix as well as handles the incorporation of conditioning camera poses $\{\bm{\pi}_{v}\}$, and the spatial attention layer handles image-level details by performing attention across the image width and height axes.
To better capture motion cues in the input monocular video $\{{\bf M}_{0,f}\}$, we concatenate its VAE latents to the noisy latents ${\bf z}_{n}$ before feeding it to the UNet.

For multi-view consistency, the view attention block transposes the features and performs attention in the multi-view axis.
By using the CLIP embedding of corresponding input frames $\{{\bf M}_{0,f}\}$ as cross-attention conditioning, it allows the network to learn spatial consistency across novel views while maintaining the semantic context from the input video.

To further ensure dynamic consistency across video frames, we insert a frame attention layer in each UNet block, which applies the attention mechanism in the video frame dimension.
The frame attention of each novel view video is conditioned on the corresponding reference view (of the first frame) $\{{\bf M}_{v,0}\}$ via cross-attention, allowing the network to preserve dynamic coherence starting from the first frame.
We initialize the weights of the frame attention layers from SVD and the rest of the network from SV3Dp, to leverage the generalizability as well as rich dynamic and multi-view priors learned from large-scale video and 3D datasets.

ObjaverseDy Dataset.
Considering that there exists no large-scale training datasets with dynamic 3D objects, we curate a new 4D dataset from the existing Objeverse dataset [; ], a massive dataset with annotated 3D objects.
Objaverse includes
animated 3D objects, however, several of these animated 3D objects are not suitable for training due to having too few animated frames or insufficient motion.
In addition, in the rendering stage, the common rendering and sampling setting may cause some issues.
For example, dynamic objects may be out of the image if the camera distance is fixed because they have global motion;
the motion of objects may be too fast or too slow if the temporal sampling step is fixed.

We follow several steps to curate and clean the 4D objects for our training purposes.
We first filter out the objects based on a review of licenses.
Then, we remove the objects whose animated frames are
too few.
To further filter out objects with minimal motion, we subsample
keyframes
from each video and apply simple thresholding on the maximum L1 distance between these frames as motion measurement.
To render the training novel view videos, we flexibly choose the camera distance from the object. Starting from a base value, we increase the camera distance
until the object fits within all frames of the rendered images.
We also dynamically adjust the temporal sampling step.
Starting from a base value, we increase the temporal sample step
until the L1 distance between consecutive keyframes exceeds a certain threshold.
These steps ensure a high-quality collection of 4D objects,
with rendered multi-view videos that form our ObjaverseDy dataset.

Training Details.
We train SV4D on our ObjaverseDy
dataset.
We choose to finetune from the SV3Dp [] model to output 40 frames ($F=5$ frames along each of the $V=8$ views), where the parameters in the frame attention layers are initialized from SVD-xt [].
Similar to SV3D, we train SV4D progressively by first training on the static camera orbits for 40K iterations, then fine-tuning it for 20K iterations on the dynamic orbits.
We use an effective batch size of 16 during training on 2 nodes of 8 80GB H100 GPUs. For more training details, please see the appendix.

Figure: Figure 3: SV4D Model Sampling (7 frames and 3 views are for illustrative purposes only). To extend the generated multi-view videos while preserving temporal consistency, we propose a novel mixed-sampling strategy during inference. We first sample a sparse set of anchor frames, then use the anchor frames as new conditioning images to densely sample/interpolate the middle frames. To ensure a smooth transition between consecutive generations, we alternatively use the first (forward) or last (backward) frame within a time window for conditioning during dense sampling.
Refer to caption: x3.png

Inference Sampling Scheme.
Due to memory limitations, we cannot generate all the novel view frames at once when $F>5$.
To generate the full $V{\mkern-2.0mu\times\mkern-2.0mu}F$ image matrix with arbitrary length videos
during inference, one can naively run the submatrix generation in an auto-regressive manner.
However, we observe that it often leads to severe artifacts due to error propagation during the generation process.
Hence, we design a novel sampling scheme to mitigate the issue.
The proposed sampling scheme is illustrated in Fig. 3.
Note that we only show motion frame extensions here for illustrative purposes.
We first generate a sparse set of anchor frames with SV4D (interleaved sampling) as shown in Fig. 3 (left).
Then we use the anchor frames as new reference views to densely sample the remaining frames (dense sampling).
To ensure a smooth transition between consecutive generations, we alternatively use the first (forward) or last (backward) anchor frame for conditioning at each diffusion step, as shown in Fig. 3 (middle).
In our experiments (Fig. 8), we demonstrate that our sampling scheme can generate novel view videos that are more temporally consistent compared to using an off-the-shelf video interpolation model [] to interpolate in between the SV4D-generated anchor frames.

Spatio-temporal CFG Scaling.
During inference sampling, SVD uses a linearly increasing scale for classifier-free guidance (CFG) over time (video frame axis) to avoid losing context from the first reference frame.
SV3D, on the other hand, adopts a triangular CFG scaling
(in the view axis) when generating an orbital video around a static 3D object to avoid over-sharpened or over-saturated outputs.
However, we observe that neither of these CFG scaling schemes suit the spatio-temporal nature in 4D image matrix generation and often cause undesirable artifacts.
Hence, we propose a novel spatio-temporal CFG scaling which integrates a linearly-increasing scale in the frame axis and a triangular wave in the view axis (linearly increase when moving away from the input view, and linearly decrease when rotating back towards the input view).
Details can be found in the appendix.

### 3.2 4D Optimization from SV4D Generated Novel View Videos

Figure: Figure 4: Overview of Optimization Framework. We first use the reference multi-view images (of the first frame) to optimize a static NeRF represented by a multi-resolution hash grid as well as density and color MLPs. Then, we unfreeze the temporal deformation MLP and optimize the dynamic NeRF with randomly sampled views and frames.
Refer to caption: x4.png

4D Representation.
As shown in Fig. 4, given the novel view videos generated by SV4D, we optimize a 4D representation to reconstruct the dynamic 3D asset.
Formally, we learn a neural representation $\Psi_{\theta}:(\mathbf{x},t)\mapsto(\sigma,c)$ that maps the sampled 3D points $\mathbf{x}=(x,y,z)$ along a camera ray, and the time embedding $t$ (i.e. continuous version of “frame” in the earlier discrete case) to its volumetric density $\sigma\in\mathbb{R}_{+}$ and color $c\in\mathbb{R}^{3}_{+}$.
Similar to D-NeRF [], we represent a 4D object by the composition of a canonical NeRF that captures the static 3D appearance and a deformation field which handles the object motion across time.
For each 3D point $\mathbf{x}$ in the canonical space, we trilinearly interpolate the multi-resolution hash grid features following Instant-NGP [], and decode them as density and color via MLPσ and MLPc, respectively.
The deformation field is represented by an MLP network MLPt conditioned on time embedding $t$, mapping temporal deformation of $\mathbf{x}$ to the common canonical space $\mathbf{x}+\Delta\mathbf{x}$.
Overall, the dynamic NeRF parameters $\theta$ include the canonical hash grid, MLPt, MLPσ, and MLPc.
Since we do not exhaustively sample temporal timestamps or dense spatial views like in prior SDS-based approaches, we observe that this dynamic NeRF representation produces better 4D results compared to other representations such as 4D Gaussian Splatting [], which suffers from flickering artifacts and does not interpolate well across time or views.

Optimization Details.
By leveraging the consistent image matrix generated by SV4D as pseudo ground-truths, we adopt a simple photometry-based optimization without the cumbersome SDS losses.
The reconstruction losses include a pixel-level MSE loss, mask loss, and a perceptual LPIPS [] loss.
Similar to SV3D [], we also use several geometric priors to regularize the output shapes, such as a mono normal
loss similar to MonoSDF [] as well as bilateral depth and normal smoothness losses [] to encourage smooth 3D surfaces where the projected image gradients are low.

For training efficiency and stability, we follow a coarse-to-fine, static-to-dynamic strategy to optimize a 4D representation. That is, we first freeze the deformation field MLPt and only optimize the canonical NeRF on the multi-view images of the first frame, while gradually increasing the rendering resolution from $128{\mkern-2.0mu\times\mkern-2.0mu}128$ to $512{\mkern-2.0mu\times\mkern-2.0mu}512$.
Then, we unfreeze MLPt and randomly sample 4 frames ${\mkern-2.0mu\times\mkern-2.0mu}$ 4 views for training.
Following the static-to-dynamic strategy, we also gradually optimize the time embedding $t$ from low to high temporal frequency.
In our experiments, we find that sampling more timestamps in one batch and progressive optimization techniques are crucial to 4D output quality.
We render the dynamic NeRF at $512{\mkern-2.0mu\times\mkern-2.0mu}512$ resolution and use an Adam [] optimizer to train all model parameters.
The overall optimization takes roughly 15-20 minutes per object. More implementation details can be found in the appendix.

## 4 Experiments

Datasets.
We evaluate SV4D-synthesized novel view videos and 4D optimization results on the synthetic datasets ObjaverseDy, Consistent4D [], and the real-world videos dataset DAVIS [; ; ].
Consistent4D dataset contains some
dynamic objects collected from Objaverse [; ].
We excluded these
objects from our training set to make a fair comparison.
We used the same input video and evaluated views as Consistent4D.
DAVIS dataset consists of
single-view videos in the wild with corresponding foreground masks.
For qualitative evaluation, we selected
single-view videos with static cameras.

Metrics.
We use the SV4D model to generate multiple novel view videos corresponding to the trajectories of the ground truth camera in the evaluation datasets.
We compare each generated image with its corresponding ground-truth, in terms of Learned Perceptual Similarity (LPIPS []) and CLIP-score (CLIP-S) to evaluate visual quality.
In addition, we evaluate the video coherence by reporting FVD [], a video-level metric commonly used in video generation tasks.
We calculate FVD with different ways (see Fig. 8):

- •
FVD-F: calculate FVD over frames at each view.
- •
FVD-V: calculate FVD over views at each frame.
- •
FVD-Diag: calculate FVD over the diagonal images of the image matrix.
- •
FV4D: calculate FVD over all images by scanning them in a bidirectional raster order.

Baselines.
For novel view video synthesis, we compare SV4D with several recent methods capable of generating multiple novel view videos from a single-view video, including
SV3D [], Diffusion^2 [], STAG4D []. It is to be noted that all of these methods are inference-only techniques, and do not involve direct training in the 4D space like our method.
We run SV3D to generate multi-view images at each video frame separately.
Diffusion^2 has not released their code yet.
STAG4D used Zero123++ [] as the multi-view generator, which fixed the view angle, and hence the novel views generated from STAG4D cannot be changed to be consistent with the views evaluated.
We reproduced Diffusion^2 and STAG4D with SV3D as the multi-view generator. SV3D has been shown to generate more consistent 3D results than Zero123++, so this serves as a better baseline.
For 4D generation, we compare SV4D with other methods that can generate 4D representations, including Consistent4D [], STAG4D [], DreamGaussian4D (DG4D) [], GaussianFlow [], 4DGen [], Efficient4D [].

**Table 1: Evaluation of Novel View Video Synthesis on the Consistent4D Dataset. SV4D can achieve better video frame consistency while maintaining comparable image quality. ^† Our reproduced version.**
| Model | LPIPS$\downarrow$ | CLIP-S$\uparrow$ | FVD-F$\downarrow$ |
| --- | --- | --- | --- |
| SV3D [] | 0.129 | 0.925 | 989.53 |
| Diffusion^2^† [] | 0.171 | 0.919 | 1430.35 |
| STAG4D^† [] | 0.131 | 0.929 | 861.88 |
| SV4D | 0.129 | 0.929 | 677.68 |

**Table 3: Evaluation of Novel View Video Synthesis on the ObjaverseDy Dataset. SV4D can get superior performance in video frame consistency and multi-view consistency. ^† Our reproduced version.**
| Model | LPIPS$\downarrow$ | CLIP-S$\uparrow$ | FVD-F$\downarrow$ | FVD-V$\downarrow$ | FVD-Diag$\downarrow$ | FV4D$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| SV3D [] | 0.131 | 0.920 | 729.67 | 375.49 | 526.78 | 690.49 |
| Diffusion^2^† [] | 0.152 | 0.905 | 1067.78 | 442.18 | 612.82 | 977.19 |
| STAG4D^† [] | 0.133 | 0.917 | 652.43 | 469.07 | 636.83 | 546.56 |
| SV4D | 0.136 | 0.920 | 585.09 | 331.94 | 503.02 | 470.46 |

**Table 4: Evaluation of 4D outputs on the ObjaverseDy dataset. SV4D consistently outperforms baselines in terms of all metrics, demonstrating superior performance in visual quality (LPIPS and CLIP-S), video frame consistency (FVD-F), multi-view consistency (FVD-V), and multi-view video consistency (FVD-Diag and FV4D).**
| Model | LPIPS$\downarrow$ | CLIP-S$\uparrow$ | FVD-F$\downarrow$ | FVD-V$\downarrow$ | FVD-Diag$\downarrow$ | FV4D$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Consistent4D [] | 0.165 | 0.896 | 880.54 | 488.38 | 741.52 | 871.95 |
| STAG4D [] | 0.158 | 0.860 | 929.10 | 453.62 | 663.50 | 1003.16 |
| DreamGaussian4D [] | 0.152 | 0.897 | 697.80 | 450.58 | 615.68 | 638.15 |
| SV4D | 0.131 | 0.905 | 659.66 | 368.53 | 525.65 | 614.35 |

### 4.1 Quantitative Comparison

Novel View Video Synthesis.
We quantitatively compare our method with the baselines in terms of novel view video synthesis results.
Table 2 reports the comparisons in the Consistent4D dataset.
Due to the fact that the evaluated views are too sparse, we only report FVD-F in the Consistent4D dataset.
Our method can achieve much better performance in terms of video frame consistency while maintaining comparable performance in terms of image quality.
In particular, our approach has a significant reduction of $31.5\%$ and $21.4\%$ in FVD-F compared to SV3D and STAG4D, respectively.
Table 3 reports the comparisons in the Objaverse dataset.
Our method can achieve much better video frame consistency as well as multi-view consistency while maintaining comparable performance in terms of image quality.
In particular, our approach has much lower FVD-F compared to baseline methods, demonstrating our generated videos are much smoother.
In addition, SV4D achieves better FVD-V which shows better multi-view consistency.
Our method also surpasses the previous state-of-the-art methods in terms of FVD-Diag and FV4D, proving that the synthesized novel view videos have better video frame and multi-view consistency.

4D Generation.
We quantitatively compare our optimized 4D outputs with the baselines in Consistent4D and ObjaverseDy dataset, as shown in Tables 2 and 4, respectively.
Our method consistently outperforms baselines in terms of all metrics, demonstrating superior performance in visual quality (LPIPS and CLIP-S), motion smoothness (FVD-F), multi-view smoothness (FVD-V), and motion-multi-view joint smoothness (FVD-Diag and FV4D).

Figure: Figure 5: Visual Comparison of Novel View Video Synthesis Results. We show two frames in the input videos and two novel-view results of the corresponding frames. Compared to the baseline methods, SV4D outputs contain geometry and texture details that are more faithful to the input video and consistent across frames.
Refer to caption: x5.png

Figure: Figure 6: Visual Comparison of Generated 4D Outputs. By leveraging the consistent multi-view videos generated by SV4D, we can learn higher-quality 4D assets compared to the prior works with SDS-based losses. Our results are more detailed, consistent, and faithful to the input videos.
Refer to caption: x6.png

### 4.2 Visual Comparison

Novel View Video Synthesis.
In Fig. 5 we show the visual comparison of our multi-view video synthesis results against other methods. We observe that applying SV3D frame-by-frame leads to inconsistent geometry and texture in novel view videos. Diffusion^2 slightly improves the temporal coherence but tends to produce blurry or flickering artifacts. STAG4D can produce smoother videos but often fails at capturing large motion. Compared to these methods, SV4D can generate high-quality multi-view videos that are detailed, faithful to input videos, and temporally consistent.

4D Generation.
We compare our generated 4D results with the prior methods in Fig. 6. For each video, we render the 4D outputs at two different timestamps and two novel views. Consistent4D and STAG4D often produce blurry outputs with inconsistent geometry and texture. DreamGaussian4D can generate finer texture details but still suffers from flickering artifacts and sometimes creates inaccurate geometry. Moreover, all these methods rely on the computationally expensive SDS loss, which is prone to spatially incoherent results and over-saturated texture. On the contrary, SV4D optimizes 4D assets using purely photometric and geometric losses, resulting in smoother videos with realistic and faithful object appearance.

### 4.3 User Study

In addition to the quantitative evaluation, we also conduct two user studies on our multi-view video synthesis results and 4D outputs, respectively. Concretely, we randomly select 10 real-world videos from the DAVIS dataset and 10 synthetic videos from the Objaverse or Consistent4D datasets. For each video, we randomly choose a novel camera view and ask the user to compare the novel view videos generated by 4 different methods (SV4D and 3 baselines). The users are asked to select a video that “looks more stable, realistic, and closely resembles the reference subject”. For multi-view video synthesis, SV4D results are preferred 73.3% over per-frame SV3D (13.7%), Diffusion^2 (5.0%), and STAG4D (8.0%) among multiple participants. For the optimized 4D outputs, SV4D achieves 60% overall preference against Consistent4D (12.7%), STAG4D (9.7%), and DreamGaussian4D (17.6%) among multiple users.

**Table 5: Evaluation of Novel View Video Synthesis (Anchor Frames Only). SV4D can effectively sample frames with faithful consistency and visual details. $\dagger$ Our reproduced version.**
|  | ObjaverseDy | Consistent4D |  |  |
| --- | --- | --- | --- | --- |
| Model | FVD-F$\downarrow$ | FV4D$\downarrow$ | FVD-F$\downarrow$ | FV4D$\downarrow$ |
| SV3D [] | 700.72 | 831.18 | 656.59 | 780.99 |
| Diffusion^2^† [] | 896.98 | 890.70 | 801.99 | 1093.24 |
| STAG4D^† [] | 765.79 | 669.28 | 601.23 | 659.79 |
| SV4D | 629.22 | 569.08 | 469.73 | 621.00 |

Figure: Figure 7: Illustrations of Video and 4D Metrics. FVD-F evaluates coherence between video frames from a fixed view. FVD-V captures multi-view consistency. We also design FVD-Diag and FV4D to evaluate 4D consistency by traversing the image matrix through different paths.
Refer to caption: x7.png

### 4.4 Ablative Analyses

We conduct several ablation experiments to validate the effectiveness of our model’s design choices.
We summarize the key findings below.

SV4D Can Generate Anchor Frames with Better Consistency.
To validate the quality of anchor frames, we compare the anchor frames (8 views ${\mkern-2.0mu\times\mkern-2.0mu}$ 5 video frames) generated from SV4D and the baselines on consistent4D and ObjaverseDy dataset, as shown in Table 6.
SV4D results have much lower FVD-F and FV4D, showing much better video frame and multi-view consistency while maintaining comparable performance in terms of visual quality.

SV4D Sampling Is Better Than Off-the-shelf Interpolation Method.
Extending anchor frames to the full $V{\mkern-2.0mu\times\mkern-2.0mu}F$ image matrix is not trivial.
In Table 6, we compare our SV4D sampling (ours) and the off-the-shelf interpolation method AMT [].
We also show the results of anchor frames as a reference (SV4D (anchor)).
The model with SV4D sampling outperforms AMT across all metrics.
Fig. 8 validates this observation, where we can see that the SV4D sampling can synthesize distinct images while the results of the off-the-shelf interpolation method have a lot of motion blur.
Table 6 also shows two intuitive sampling strategies, independent and interleaved sampling, which generate the videos with worse consistency.
See the Appendix for details on these two sampling strategies.
These results validate that the SV4D sampling is effective.

## 5 Conclusion

We present SV4D, a latent video diffusion model for novel view video synthesis and 4D generation. Given an input video, SV4D can generate multiple novel view videos that are dynamically and spatially consistent, by leveraging the dynamic prior in SVD and multi-view prior in SV3D within a unified architecture. The generated novel view videos can then be used to efficiently optimize a 4D asset without SDS losses from one or multiple diffusion models. Our extensive experiments show that SV4D outputs are more multi-frame and multi-view consistent than existing methods and generalizable to real-world videos, achieving the state-of-the-art performance on novel view video synthesis and 4D generation. We believe that SV4D provides a solid foundation model for further research on dynamic 3D object generation.

Acknowledgments.
We thank Prem Akkaraju, Christian Laforte, Adam Letts, Josephine Parquet, Savannah Martin, Cathy Lee, Amir Etemadieh, Patricia Wood, Nadiia Klokova, Shan Shan Wong, Ryan Holeman, Jerry Chi, Javad Azimi for their help in various aspects of the model development and release. We also thank Eric Courtemanche for his help with visual results.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to weaknesses.
soundness: 2
strengths: 1. Performance: The proposed method SV4D achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods.

2. Clarity: The paper is well-written and clearly structured, making the methodology and results easy to understand and follow.

3. Technical Novelty: This paper presents Stable Video 4D (SV4D), a latent video diffusion model that simultaneously handles multi-frame video generation and multi-view synthesis for dynamic 3D objects. Unlike previous approaches that utilize separate generative models for each task, SV4D ensures temporal and spatial consistency by generating novel view videos directly from a monocular reference video.
summary: Stable Video 4D (SV4D) introduces a unified latent video diffusion model for generating multi-frame and multi-view consistent 4D content. Unlike previous methods that use separate models for video generation and novel view synthesis, SV4D simultaneously generates novel view videos from a monocular reference video to ensure temporal consistency. These generated videos are then used to optimize a dynamic NeRF without SDS-based optimization. Trained on a curated dynamic 3D object dataset from Objaverse, SV4D demonstrates sota performance in novel-view video synthesis and 4D generation.
weaknesses: 1. For the claim in Section 3.2, line300, "we observe that this dynamic NeRF representation produces better 4D results compared to other representations such as 4D Gaussian Splatting (Wu et al., 2024), which suffers from flickering artifacts and does not interpolate well across time or views.", do you have any experimental results to support this? 

2. The design of SV4D is closely aligned with that of EG4D, as both utilize SVD and SV3D priors. SV4D uses dynamic NeRF and EG4D uses dynamic GS. This significant overlap raises concerns regarding the distinctiveness and novelty of SV4D. To clearly differentiate these two works, the authors should introduce new innovative elements or highlight unique aspects.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: 1) training dataset: I can't find the size of the training dataset in the paper. (Maybe I miss it?) However, as far as I know, many animated objects in objaverse datasets are of low-quality, so I'm curious how many models is used in this work after data filtering. 
2) I'm curious why the authors choose dynamic nerf as the 4D representations instead of dynamic gaussians.
soundness: 3
strengths: 1) The work makes attempts to train a mult-view video diffusion model for 4D generation, which is encouraging and could benifit the community.
 2) The authors test their model on both real-world and synthetic objects to prove the generalization ability of their model.
 3) The authors conduct extensive compairsons with previous mehtods.
summary: This work makes attempts to build a multi-view video diffusion model conditioned on reference single-view video and reference multi-view images.  To inherit knowledge learned by previous 3D diffusion models and video diffusion models, the authors carefully design the multi-view video generation network to combine both of them. Specifically, they insert the frame-attention module of the video generation model to 3D generation model to achieve spatial and temporal consistency. To handle GPU memory problem when inferring long-videos, they propose a mixed-sampling scheme. Extensive experiments with regard to novel view video synthesis and 4D generation demonstrate the advantages of the proposed method.
weaknesses: 1) network design: According to Figure 2, frame-attention is inserted after view-attention, and the output of frame-attention directly serves as the final result. However, since frame-attention only operates on time axis and is not aware of multi-view information, latents processed by it might suffer from multi-view inconsistency, and cannot directly serve as the final result. It is observed that the 4D reconstruction results suffer from blurry effect, and I suspect it is due to multi-view inconsistency of the generated videos.
2) problem setting: The proposed model takes generated reference video and generated multi-view images of the first frame as the input. However, it is possible that the two inputs have conflict with each other since they are generated independently. For example, if the video depicts a object is turning around, it may not be consisent with the generated multi-view images espicially in side views and back views. I don't see the authors show objects with such movements in the experiments.
3) experiments: I'm curious about the real-world video experiements. The authors only train their model on synthetic data, so the generailization to real-world cases might be difficult. I don't find many visualizations of real-world generated objects. Could the authors point out where the visualizations are?
4) Some citations regarding to concurrent works are not accurate, for example (Line245 Wang et al, 2024a) and (Line 149 4Diffusion Yang et al, 2024). The authors are suggested to further check them.
5) The authors claim they can handle arbitrary length videos; however, when the input video is very long, the anchor frames will have little temporal coherence and be out-of-distribution for the proposed model.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	The manuscript has not introduced the scale of ObjaverseDy. Does the author think that the scale of data is sufficient for training a diffusion model for object-centric 4D generation? Is there possibility or necessity for further scaling the data?

2.	L104 claims the capability to handle arbitrarily long input videos. Have the authors tested on substantively longer videos, or is this capability just theoretical?

3.	L264 says that "we only show motion frame extensions here for illustrative purposes." Does this imply that similar extensions could be applied to views?
soundness: 3
strengths: 1.	The paper is overall well-organized and easy to follow.

2.	Their efforts and experience in data curation and training of the 4D generative model have sufficient value to the community, and it can be more valuable with more substantive ablation studies to validate the final design choices.

3.	The proposed mix-sampling strategy is interesting; it effectively extends the local spatial-temporal consistency to a broader scale while avoiding memory constraint.
summary: This manuscript presents SV4D, a framework tailored for the generation of dynamic 3D content. The key contribution of this framework is a diffusion model characterized by a dual-attention design, capable of simultaneously sampling 5 frames $\times$ 8 views with satisfactory consistency in both dimensions. To generate videos of meaningful length within a reasonable memory budget, this work also proposes a mixed-sampling strategy, which includes interleaved sampling of anchor frames and dense sampling of middle frames conditioned on reference at both sides. For the training of this model, the authors meticulously curated a subset of Objaverse comprising high-quality animated 3D objects.
weaknesses: 1.	The main contribution of this work lies in the 4D diffusion model; however, there is no ablation on its design and training. It is important to demonstrate to which extent the results are non-trivial and could provide inspiration for subsequent research. In contrast, the ablation provided in this manuscript primarily focuses on the mixed-sampling strategy, which is , while necessary, more like a stopgap solutions to mitigate the memory limitation in this framework. 

2.	It is encouraged to provide more generated 4D assets in supplementary videos. 
Now the supplementary video is more like a introduction to the motivation and methodology of this work, which is slightly repetitive with the role of the main paper. The reviewer notices that there are only 12 generated dynamic objects shown in the video, each of them rendered in single fixed-view, which is insufficient to demonstrate multi-view consistency of generated assets.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How does SV4D perform on complex, out of distribution high-detail objects or scenes with intricate textures For example, some of the scenes shown in L4GM website https://research.nvidia.com/labs/toronto-ai/l4gm/.
soundness: 3
strengths: 1. SV4D combines video generation and novel view synthesis within a single framework, overcoming the limitations of separate models and ensuring multi-frame and multi-view consistency.
2. The use of view attention and frame attention blocks ensures alignment across different frames and viewpoints, enhancing dynamic consistency.
summary: This paper introduces Stable Video 4D (SV4D), a novel latent video diffusion model designed to generate dynamic, multi-view consistent 3D videos from a single input video. Unlike previous models that separately handled video generation and novel view synthesis, SV4D uses a unified approach to generate multi-frame, multi-view videos that retain both temporal and spatial consistency.  Extensive evaluations on synthetic and real-world datasets show SV4D’s superior performance in novel-view video synthesis and 4D generation.
weaknesses: Although SV4D demonstrates promising results on selected datasets, its performance on a broader range of objects, particularly those with complex or intricate details, is not thoroughly evaluated.

Meta-review: This paper receives ratings of 5,6,8,6. The AC follows the suggestions of the reviewers to accept the paper. The paper is well-written and is easy to follow. The proposed method of mix-sampling strategy, and the use of view attention and frame attention blocks are effective in solving the task. The attempt to train a multi-view video diffusion model for 4D generation is encouraging and is valuable to the community. The paper shows extensive comparisons to previous methods. Although there's a reviewer who gave a negative rating of 5, the AC thinks that the weaknesses pointed out by the reviewer are well-addressed by the authors in the rebuttal and are not significant to reject the paper. Furthermore, the reviewer also mentioned the performance, clarity and technical novelty as the strengths of the paper. These strengths are well-aligned with those pointed out by the other reviewers. Overall, all the weaknesses pointed out by the reviewers are also well-addressed by the authors, and the AC advises the authors to include these updates in their final camera-ready version.

Decision: Accept (Poster)