EG4D: Explicit Generation of 4D Object without Score Distillation

## Abstract

Abstract In recent years, the increasing demand for dynamic 3D assets in design and gaming applications has given rise to powerful generative pipelines capable of synthesizing high-quality 4D objects.
Previous methods generally rely on score distillation sampling (SDS) algorithm to infer the unseen views and motion of 4D objects, thus leading to unsatisfactory results with defects like over-saturation and Janus problem.
Therefore, inspired by recent progress of video diffusion models, we propose to optimize a 4D representation by explicitly generating multi-view videos from one input image.
However, it is far from trivial to handle practical challenges faced by such a pipeline, including dramatic temporal inconsistency, inter-frame geometry and texture diversity, and semantic defects brought by video generation results.
To address these issues, we propose EG4D , a novel multi-stage framework that generates high-quality and consistent 4D assets without score distillation.
Specifically, collaborative techniques and solutions are developed, including an attention injection strategy to synthesize temporal-consistent multi-view videos, a robust and efficient dynamic reconstruction method based on Gaussian Splatting, and a refinement stage with diffusion prior for semantic restoration.
The qualitative results and user preference study demonstrate that our framework outperforms the baselines in generation quality by a considerable margin.
Code will be released at https://github.com/jasongzy/EG4D .

## 1 Introduction

Recent years have seen a surge in the development of generative models capable of producing intelligible text ; ; , photo-realistic images ; ; ; , video sequences ; ; , 3D ; ; ; and 4D (dynamic 3D) assets ; ; ; .
Particularly with 4D assets, manual creation is a laborious task that requires considerable expertise from highly skilled designers. Systems capable of automatically generating realistic and diverse 4D content could greatly streamline the workflows of artists and designers, potentially unlocking new realms of creativity through “generative art” .

Due to the scarcity of open-sourced annotated multi-view dynamic data,
previous works ; ; ; ; ; ; ; ; ; rely on the score distillation sampling (SDS)  or its variants from pre-trained 2D diffusion models to distill information about unseen views and motion of objects.
Despite the impressive performance, their rendering results still suffer from highly saturated texture  and multi-face geometry (Janus problem) , thus leading to less photo-realistic generations.

Motivated by recent progress in video diffusion models ; ; , we propose a novel multi-stage framework, EG4D, for Explicitly Generating 4D videos and then reconstructing 4D assets from them.
EG4D goes beyond simply adapting video generation results, as the synthesized frames inevitably suffer from temporal inconsistency and limited visual quality. More specifically, in the vanilla “frame-by-frame” reconstruction, the independence and diversity of multi-view diffusion will cause appearance inconsistency across different timestamps, particularly in unseen views.

To address these challenges, we first design an attention injection mechanism, allowing each multi-view diffusion inference to perceive temporal information through cross-frame latent exponential moving average (EMA). This training-free strategy effectively alleviates the inconsistency issue at the video level and ensures high-quality training samples for optimizing the following 4D representation. In the next stage of 4D reconstruction, we choose 4D Gaussian Splatting (4D-GS)  as our representation to take advantage of its efficient training and rendering capability.

Moreover, existing GS-based dynamic reconstruction methods ; ; commonly assume that appearance variations between different timestamps are caused by the geometric deformation of Gaussian splats. However, this assumption does not hold since unwanted color variations of texture details still exist in our synthesized images produced by video diffusions, even with the proposed attention injection strategy. We manage to disentangle such detailed texture inconsistencies from desired geometric deformation by introducing an extra color transformation network, enabling texture-consistent 4D rendering.
Furthermore, we leverage image-to-image diffusion models to refine the rendered images and fine-tune our 4D representation, achieving better generation quality.

The qualitative results and human preferences validate that our EG4D outperforms SDS-based baselines by a large margin, producing 4D content with realistic 3D appearance, high image fidelity, and fine temporal consistency.
Extensive ablation studies also showcase our effective solutions to the challenges in reconstructing 4D representation with synthesized videos.

## 2 Related Works

In this section, we present the recent progress of video diffusion models and 4D generation.
More discussion on related works can be found in Appendix A.

Video diffusion models.
Diffusion models , characterized by their superior generative capabilities, have become dominant in the field of video generation  ; ; ; ; . Among them, VDM  replaces the typical 2D U-Net for modeling images with a 3D U-Net. Make-A-Video  successfully extends a diffusion-based T2I model to T2V without text-video pairs.
Text2Video-Zero  achieve zero-shot text-to-video generation using only a pre-trained text-to-image diffusion model without any further fine-tuning or optimization. Following Latent Diffusion Models , Video-LDM  and AnimateDiff  introduce additional temporal layers designed to model the temporal consistency. Stable Video Diffusion , trained on well-curated high quality video dataset, presents robust text-to-video and image-to-video generation capabilities across various domains. Recently, SV3D  adapts image-to-video generation for novel view synthesis by leveraging the generalization and multi-view consistency of the video models.
Different from these works, we aim to explicitly generate 4D videos with both temporal and multi-view consistency using two orthogonal video diffusion models.

4D generation.
Following the line of text-to-3D synthesis ; ; , one line of research explores the text-conditioned 4D generation ; ; ; ; ; ; ; ; . They use score distillation sampling (SDS)  to optimize the 4D representations, like KPlanes , Hexplanes  and Deformable Gaussians .
MAV3D  employs temporal SDS to transfer the motion from text-to-video diffusions ) to a dynamic NeRF.
4D-fy  exploits hybrid score distillation methods by alternating optimization procedure to improve the structure and quality of the 4D model.
AYG  explores compositional 4D generation with 3D Gaussian Splatting.
Inspired by recent advancement in image-to-3D models ; ; , several works ; ; explore the field of image/video-conditioned 4D generation.
Animate124  pioneers on this task in a coarse-to-fine fashion: it first optimizes deformation with multi-view diffusions, then corrects the details with ControlNet .
DreamGaussian4D  adopts explicit modeling of spatial transformations in Gaussian Splatting, achieving minute-level generation.
Although score distillation algorithm can infer motion and unseen views from 2D diffusion models, it suffers from some imperfections like over-saturation and Janus problem.
Our framework gets around these problems by explicitly generating multi-views of dynamic object, and then uses them to reconstruct 4D representations.

## 3 Preliminaries

Video diffusion.
In this work, we use two different video diffusion models: Stable Video Diffusion  (SVD) and SV3D . SVD generates a sequence of video frames $\{I_{t}|t\in\{0,\cdots,T\}\}$ conditioned on an initial image $I_{0}$ or text prompt.
SV3D is a pose-conditioned image-to-multiviews model that takes a reference image $I_{0}$ and a series of camera poses $\{c_{p}|p\in\{1,\cdots,N\}\}$, producing a sequence of video frames $\{I_{p}|p\in\{1,\cdots,N\}\}$ corresponding to the specified pose (camera parameters) sequence.
Both SVD and SV3D adopt similar video diffusion architecture  with spatial and temporal attention layers.

3D Gaussian Splatting.
3DGS  is an explicit representation using millions of 3D Gaussians to model a scene. Each Gaussian is characterized by a set of learnable parameters as follows:
1) 3D center;
2) 3D rotation;
3) 3D size (scaling factor);
4) view-dependent RGB color represented by spherical harmonics coefficients (with degrees of freedom $k$): $\bm{h}\in\mathbb{R}^{3(k+1)^{2}}\rightarrow\bm{c}\in\mathbb{R}^{3}$;
5) opacity.
Here a color decoder $\Phi^{sh}$ is used to turn the spherical harmonics coefficients $\bm{h}$ and the view direction $\bm{\gamma}$ into an actual RGB color $\bm{c}$.
For a position
in the scene,
each
Gaussian makes its contribution at that coordinate according to the standard Gaussian function weighted by its opacity.
The differentiable rendering of 3DGS applies the splatting techniques .
For a certain pixel, the point-based rendering computes its color by evaluating the blending of depth-ordered points overlapping that pixel via the volume rendering equation .
The optimization of Gaussian parameters is then supervised by the reconstruction loss (difference between rendered and ground-truth images).

Figure: Figure 1: Framework of EG4D. In video generation (right, Sec. 4.1), we use SVD to produce dynamic frames, and then use SV3D equipped with attention injection to generate temporal-consistent multi-view images. In coarse 4D reconstruction (left top, Sec. 4.2), we optimize the 4D Gaussian Splatting with additional color affine transformation with the annotated multi-view images produced by Stage I. In diffusion refinement (left bottom, Sec. 4.3), we freeze the canonical Gaussians and further fine-tune the temporal deformation network with images refined by an image-to-image diffusion model.
Refer to caption: x1.png

## 4 4D Object Generation

Given an object image, we want to generate the 4D representation of it, enabling free-view dynamic rendering.
To this end, we introduce a multi-stage framework (generation-reconstruction-refinement) for 4D object generation, as illustrated in Figure 1. Network details can be found in Appendix B.

### 4.1 Stage I: View and Dynamic Generation with Video Diffusions

In this stage, we employ two orthogonal video diffusion models to generate samples for the later 4D representation optimization. Given a reference image, we use SVD  to generate a sequence of video frames $\{I_{t}|t\in\{0,\cdots,T\}\}$, where $t$ is the timestamp.
Next, we utilize SV3D  to generate multi-view images $\{I_{t,p}|p\in\{1,\cdots,N\}\}$ with a predefined camera pose sequence for each frame $I_{t}$.
However, vanilla “frame-by-frame” reconstruction causes significant *temporal differences* due to the diverse nature of SV3D inferences for those frames.
Hence, we hope to exploit temporal context to guide the otherwise independent generating process, thereby obtaining results that are as temporally consistent as possible.
To this end, we introduce the training-free *attention injection* strategy during our SV3D inference.

Specifically, in each self-attention module of the spatial layers of a diffusion UNet, we simultaneously consider the visual information from the current reference frame and the frames at previous timestamps, and implement the attention injection by *spatial KV latent blending*
formulated as

$$ ${\bm{z}_{t}}\leftarrow\alpha\bm{z}^{*}_{t}+(1-\alpha)\bm{z}_{t-1},$ (1) $$

$$ $\bm{Q}=\bm{W}^{q}{\bm{z}}^{*}_{t},\bm{K}=\bm{W}^{k}{\bm{z}_{t}},\bm{V}=\bm{W}^ {v}{\bm{z}_{t}},$ (2) $$

$$ $\text{Attention}(\bm{Q},\bm{K},\bm{V})=\text{Softmax}(\frac{\bm{Q}\bm{K}^{T}}{ \sqrt{d_{k}}}\bm{V}),$ (3) $$

where $\bm{z}_{t}$ is the exponential moving average (EMA) of the current multi-view latent ${z_{t}}^{*}$ and the one from the previous timestamp $z_{t-1}$, with the blending weight $\alpha$. $d_{k}$ is the key dimension.

### 4.2 Stage II: Coarse Reconstruction with Gaussian Splatting

With the synthesized multi-view images $\{I_{t,p}|t\in\{0,\cdots,T\},p\in\{1,\cdots,N\}\}$ of the dynamic object, we optimize a 4D representation of it to enable free-viewpoint rendering.
It is worth noting that in this stage, our objective is *not* simply reconstructing an object according to multi-view observations.
Although the design in Sec. 4.1 significantly alleviates the temporal inconsistency problem, those synthesized “ground-truth” images still suffer from varying degrees of inconsistency in color details.
Therefore, we propose to optimize a 4D representation based on 3D Gaussian Splatting  with additional insights into the robustness against texture inconsistencies and semantic defects.

#### Canonical Gaussians & deformation field.

Considering both performance and efficiency, we build our 4D representation upon 4D Gaussian Splatting (4D-GS) .
4D-GS utilizes a deformation field to predict each Gaussian’s geometric offsets at a given timestamp relative to a mean canonical state.
This deformation field is composed of a multi-resolution HexPlane  and MLP-based decoders.
For each Gaussian at a certain timestamp, the model queries the Hexplane with a 4D coordinate ($x\text{-}y\text{-}z\text{-}t$) and decodes the obtained feature $\bm{f}_{t}$ into the position, rotation, and scaling deformation values. The entire dynamic scene is then jointly reconstructed by optimizing both canonical Gaussians and the deformation field, enabling implicit global interactions of visual information.

#### Color transformation against texture inconsistency.

While vanilla 4D-GS is theoretically able to model temporal inconsistencies through per-frame geometric deformation of Gaussians, it is hard to optimize and leads to significant redundancy in Gaussian quantity .
Even if all the inconsistencies are faithfully reconstructed, these unnatural variations in texture details across time will result in significant degradation of visual performance.
To address this problem, we want to disentangle such detailed texture inconsistencies from geometric deformation. Those temporal differences can still be modeled as per-timestamp states, while one of them can be manually selected to dominate the final temporal-consistent rendering.
We choose a simple but effective way that performs time-specific color transformation.
Formally, a new color decoder denoted by $\Phi^{c}$ is introduced as follows:

$$ $\bm{c}=\Phi^{c}(\bm{h},\bm{\gamma})=\bm{W}^{c}_{t}\Phi^{sh}(\bm{h},\bm{\gamma} )+\bm{b}^{c}_{t},$ (4) $$

$$ $\bm{W}^{c}_{t},\bm{b}^{c}_{t}=\text{MLP}(\bm{f}_{t}),$ (5) $$

where $\bm{h}$ is the spherical harmonics coefficients of Gaussians, $\bm{\gamma}$ is the view direction, and $\Phi^{sh}$ is the spherical harmonic decoder. $\bm{W}^{c}_{t}$ and $\bm{b^{c}_{t}}$ are weights and bias predicted by an extra MLP-based color head from per-Gaussian time-specific feature $\bm{f}_{t}$ from the HexPlane.
Such kind of affine transformation is competent in modeling texture inconsistencies caused by ambient occlusion and other factors ; .
During 4D rendering at test time, we take one of the timestamps, *e.g.*, the first frame, as the reference time and use the corresponding feature $\bm{f}_{0}$ to get the Gaussian colors, thereby rendering texture-consistent 4D assets.

#### Multiscale rendering augmentation.

Generally, for a reconstruction task, supervision with high-resolution ground-truth images can provide more information about high-frequency details and benefit the rendering quality.
However, in our task, those synthesized images often have high-frequency noises at specific views or timestamps.
Training with them leads to meaningless view- and frame-overfitting and adds more burden to later refinement.
To address this issue, we propose a multiscale augmentation strategy. During optimization, we randomly downsample the ground-truth images within a reasonable ratio range. The rendering parameters of the Gaussian rasterizer are modified accordingly, enabling multiscale supervision with the reconstruction loss.

### 4.3 Stage III: Refinement with Diffusion Priors

Figure: Figure 2: Illustration of diffusion refinement.
Refer to caption: x2.png

Videos produced by diffusion models often suffer from semantic defects (Figure 2 left) and motion blur.
Fortunately, image-to-image diffusion models provide a strong prior to refine the semantic details while preserving object identity and style.
We leverage these diffusion-refined images (Figure 2 right) to fine-tune our 4D representation further.
Specifically, we first render an image $I_{t,p}$ at the timestamp $t$ and camera pose $p$.
Then we encode the image $I_{t,p}$ into a VAE latent $w$, add noise to the latent, and feed it into the diffusion UNet for denoising. Finally, the refined image $\hat{I}$ is decoded from the denoised latent $\hat{w}$.
Additionally, to account for per-view quality variations, we introduce a pre-defined view-dependent weight $f(p)$ to the reconstruction loss.
Empirically, we select a sine scheduler for pose-dependent weight, formulated as $f(p)=\sin(\pi\cdot\text{d}(x_{p},x_{0}))$, where $x_{0}$ is the camera center of the first frame and $\text{d}(\cdot,\cdot)$ is the normalized L2 distance.
In total, the diffusion refinement loss $\mathcal{L}_{\text{ref}}$ is formulated as

$$ $\mathcal{L}_{\text{ref}}=f(p)\cdot(\mathcal{L}_{\text{L1}}(I_{t,p},\hat{I})+ \lambda\cdot\mathcal{L}_{\text{LPIPS}}(I_{t,p},\hat{I})),$ (6) $$

where $\mathcal{L}_{\text{LPIPS}}(\cdot,\cdot)$ is the perceptual loss and $\mathcal{L}_{\text{L1}}(\cdot,\cdot)$ is the pixel-wise L1 loss.
To preserve the coarse geometry and texture from Stage II, we use this loss to fine-tune the coarse deformation field while keeping canonical Gaussians frozen.
To avoid error accumulation and unstable supervision, we conduct one-pass refinement: for each view/timestamp, the rendered image at the first iteration are used as the input of diffusion, and the refinement output is shared with all the iterations later.

## 5 Experiments

Figure: Figure 3: Comparison with Animate124 , Consistent4D , and DreamGaussian4D (DG4D) in three cases zelda, monkey-bike and tiger-guitar (better zoom in). The first two columns show the animation results in the same view, and the 3-5 columns demonstrate three other views. The last column illustrates the zoom-in image of the last rendered view.
Refer to caption: x3.png

### 5.1 Experimental Settings

#### Implementation details.

In Stage I, we use SVD-img2vid-xl  to generate 25-frame videos. For multi-view generation, we employ SV3D^p conditioned on a camera pose sequence, *i.e.*, 21 azimuth angles (360^∘ evenly divided) and a fixed 0^∘ elevation. All images are set to a resolution of 576$\times$576. In Stage III, we use SDXL-Turbo  with small strength (0.167) to provide the diffusion prior. For more reproduction details, please refer to the optimization settings in Sec. C.1.

Figure: Figure 4: Qualitative results of our generated 4D objects. We present three consecutive frames rendered from our 4D model from two different views.
Refer to caption: x4.png

#### Evaluation metrics.

Following previous methods ; , we use CLIP-I score that measures the cosine similarity of CLIP  embedding of the given image and the rendered views.
We also conduct a user preference study to evaluate the 3D appearance, image-3D alignment, motion realism, motion range, and overall 4D quality. More details on user study can be found in Sec. C.3.

#### Baselines.

We compare our results with the state-of-the-art image-to-4D methods: Animate124  and DreamGaussian4D .
We also compare with the state-of-the-art video-to-4D method Consistent4D .
For a fair comparison, we feed the SVD-generated videos (exactly the same as ours) to Consistent4D for direct video-to-4D generation.

### 5.2 Results

#### Qualitative results.

Figure 3 demonstrates three cases for comparison between our EG4D and the baselines ; ; .
Our generated results present better image-4D alignment and more realistic 3D appearance, especially in facial details.
Animate124 can not generate image-aligned 4D models because of its strong text guidance.
Consistent4D and DreamGaussian4D produce models with over-saturated and non-realistic appearance (especially in face) due to the inherent limitation of score distillation algorithm.
Figure 4 shows detailed results produced by EG4D.
For each case, we present three temporal-continuous rendered frames from two views.
Illustration of more cases can be found in Appendix E.
Rendered videos are provided in the supplementary for better motion visualization.

**Table 1: User study on image-to-4D methods. Each number represents the percentage of user preference. Error bars correspond to the 95.6$\%$ confidence interval. Bold denotes the best result.**
| Method | Overall Quality | Ref. View Alignment | 3D Appearance | Motion Realism | Motion Range |
| --- | --- | --- | --- | --- | --- |
| Animate124 | 1.10 $\pm$1.24 | 2.24 $\pm$1.77 | 1.65 $\pm$1.52 | 2.19 $\pm$1.75 | 5.39 $\pm$2.70 |
| Consistent4D | 3.88 $\pm$2.31 | 5.00 $\pm$2.60 | 3.99 $\pm$2.34 | 5.66 $\pm$2.76 | 8.67 $\pm$3.36 |
| DreamGaussian4D | 11.27 $\pm$3.78 | 12.17 $\pm$2.91 | 10.29 $\pm$3.63 | 15.42 $\pm$4.32 | 39.96 $\pm$5.83 |
| EG4D (Ours) | 83.75 $\pm$4.41 | 80.59 $\pm$4.73 | 84.07 $\pm$4.37 | 76.73 $\pm$5.05 | 45.98 $\pm$5.93 |

**Table 2: Quantitative results.**
| Method | CLIP-I $\uparrow$ |
| --- | --- |
| Animate124 | 0.8544 |
| Consistent4D | 0.9214 |
| DreamGaussian4D | 0.9227 |
| EG4D (Ours) | 0.9535 |

#### Quantitative results & User study.

LABEL:tab:clip shows that our method has the highest CLIP-I score, which means the rendered images are more semantically similar to the reference image.
User study (Table 1) shows that the recipients are overwhelmingly inclined towards the 4D results generated by our framework.
Almost 80% of the participants think our method is superior in overall quality, reference view consistency, 3D appearance, and motion realism.
Meanwhile, our motion range is on par with the strongest baseline, which is further discussed in Sec. 6.

### 5.3 Ablation Studies

Figure: Figure 5: Ablation on attention injection. Video generation results are shown with two cases at time 0 and a timestamp X afterward. “S-” and “T-” stand for operations in spatial and temporal attention layers of SV3D, respectively. “EMA” denotes the proposed KV latent blending with Exponential Moving Average. “Linear” denotes KV blending with only the first frame. “Res” denotes injection on residual connection instead of KV. $\alpha$ is the blending weight. Different degrees of temporal inconsistency can be observed in all settings except ours (S-EMA, $\alpha=0.5$).
Refer to caption: x5.png

Figure: Figure 6: Effects of color transformation. Our color affine transformation effectively disentangles the texture variation at different timestamps, enabling the rendering of color-consistent dynamics with arbitrary time-specific feature $\bm{f}_{t}$.
Refer to caption: x6.png

#### Attention injection.

In Figure 5, we explore the effect of attention injection by generating videos (Stage I) with different blending weight $\alpha$ and replacing our spatial KV latent blending with three variants:
1) *T-EMA*: similar KV blending is adopted but in temporal attention layers of SV3D, *i.e.*, one frame is blended with all views of the reference timestamp, which results in almost identical (static) results.
2) *S-Res*: the residual term (skip connection) instead of KV latent is blended, which leads to collapse results.
3) *S-Linear*: KV blending is used but only with the first frame. Without EMA, the diffusion model shows degraded generating capability for large motions departing from the reference frame (luigi in Figure 5 right).
Moreover, we observe that the temporal consistency is highly sensitive to blending weight $\alpha$.
For comparison, without any attention injection strategy ($\alpha=0$), views of different timestamps are generated independently, leading to dramatic temporal inconsistency in the back view of android (Figure 5 left). Our proposed spatial KV blending with EMA effectively improves the consistency when $\alpha$ is increased to $0.5$. Please refer to Appendix D for the dynamic attenuation phenomenon when $\alpha>0.5$.

#### Color transformation.

Figure 6 shows the effectiveness of our proposed color transformation in Stage II. Dynamic 3DGS typically models all the inter-frame texture diversity as part of time-specific deformation. With color affine transformation, we manage to disentangle unwanted color inconsistencies and render temporal-consistent texture details from whichever timestamp we select.

#### Multiscale renderer.

Figure 7 shows the effectiveness of our multiscale renderer of Stage II.
We show the training and test PSNR during optimization in the left panel.
It can be observed that the multiscale renderer plays the role of a regularizer that effectively avoids model overfitting (lower training PSNR and similar test PSNR).
The qualitative result in the right panel illustrates that this design avoids overfitting to the noise introduced by the video diffusions.

Figure: (a) Reconstruction quality comparison.
Refer to caption: x7.png

Figure: Figure 8: Ablation on different refinement methods. The leftmost column shows the image rendered by 4D model after the first two stage optimization for the case tiger-guitar. Panels (a) - (e) demonstrate different refinement methods aimed at addressing the semantic defects. However, only the one-pass refinement (ours) successfully adds facial details while keeping the original structure.
Refer to caption: x9.png

#### Refinement strategies.

Figure 8 illustrates ablations of refinement by comparing the visual details before and after applying various refinement techniques.
(a) *Adversarial training*:
many previous works ; ; leverage a GAN discriminator to optimize neural fields. However, we observe that although the discriminator loss converges quickly, the Gaussian points gradually diverge from the object surface, resulting in rendered images turning black after several iterations.
(b) *SDS (score distillation refinement)*:
SDS seeks a single mode for text-aligned 4D representation, leading to unsuccessful refinement.
(c) *Iterative refinement*:
InstructN2N  iteratively updates the supervised dataset (each image is refined for multiple times, different from our *one-pass refinement*) for 3D scene editing. In our task, the diverse outputs from diffusion model result in blurred 4D model under pixel-wise supervision.
(d) *Textured mesh extraction* : experiments show that meshes extracted from 3D Gaussians are not watertight and smooth ; , leading to incoherent appearance.
(e) *One-pass refinement* (Ours): in this way, the refined (supervised) images achieve a balance between detail restoration and preservation of structural integrity and consistency. It can be observed that reasonable details are introduced in regions with noise or semantic defects.

## 6 Conclusion and Discussion

Conclusion.
In this paper, we propose EG4D, a novel framework for 4D generation from a single image.
This approach departs from previous score-distillation-based methodologies, promising not only intrinsic immunity against problems like over-saturation but also capabilities for consistent visual details and dynamics.
We first equip the video diffusions with a training-free attention injection strategy to explicitly generate consistent dynamics and multi-views of the given object.
Then a coarse-to-fine 4D optimization scheme is introduced to further address practical challenges in synthesized videos.
Qualitative and quantitative results demonstrate that EG4D produces 4D objects with more realistic and higher-quality appearance and motion compared with the baselines.

Limitations & Future work.
One limitation is that our framework can not generate high-dynamic motion due to the limited capability of the base image-to-video model  and the consistency-motion trade-off in our attention injection strategy.
Another problem lies in the multi-view diffusion model , which currently struggles to apply precise camera pose conditioning, leading to unsatisfactory reconstruction.
One solution for dynamics is to leverage more advanced video diffusions to generate high-quality and high-dynamic video frames.
Future work could also incorporate adaptive camera pose techniques ; in 4D reconstruction to further improve the robustness.

Broader impact.
Our work transforms a single image into a dynamic 3D object, which could present challenges related to copyright issues, as well as wider implications for privacy and data security.

## Appendix A More Related Works

#### 3D refinement with generative priors.

To deal with view-inconsistency and low quality problems, many works [, , , , , , ] take advantages from generative priors, *e.g.*, adversarial training [] and score distillation sampling (SDS) [] to optimize the 3D representation.
GANeRF [] refines the rendered images with an image-conditional generator and leverages the re-rendered image constraints to guide the NeRF optimization in the adversarial formulation.
InstructNeRF2NeRF [] uses the text-conditioned image generator, InstructPix2pix [], to edit the image rendered by pre-trained NeRF in an iterative manner and updates the underlying 3D representation with the edited images.
ReconFusion [] uses the diffusion priors, Zero-123 [], as a drop-in regularizer to enhance the 3D reconstruction performance, especially for sparse-view scenarios.
In contrast to directly optimizing the implicit representation, another line of researches [, ] first extracts the explicit textured mesh, and then refine the texture in UV-space with diffusion prior and differentiable rendering.
In our paper, in consideration of the artifacts generated in video diffusion, we extend the refinement techniques to the 4D representation.

## Appendix B Network Details

In this section, we unpack the network design in Figure 1.

#### Attention injection.

In Sec. 4.1, we exploit the attention injection strategy to alleviate the temporal difference between multi-view diffusion models.
Figure 9 illustrates its network details: in each spatial attention layer, we replace the self-attention by simultaneously considering the current $z^{*}_{t}$ and previous visual information with EMA.

#### Deformation field with color transformation.

In Sec. 4.2, we use color affine transformation to model the temporal texture variation.
Figure 10 shows the detailed architecture of it.
We first query the time-specific feature $f_{t}$ from the HexPlane [] with the canonical Gaussian positions.
After that, the geometric deformations of Gaussian properties ($\mu$ location, $r$ rotation, and $s$ scale) are predicted with a lightweight decoder.
Additionally, we use the affine color transformation to model the temporal texture variations.
Finally, these deformed Gaussians are rendered into an image.

## Appendix C Additional Experimental Settings

Figure: Figure 9: Network details of Attention Injection. \scalebox{0.8}{\textbf{b}}⃝ denotes the EMA blending operator mentioned in Sec. 4.1. $z_{t}^{*}$ is the multi-view latent at current timestamp $t$, and $z_{t}$ is the blended latent. Previous visual information is injected into the current latent by modifying the original spatial self-attention mechanism.
Refer to caption: x10.png

Figure: Figure 10: Network architecture of our 4D representation. $\bar{\mu},\bar{r},\bar{s}$ represents the canonical Gaussian properties: 3D location, rotation and scale from the coarse stage training in 4DGS []. The time-specific local feature $f_{t}$ is queried from the HexPlane [], where the subscribe $t$ means the time-specific property. Different from vanilla 4DGS, we employ additional color affine transformation to obtain the time-specific color $c_{t}$. The geometric deformations are predicted by a lightweight decoder. Finally, the time-specific Gaussians are rendered to produce an image (right).
Refer to caption: x11.png

### C.1 Optimization Details

We report the optimization of 4D Gaussian splatting for the purpose of reproduction.
Basically, we follow the training recipe from 4DGS [] in the coarse 4D reconstruction stage.
In the semantic refinement stage (Stage III), we fine-tune 4DGS for 5k steps with Adam optimizer. The initial learning rate is set to 1e-4 with exponential decay.
The weight $\lambda$ in diffusion refinement loss is set to 0.5.
Our implementation is primarily based on the PyTorch framework and tested on a single NVIDIA RTX 3090 GPU.

### C.2 Reproduction, Data and Code

We reproduced our baselines (Animate124 [], DreamGaussian4D [], and Consistent4D []) using their official code. Additionally, we have included the input images and videos generated by SVD in the *supplementary materials*.
Apart from the data provided by Animate124 and DreamGaussian4D, we have added three more examples: android, chicken-basketball, and penguin.
The code is also available in the *supplementary materials*.

### C.3 User Study Details

We provide details of the user preference study with two screenshots. Figure 16 illustrates the guidelines: each participant is asked to evaluate images and videos rendered by four different methods across five metrics. Figure 17 shows the image and video samples presented to the participants.
After comparing the images (Figure 17(a)) rendered by different models, participants select the method with the highest "reference image consistency" and "3D appearance". After watching the videos (Figure 17(b)) rendered by different models, participants select the method with the highest "motion realism" and "motion range". Finally, they choose the method with the best overall quality.
We presented several cases to 47 participants and compiled the statistics.
For statistical significance, we make the assumption of multinomial distribution, and report the 2-sigma error bar (95.6% CI). We use standard deviation for error bar calculation.

## Appendix D Extended Ablations

#### Attention injection weight.

Figure 11 analyzes different EMA blending weights $\alpha$ of attention injection in the spatial attention layers.
It is obvious that the increasing blending weight benefits the temporal consistency in texture, *e.g.*, similar white texture in the back and consistent leg geometry.
We also observe that overly high ($>0.5$) blending weight significantly attenuates the object motion range.
This trade-off can be better illustrated by the videos provided in the *supplementary materials*.
Taking both motion range and temporal consistency into consideration, we choose $\alpha=0.5$ as an appropriate blending weight without sacrificing the dynamics.

Figure: Figure 11: Ablation on the EMA blending weight of attention injection. As the blending weight increases, the temporal consistency is significantly improved (similar white textures and consistent leg geometry). However, the overly high ($>0.5$) blending weight leads to a very small motion range (dynamics). To balance the motion range and temporal consistency, we choose the EMA weight as $\alpha=0.5$. Video demonstration can be found in our *supplementary materials*.
Refer to caption: x12.png

#### Number of Gaussians.

In Figure 13, we show the number of Gaussians before and after adding the multiscale renderer.
Guo et al. [] observed that visual overfitting often leads to redundant Gaussian splats in dynamic scene reconstruction, which is hard to optimize and causes unsatisfying rendering results.
With the multiscale renderer, we observe a significant decline of Gaussian points, in addition to the dropped training PSNR reported in Figure 7.

#### Additional results for diffusion refinements.

In Figure 12, the effectiveness of our diffusion refinement is illustrated with zoomed-in details.
It can be observed that the facial and hand details become finer and Gaussian noises are removed after the refinement stage.

Figure: Figure 12: Ablation on diffusion refinement. The left and right panels depict two different view of renderings with the case anya. The results after adding the diffusion refinement show finer facial and hand details with less noisy Gaussians.
Refer to caption: x13.png

Figure: Figure 13: Additional ablation on the multiscale renderer. With the multiscale rendering augmentation in Stage II (darkred), the number of Gaussians declines significantly.
Refer to caption: x14.png

## Appendix E Extended Results

#### Dynamics of our results.

For the best demonstration of our 4D model dynamics, please refer to the *supplementary materials* where you can find videos generated by our 4D model.

#### Multi-view results of our results.

Figure 18 shows the multi-view results of our 4D model, which is a supplement of Figure 4.
Due to the page limit of the main paper, we only show two views of the 4D model there, which is not enough to illustrate the 3D appearance of our model.
To this end, we render our model in more viewpoints: 0^∘, 90^∘, 135^∘, 180^∘, 225^∘, and 270^∘.
The rendered multi-view images show that our method can produce images with high 3D consistency and satisfactory quality.

#### More visual comparisons.

Figure 19 provides additional visual comparisons with our baselines, continuing from Figure 3 in the main paper. We use three additional cases: luigi, anya, and chicken-basketball. The first two columns show animation results from the same view, while column 3 to 5 display three different views. The last column presents a zoomed-in image of the final rendered view.
Multi-view videos for visual comparison can be found in the *supplementary materials*.

#### Efficiency.

Our framework takes approximately 1 hour and 25 minutes on average for each 4D object generation. Specifically, Stage I requires about 40 minutes for video and multi-view generation; Stage II, involving 4D Gaussian Splatting optimization, takes around 25 minutes; and the refinement process takes about 40 minutes.
In previous works, Consistent4D [] and Animate124 [] take about 2.5 and 9 hours, respectively, for 4D generation. Notably, DreamGaussian4D [] achieves extremely short optimization time of 7 minutes.
Our optimization time falls between these, but our framework offers superior view consistency, 3D appearance, and motion quality. Since Stage I appears to be one of the efficiency bottlenecks, future work should focus on incorporating efficient sampling for video diffusion models to boost speed.

#### More applications.

Benefiting from our explicit generation, we can easily adapt EG4Dto both text-to-4D and video-to-4D tasks.
Figure 14 shows the generation results of the text-to-4D. We first feed an example text prompt into SDXL [] to get the high-resolution image. Then this image is transformed into a 4D model with our framework.
Figure 15 shows the results of the video-to-4D. We just skip the dynamic generation step and start with our view synthesis pipeline.

Figure: Figure 14: Text-to-4D results. We feed a text prompt (left top) into SDXL [] to generate a ninja image (left bottom). This image can be transformed into 4D objects with our framework, presenting indirect text-to-4D application. The right panel shows multi-view renderings of the 4D model.
Refer to caption: x15.png

Figure: Figure 15: Video-to-4D results. Our framework can be seamlessly extended to video-to-4D generation. The right panel shows the renderings of our 4D model from four viewpoints. This bird video is taken from Consistent4D [].
Refer to caption: x16.png

Figure: Figure 16: Screenshot of our user study guidelines. Each participant is asked to evaluate the images and videos rendered by 4 different methods with 5 metrics, *i.e.*, reference view consistency, 3D appearance, motion realism, motion range, and overall quality.
Refer to caption: x17.png

Figure: (a) Screenshot of the image evaluation.
Refer to caption: extracted/5626281/figures/user_study_screenshot_image.png

Figure: Figure 18: Multi-view results of our models. This figure is a supplement of Figure 4 in the main paper. The 6 columns show the images rendered by our model in different views: 0^∘, 90^∘, 135^∘, 180^∘, 225^∘, and 270^∘. Multi-view renderings demonstrate the geometry/texture consistency and promising quality of our 4D representation.
Refer to caption: x18.png

Figure: Figure 19: Comparison with Animate124 [], Consistent4D [], and DreamGaussian4D (DG4D) [] in three cases luigi, anya and chicken-basketball (better zoom in).
Refer to caption: x19.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Could the authors provide more quantitative insights into the effects of the attention injection mechanism on temporal consistency?
2. How sensitive is the model’s performance to variations in the blending weight (α) during attention injection?
soundness: 2
strengths: 1. EG4D introduces a unique multi-stage approach that successfully avoids issues associated with score distillation, such as over-saturation and Janus artifacts.
2. The attention injection mechanism helps ensure temporal consistency.
3. Quantitative and qualitative results demonstrate that EG4D produces high-quality 4D content, achieving superior alignment with the reference view and more realistic motion realism compared to baselines like DreamGaussian4D and Animate124.
4. The evaluation is comprehensive. The evaluation metrics, including CLIP-I, PSNR, SSIM, FVD, and CD-FVD, along with a user study, offer robust evidence of the framework's advantages.
summary: The paper introduces EG4D, a multi-stage framework designed for generating 4D objects from a single image input without relying on score distillation sampling (SDS). EG4D employs three key stages: view generation using video diffusion models, coarse 4D reconstruction using Gaussian Splatting, and refinement with diffusion priors. The framework addresses common issues like temporal inconsistencies, over-saturation, and the Janus problem, achieving superior performance over existing 4D generation models. The authors demonstrate the effectiveness of EG4D through both qualitative and quantitative analyses, showing significant improvements in 4D object realism and consistency.
weaknesses: 1. Lack of baseline methods. Recent approaches such as L4GM and efficient4D ( which also leverage 4D Gaussian Splatting) should definitely be included for comparison.
2. Lack of Comparative Analysis. The attention injection technique is novel, but without comparisons to other methods, its effectiveness remains uncertain.
3. Texture Inconsistencies. Despite color transformation, subtle color and texture shifts remain, questioning the robustness of temporal consistency.
4. Limited Support for High-Dynamic Motion. The model struggles with high-dynamic motion, restricting its applicability in scenarios needing greater flexibility.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - For the main comparison results. Did the authors use the same video to compare the proposed method and Dreamgaussian4D? It seems that the generated object motions are very different. If not, is it possible to additionally compare a variant of DreamGausisan4D? In other words, can we use the foreground video rendered by the proposed method and reconstruct to 4D with the framework of DreamGausisan4D?

- For the Mario case, as shown in the supplementary video, contains flickering artifacts on the face regions. Why is this happening? Is it because the refinement stage suffers from the Janus issue?
soundness: 2
strengths: - A multiscale augmentation is proposed for rendering more details.
- A training-free attention injection strategy is introduced to ensure consistency.
- A Diffusion Refinement stage is included to refine the details.
summary: The paper proposes a 4D generation framework that avoids the need of score distillation. Instead, multi-view videos are generated from one input image and a 4D representation is optimized directly.
weaknesses: - In Fig. 1, it seems that the multiscale renderer is not used for the "Diffusion Refinement" stage. What is the motivation for this change? Is it necessary?
- What is the difference of the proposed Diffusion Refinement stage and that in DreamGaussian4D?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Another way is to first generate 3D models and then animate the mesh [4]. I wonder if this line of methods are more effective and flexible.  Could you discuss the potential advantages and limitations of their approach compared to the 3D-then-animate pipeline. Specifically, the authors are  suggested they address aspects like computational efficiency, animation quality, and flexibility in handling different types of objects or motions.

[4] Jiang, Yanqin, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. "Animate3d: Animating any 3d model with multi-view video diffusion." arXiv preprint arXiv:2407.11398 (2024).
soundness: 3
strengths: 1.	The proposed pipeline is clearly motivated and reasonable.
2.	The writing is clear and easy to follow, with a well-structured introduction and related work section
3.	Extensive experiments shows clear performance improvements over previous methods (Table 1, Table 2).
4.	The authors provide a comprehensive ablation study, which helps justify the effectiveness of attention injection, color transformation, multi-scale renderer, refinement strategies.
5.	The supplementary materials are sufficient.
summary: The paper proposes an explicit 4D object generalization method without using SDS loss. The method leverages SVD and SV3D to optimize the 4D representation from the generated videos. Experimental results demonstrate superior performance over previous baselines.
weaknesses: 1.	As shown in the video, the results are flickering and blurry. It seems that the quality is not as good as Diffusion4D [1] (see the examples of the project page of Diffusion4D).  It would be valuable to add a discussion and analysis about the proposed method with Diffusion4D (e.g., generation time, temporal consistency, image fidelity, 3D consistency).

2.	The method is built on SVD and SV3D for 4D generation. It would strengthen the paper if the authors could discuss the different designs between SV4D [2] and proposed method. The authors are suggested to provide both qualitative and quantitative comparisons. In specific, the authors can quantitative comparison table between their method and SV4D on metrics like overall quality (view alignment, 3D consistency, multi-view consistency,  temporal consistency), animation flexibility (e.g. motion range, motion fidelity), and computational efficiency (e.g. FPS, generation time). 

3.	I noticed the examples are some simple images. I wonder if the proposed can handle more complex images like examples in https://mrtornado24.github.io/DreamCraft3D/ and other cases (with multiple objects, complex backgrounds, or challenging lighting conditions).

[1] Liang, Hanwen, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N. Plataniotis, Yao Zhao, and Yunchao Wei. "Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models." arXiv preprint arXiv:2405.16645 (2024).
[2] Xie, Yiming, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. "Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency." arXiv preprint arXiv:2407.17470 (2024).

Meta-review: This paper receives 3x ratings of 6s. The AC follows the recommendations of the reviewers to accept the paper. The reviewers think that the paper is well-written, the proposed method is effective and the experimental results are good. The weaknesses are well-addressed by the authors during the rebuttal and discussion phases, and the reviewers decided to raise their scores.

Decision: Accept (Poster)