EG4D: Explicit Generation of 4D Object without Score Distillation
Authors: Qi Sun
Zhiyang Guo
Ziyu Wan
Jing Nathan Yan
Shengming Yin
Wengang Zhou
Jing Liao
Houqiang Li
Venue: ICLR 2025 Poster (2025)
Keywords: 4D Generation
Primary area: generative models

Abstract: In recent years, the increasing demand for dynamic 3D assets in design and gaming applications has given rise to powerful generative pipelines capable of synthesizing high-quality 4D objects.
  Previous methods generally rely on score distillation sampling (SDS) algorithm to infer the unseen views and motion of 4D objects, thus leading to unsatisfactory results with defects like over-saturation and Janus problem.
  Therefore, inspired by recent progress of video diffusion models, we propose to optimize a 4D representation by explicitly generating multi-view videos from one input image.
  However, it is far from trivial to handle practical challenges faced by such a pipeline, including dramatic temporal inconsistency, inter-frame geometry and texture diversity, and semantic defects brought by video generation results.
  To address these issues, we propose EG4D, a novel multi-stage framework that generates high-quality and consistent 4D assets without score distillation.
  Specifically, collaborative techniques and solutions are developed, including an attention injection strategy to synthesize temporal-consistent multi-view videos, a robust and efficient dynamic reconstruction method based on Gaussian Splatting, and a refinement stage with diffusion prior for semantic restoration.
  The qualitative comparisons and quantitative results demonstrate that our framework outperforms the baselines in generation quality by a considerable margin.

Published as a conference paper at ICLR 2025
EG4D: EXPLICIT GENERATION OF 4D OBJECT WITHOUT SCORE DISTILLATION
Qi Sun Wengang Zhou Jing Liao?.t 2.4
Sun 1.2.
Zhiyang Guo
Ziyu Wan
Jing Nathan Yan
Shengming Yin
Houqiang Li
University of Science and Technology of China "City University of Hong Kong *Comnell University
ABSTRACT
In recent years, the increasing demand for dynamic 3D assets in design and gam- ing applications has given rise powerful generative pipelines capable of syn- thesizing high-quality 4D objects. Previous methods generally rely on score dis- tillation sampling (SDS) algorithm to infer the unseen views and motion of 4D objects, thus leading to unsatisfactory results with defects like over-saturation and Janus problem. Therefore, inspired by recent progress of video diffusion models. we propose to optimize a 4D representation by explicitly generating multi- videos from one input image. However, it is far from trivial to handle practical challenges faced by such pipeline, including dramatic temporal inconsistency, inter-frame geometry and texture diversity, and semantic defects brought by video generation results. To address these issues, we propose EG4D, novel multi-stage framework that generates high-quality and consistent 4D assets without score dis- tillation. Specifically, collaborative techniques and solutions are developed, in- cluding an attention injection strategy to synthesize semporal-consiteen multi- view videos, robust and efficient dynamic reconstruction method based on Gaus- sian Splatting, and a refinement stage with diffusion prior for semantic restoration. The qualitative and quantitative evaluations demonstrate that our framework out- performs the baselines in generation quality by a considerable margin.
INTRODUCTION
Recent years have seen surge in the development of generative models capable of producing intelli-
gible text (Brown et al. 2020 Touvron et al. 2023 OpenAl 2023), photo-realistic images (Ramesh
let al.
2021: Rombach et al. 2022a a Sauer et al. (003a)) video sequences (Skorokhodov et al.,
2022
Bahmani et al. 2023] Singer et al. 2023a), 3D (Chan et al., 2022 Poole et al.] 2023] Lin
al. 2023 Tang et (024b) and 4D (dynamic 3D) assets (Ren al. [023 Jiang al. [024bb] Singer et al. 2023b). Particularly with 4D assets, manual creation is a laborious task that requires considerable expertise from highly skilled designers. Systems capable of automatically generating realistic and diverse 4D content could greatly streamline the workflows of artists and designers, potentially unlocking new realms of creativity through "generative art" (Bailey 2020).
Due to the scarcity of open-sourced annotated multi-view dynamic data, previous works (Yin et al., 20236 Xu al., 2024 Bahmani et al.) 2024a b. Ren et al.
2023 Zhao al., 2023 Jaang et al,, 2024b Singer et al. 2023b; Zheng et al.) 2024 Ling et al.
(024 rely on the score distillation
sampling (SDS) (Poole et al. 2023) or its variants from pre-trained 2D diffusion models to distill information about unseen views and motion of objects. Despite the impressive performance, their rendering results still suffer from highly saturated texture (Wang et al. 2023) and multi-face geom- etry (Janus problem) (Armandpour et al. 2023), thus leading to less photo-realistic generations.
Motivated by recent progress in video diffusion models (Voleti et al. 2024 Blattmann et al., 2023a Brooks et al. 2024), we propose novel multi-stage framework, EG4D, for Explicitly Generating 4D videos and then reconstructing 4D assets from them. EG4D goes beyond simply adapting video
* EEqual contributions; Correspondence authors: Jing Liao and Wengang Zhou;
Code available: gifuuh.com/jasongyy/GG..

---

Published as a conference paper at ICLR 2025
generation results, as the synthesized frames inevitably suffer from temporal inconsistency and lim- ited visual quality. More specifically, in the vanilla frram-by-ffin by-framee reconstruction, the indepen- dence and diversity of multi-view diffusion will cause appearance inconsistency across different timestamps, particularly in unseen views.
To address these challenges, first design an attention injection mechanism, allowing each multi- view diffusion inference to perceive temporal information through cross-frame latent exponential moving average (EMA). This training-free strategy effectively alleviates the inconsistency issue at the video level and ensures high-quality training samples for optimizing the following 4D represen- tation. In the next stage of 4D reconstruction, we choose 4D Gaussian Splatting (4D-GS) (Wu al.) 2024a) as our representation to take advantage of its efficient training and rendering capability.
Moreover, existing GS-based dynamic reconstruction methods (Wu al. 2024a| Yang al. 2023 Huang et al. 2024b) commonly assume that appearance variations between different timestamps are caused by the geometric deformation of Gaussian splats. However, this assumption does not hold since unwanted color variations of texture details still exist in our synthesized images produced by video diffusions, even with the proposed attention injection strategy. We manage to disentangle such detailed texture inconsistencies from desired geometric deformation by introducing an extra color transformation network, enabling texture-conen consistent 4D rendering. Furthermore, we leverage souge-to-image diffusion models to refine the rendered images and fine-tune our 4D representation, achieving better generation quality.
The qualitative results, quantitative evaluations and user preferences validate that our EG4D outper- forms based baselines by a large margin, producing 4D content with realistic 3D appearance, high image fidelity, and fine temporal consistency. Extensive ablation studies also showcase our effective solutions to the challenges in reconstructing 4D representation with synthesized videos.
RELATED WORKS
In this section, present the recent progress of video diffusion models and 4D generation. More discussion on related works can be found in Appendix A
Video diffusion models. Diffusion models (Ho et al.
2020), characterized by their superior gener-
ative capabilities, have become dominant in the field of video generation (Ho et al. 20225] Siinger
ctt al. 2022bb Ho et al. 2022a Blattmann et al. 2023a Yin et al.) 2023a). Among them, VDM (Ho iet al. 2022b) replaces the typical 2D U-Net for modeling images with a 3D U-Net. Make-A- Video (Singer al. 20226) successfully extends a diffusion-based T2I model to T2V without text- video pairs. Video-
(Khachatryan et al., 2023) achieve zero-shot text-to-video generation using only pre-trained text-to-image diffusion model without any further fine-tuning or optimiza- tion. Following Latent Diffusion Models (Rombach et al. 2022b), Video-LDM (Blattmann et al.) 2023b) and AnimateDiff (Guo al. 2024a) introduce additional temporal layers designed to model the temporal consistency. Stable Video Diffusion (Blattmann et al. 2023a), trained on well-curated high quality video dataset, presents robust text-to-video and image-to- to-ige--o-vicoo generation capabilities across various domains. Recently, SV3D (Voleti al. 2024) adapts image-to-video generation for novel view synthesis by leveraging the generalization and multi-view consistency of the video mod- els. Different from these works, we aim to explicitly generate 4D videos with both temporal and multi-view consistency using two orthogonal video diffusion models.
4D generation. Following the line of text-to-3D synthesis (Poole et al. 2023 Wang et al. 2023 Wan et al., 2024), one line of research explores the text-conditioned 4D generation (Yin et al. 2023b Cai et al. [2023 Xu et al. 2024 Bahmani et al. [2024a] Zheng al. 2024 Bahmani al., 2024b. Singer et 2003bb Zheng et al. [024] et 2024). They use score distillation sam- pling (SDS) (Poole et al.) 2023) to optimize the 4D representations, like KPlanes (Fridovich- Keil et al., 2023), Hexplanes (Cao & Johnson, (023) and Deformable Gaussians (Yang et al.) (023).
MAV3D (Singer et al. 2023b) employs temporal SDS to transfer the motion from text-to-video diffusions (Singer et al. (022a)) to dynamic NeRF AYG (Ling et al., 2024 explores composi-
tional 4D generation with 3D Gaussian Splatting. Inspired by recent advancement in image-to-31
models (Liu et al. 2023b a Shi et al. 2024), several works (Zhao et al. 2023 Ren et al.) 2023
Jiang et al. 2024b| Pan et al., 2024) explore the field of waage//vdco-conditio
4D generation.
Animate 124 (Zhao et al. 2023) pioneers this task in a coarse-to-fine fashion: it first optimizes
2

---

Published as a conference paper at ICLR 2025
Coarse 4D Reconstruction
View & Dynamic Generation
Multiscale
Coarse Deformation Field
Gaussian
Attention Injestion
Canonical Goussians
Text Prompt
DorL-Turbo
Deformation Field
Renderer
Canonical Gaussians
LIMPS
D Diffusion Refinement
gradlent
stop
SV3D
view
Figure I: Framework of EG4D. In video generation (right, Sec. 4.1 we use SVD to produce dy- namic frames, and then use SV3D equipped with attention injection to generate multi-view images. In coarse 4D reconstruction (left top. Sec. [4.2), we optimize the 4D Gaussian Splatting with additional color affine transformation with the annotated multi-view images produced by Stage I. In diffusion refinement (left bottom, Sec. 4.3), we freeze the canonical Gaussians and fine-tune the temporal deformation network with images refined by image-to-image diffusion model.
deformation with multi-view diffusions, then corrects the details with ControlNet (Zhang al. 2023). DreamGaussian4D (Ren et al. 2023) adopts explicit modeling of spatial transformations in Gaussian Splatting, achieving minute-leve generation. L4GM (Ren al. (022) trains a large-scale feed-forward network for Gaussian sequences to extend LGM (Tang et al. (2024a) in the temporal dimension without optimization. Recently some concurrent works (Xie al. 2024 Zeng et al.
2024 Liang et al. 2024 Jiang et al., 2024a) also try to exploit the video diffusion models for 4D content generation, by jointly optimize the temporal and view dimensions to train 4D-aware video diffusion model. Although achieving appearance consistency, this approach entangles motion and multi-view generating processes, and typically requires limited 4D datasets and considerable com- puting resources for the training. Although score distillation algorithm infer motion and unseen views from 2D diffusion, suffers from imperfections like over- saturation and Janus problem. Our framework gets around the above problems by explicitly generating temporal-consisentl conssittet multi-vin views of dynamic objects in training- manner, and then uses them to reconstruct 4D representations.
3 PRELIMINARIES
Video diffusion. this work, we use two different video diffusion models: Stable Video Diffu- sion (Blattmann al. 2023a) (SVD) and SV3D (Voleti et al.) 2024). SVD generates a sequence of video frames { E (0, -.. T}} conditioned on an initial image In or text prompt. SV3D is pose-conditioned image-to-muliin tonnge-t-multiiiewss model that takes reference image Io and a series of camera poses {cplp E sponding the specified pose (camera parameters) sequence. Both SVD and SV3D adopt similar video diffusion architecture (Ling et al. 2024) with spatial and temporal attention layers.
producing a sequence of video frames {Lppp E {1.... ,N} corre-
3D Gaussian Splatting. 3DGS Kerbl al. 2023) is explicit representation using millions of 3D Gaussians to model a scene. Each Gaussian is characterized by a set of learnable parameters follows: 1) 3D center; 2) 3D rotation; 3) 3D size (scaling factor); 4) view-dependent RGB color represented by I:D spherical harmonics coefficients: h E 3{+++1))*
c E R3; 5) opacity. Here color decoder hh is used to turn the spherical harmonics coefficients h and the view direction into an actual RGB color e. For position the scene, each Gaussian makes its contribution at that coordinate according to the standard Gaussian function weighted by its opacity. The differentiable rendering of 3DGS applies the splatting techniques (Kerbl et al. 2023). For certain pixel, the point- points overlapping
based rendering computes its color by evaluating the blending of depth- that pixel via the volume rendering equation (Max 1995). The optimization of Gaussian parameters then supervised by the reconstruction loss (difference between rendered and ground-truth images).
3

---

Published as a conference paper at ICLR 2025
4 4D OBJECT GENERATION
Given an object image, we want to generate the 4D representation of it, enabling free-view dy- namic rendering. To this end, introduce multi-stage framework (gennratior-eccmssuuctin---- refinement) for 4D object generation, as illustrated in Figure !| Appendix B
Network details can be found in
4.1
STAGE I: VIEW AND DYNAMIC GENERATION WITH VIDEO DIFFUSIONS
In this stage, employ two orthogonal video diffusion models to generate samples for the later 4D representation optimization. Given a reference image, we use SVD (Blattmann et al. 2023a to generate sequence of video frames { {0,.... T}}, where is the timestamp. Next, we utilize SV3D (Voleti et al. 2024) to generate multi-view images (/1,pP E {1..... [I...,,,]) with a predefined camera pose sequence for each frame 4. However, vanilla Trrame-byfrrame
reconstruction causes significant temporal differences due to the diverse nature of SV3D inferences for those frames. Hence, we hope to exploit temporal context to guide the otherwise independent generating process, thereby obtaining results that are as temporally consistent as possible. To this end, we introduce the training-free attention injection strategy during our SV3D inference.
Specifically, in each self-attention module of the spatial layers of a diffusion UNet, we simultane- ously consider the visual information from the current reference frame and frames at previous timestamps, and implement the attention injection by spatial KV latent blending formulated
21 f az (1 (1-a)2 a)
Q W K Wkzi,
ze: V
W 21, QKT Vdl
(2)
Attention Q, K, V) Softmax
V).
(3)
where 2, is the exponential moving average (EMA) of the current multi-view view latent 2, and the one from the previous timestamp 1;-1, with the blending weight a. dy is the key dimension.
4.2 STAGE II: COARSE RECONSTRUCTION WITH GAUSSIAN SPLATTING
With the synthesized multi-view images [Lr.p / E (0....,T),, E ......,NH of the dynamic object, we optimize a 4D representation of it to enable free-viewpoint rendering. It is worth not- ing that in this stage, our objective is not simply reconstructing an object according to multi-view observations. Although the design in Sec. 4.1 significantly alleviates the temporal inconsistency problem, those synthesized "ground-truth" images still suffer from varying degrees of inconsistency in color details. Therefore, propose to optimize 4D representation based on 3D Gaussian Splat- ting (Kerbl et al., (202)) with additional insights into the robustness against texture inconsistencies and semantic defects.
Canonical Gaussians & deformation field. Considering both performance and efficiency, we build our 4D representation upon 4D Gaussian Splatting (4D-GS) (Wu et al., 2024a). 4D-GS uti- lizes a deformation field to predict each Gaussian's geometric offsets at a given timestamp relative to a mean canonical state. This deformation field is composed of a multi-resolution HexPlane (Cao) & Johnson 2023) and MLP-based decoders. For each Gaussian at certain timestamp, the model queries the Hexplane with 4D coordinate (x-y-z-() and decodes the obtained feature ft into the position, rotation, and scaling deformation values. The entire dynamic scene is then jointly recon- structed by optimizing both canonical Gaussians and the deformation field, enabling implicit global interactions of visual information.
Color transformation against texture inconsistency. While vanilla 4D-GS is theoretically able to model temporal inconsistencies through fer-frame geometric deformation Gaussians, it is hard to optimize and leads significant redundancy in Gaussian quantity (Guo et al. 2024b). Even all the inconsistencies are faithfully reconstructed, these unnatural variations in texture details across time will result in significant degradation of visual performance. To address this problem,
4

---

Published as a conference paper at ICLR 2025
want to disentangle such detailed texture inconsistencies from geometric deformation. Those temporal differences can still be modeled as per-timestamp states, while one of them can be manually selected to dominate the final temporal-consister rendering. We choose simple but effective way that performs time-specific color transformation. Formally, a new color decoder denoted by te is introduced follows:
c = (h) = Wi (h. + b;.
(4)
WF, Dj = MLP(f(),
(5)
where h is the spherical harmonics coefficients of Gaussians, 7 is the view direction, and Ta the spherical harmonic decoder. W and bj are weights and bias predicted by an extra MLP- based color head from per-Gaussian time-specific feature ft from the HexPlane. Such kind of affine transformation competent in modeling texture inconsistencies caused by ambient occlusion and other factors (Li et al. [2022] Darmon et al.j (024). During 4D rendering at test time, we take one of the timestamps, e.g., the first frame, as the reference time and use the corresponding feature fo to get the Gaussian colors, thereby rendering texture-consistent 4D assets.
Multiscale rendering augmentation. Generally, for a reconstruction task, supervision with high- resolution ground-trute tound--rruhh images can provide more information about high-frequenc" details and ben- efit the rendering quality. However, in our task, those synthesized images often have high- sighffreueecc noises at specific views or timestamps. Training with them leads to meaningless view- and frame- overfitting and adds more burden to later refinement. To address this issue, we propose a multiscale augmentation strategy. During optimization, randomly downsample the ground-trui images within a reasonable ratio range. The rendering parameters of the Gaussian rasterizer are modified accordingly, enabling multiscale supervision with the reconstruction loss.
4.3 STAGE III: REFINEMENT WITH DIFFUSION PRIORS
Videos produced by diffusion models often suffer from semantic defects (Figure left) and motion blur. Fortunately, image-to-image diffusion mod- els provide a strong prior to refine the semantic details while preserving object identity and style. We leverage these diffusion-refinee images (Fig- uregtt)hht) to fine-tune our 4D representation fur- ther. Specifically, we first render image It.p at the timestamp t and camera pose p. Then we encode the image It,p into VAE latent w. add noise to the latent, and feed it into the diffusion UNet for denoising. Finally, the refined image i is decoded from the denoised latent τ. Additionally, to account for per-view quality variations, we introduce a pre-defined view-dependent weight fp) the reconstruction loss. Empirically, we se- lect sine scheduler for pose-dependent weight, formulated f(p) = sin (Tr d(zp. zo)), where zo the camera center of the first frame and d(.,,) .) is the normalized L2 distance. In total, the diffusion refinement loss Lief is formulated as
Prompt Interrogate
Text Prompt
SDXL-Turbo
Defective Images
Refined Images
Figure 2: Illustration of diffusion refinement.
Lae = f(x) (Cra 7) + X- LLPIPS (Ie.p, i)),
(6)
where (IPIPS(:.)) )) is the perceptual loss and (..(..)) is the pixel-wise L, loss. To preserve the coarse geometry and texture from Stage II, we use this loss to fine-tune the coarse deformation field while keeping canonical Gaussians frozen. To avoid error accumulation and unstable supervision, we conduct one pass refinement: for each view/timestamp. the rendered image at the first iteration used as the input of diffusion, and the refinement output is shared with all the iterations later.
5

---

Published as a conference paper at ICLR 2025
Consistent4D Animate122
Input Image
DGAD
Ours
ConsistentAD Animate124
Input Image
DGAD
Ours
Animate124
Input Image
Consistent4D
DGAD
Ours
Figure 3: Comparison with Animate 124 (Zhao et al. 2023), Consistent4D (Jiang et al. 2024b) and DreamGaussian4D (DG4D) (Ren et al.) 2023) in three cases zelda, monkey-bike and tiger-quitar (better zoom in). The first columns show the animation results in the same view, and the 3-5 columns demonstrate three other views.

---

Published as a conference paper at ICLR 2025
Input Image
View
View 2
Figure 4: Qualitative results of our generated 4D objects. We present three consecutive frames rendered from our 4D model from two different views.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Implementation details. In Stage I, SVD-img2vid-xx (Blattmann al. 2023a) generate 25-frame videos. For multi-view generation, we employ SV3D² conditioned on a camera pose sequence, i.e., 21 azimuth angles (360 evenly divided) and fixed 0 elevation. All images are to resolution of 576. In Stage III, we use DDXL-Turbo (Sauer et al. 2023b) with small strength (0.167) to provide the diffusion prior. More reproduction details are included Appendix C.I
Evaluation metrics. Following previous methods (Ren et al. 2023 Zhao et al. 2023). we use CLIP-I score that measures the cosine similarity of CLIP (Radford et al. 2021 embedding of the given image and the rendered views. We use novel view synthesis metrics like PSNR, LPIPS, SSIM between the given image and rendered results to measure the image quality. We also use FVD (Un- terthiner al. 2018) between reference video and 4D rendered video to assess the generated 4D quality. consideration quantifying the temporal consistency, we adopt a recent proposed met- ric, CD-FVI (Ge et al., 2024), which prefers temporal consistency instead of the per-frame quality.

---

Published as a conference paper at ICLR 2025
Time
Reference
Independent [a = 0j
[or= 0.3)
(a D.3)
[o=
(o== 0.5)
(a 0.5)
[e 0.5)
(a= 0.5)
Figure 5: Ablation on attention injection. Video generation results are shown with two cases at time 0 and timestamp X afterward. "S-" and "T-" stand for operations in spatial and temporal attention layers of SV3D, respectively. "EMA" denotes the proposed KV latent blending with Ex- ponential Moving Average. "Linear" denotes KV blending with only the first frame. "Res" denotes injection on residual connection instead of KV. a is the blending weight. Different degrees of tem- poral inconsistency be observed in all settings except ours (S-EMA, = 0.5).
Table I: User study on image-to-4D methods. Each number represents the percentage user preference. Error bars correspond to the 95.6% confidence interval. Bold denotes the best result.
Method
Overall Quality
Ref. View Alignment
3D Appearance
Motion Realism
Motion Range
A2 Zhao et al_ 2023
1.10 #1.24
2.24 +1,77
1.65 +1.52
2.19 +1.75
5,39 +2,70
Consistem4DD Orang 20z4h
3,99 +2.34
8.67
DreamGanssinn4D (Ken al. 2023
11.27 +3.78
12.17 +2.91
10.29 #3.63
15.42 +4.32
39.96 #:5.3
(Ours)
84.07
+5.05
uses the features extracted by sulf-supe video representation learning model, instead of pre- trained I3D) (Carreira & Zisserman 2017) feature, mitigate this content bias of FVD to large extent. We also conduct user preference study to evaluate the 3D) appearance, view alignment, mo- tion realism, motion range, and overall quality. Details on user study can be found Appendix C.3
Baselines. We compare our results with the state-of-the-art open-sourced image-to-4D methods: Animate 24 (Zhao et al., 2023) and DreamGausian4DD (Ren et al., 2023). We also compare with the state-of-ther oott-----tth--ttt- video-to-4D method Efficient4D (Pan et al. 2024) and Consistent4D (Jiang et al.) 2024b). For a fair comparison, we feed the SVD-generated vidcos (same as ours) to video-based methhods/DremGGausiinn for direct video-to-4D generation.
5.2 RESULTS
Qualitative results. Figure demonstrates three cases for comparison between our EG4D and the baselines (Ren et al. 2023 Zhao al. 2023 Jiang al. 2024b). Our generated results present bet- ter image-4D alignment and more realistic 3D appearance, especially in facial details. Animate 244 can not generate image-aligned 4D models because of its strong text guidance. Consistent4D and DreanGaassian4D produce models with over-saturated and non-realistic appearance (especially in face) due to the inherent limitation score distillation algorithm. Figure 4 shows detailed results produced by EG4D. For each case, we present three cemporal-continuon rendered frames from two views. Note that our framework is not limited to SVD, which can hardly produce large motions. Our approach can seamlessly incorporate advanced 12V models such as AnimateAnyone Hu 2024 enabling larger and controllable motion synthesis..
Quantitative results & User study. Table Shows that our method has the highest CLIP- I score, which means the ren- dered images are more semanti- cally similar to the reference im- age. In the novel view synthesis metrics (PSNR, SSIM, LPIPS), our results surpass the baselines by a very large margin, which highlight the advantage of pixel-wise optimization of our 4D model over the score distillation. The substantial improvement of FVD and CD-FVD indicate that our framework produces 4D object with the best per-frame quality and tempo- ral consistency. User study (Table shows that the recipients overwhelmingly inclined towards
Table 2: Quantitative results Animate 144 benchmark. Bold
denotes the best result.
Methal
CLIP-11
SSIM
LPIPS
FVDJ
CD-FVD
Animate124 Khoo at pu2s 2u22
0L8544
9,972
0.643
0.361
1904.]
2435.9
Concstemntl Pintocet mooceolai AoI-- Fortll
0.9114
14.17
0.167
0.225
738.02
1006.5
EificenaD (Piee Peris
0.9358
20.38
0.822
0.179
753.1)
891.33
DrounCiusssaa aoreral. 2023
0.9227
9.039
0.637
0.512
801.10
$16.23
BG4D (Chrs)
0.938
23.28
0.904
0.173
142.34
489.10
8

---

Published as a conference paper at ICLR 2025
24
22
20
baseline [train]
multiscale (train)
baseline [test]
multiscale [test]
14
30k
step
baseline
+ multiscale renderer
(a) Reconstruction quality comparison.
(b) Visual details illustration.
Figure 7: Effects of multiscale renderer. (a) demonstrates the training (solid line) test (dashed line) curve before (dark gray) and after (dark red) adding the multiscale renderer. Multiscale render- ing avoids the meaningless overfitting of our model (lower training PSNR, but comparative even higher test PSNR). (b) shows one viewpoint of rendering for case ast tiscale render effectively prevents the model from overfitting noise introduced in video diffusions.
The mul-
the 4D results generated by our framework. Almost 80% of the participants think our method is superior in overall quality, reference view consistency, 3D appearance, and motion realism. Mean- while, our motion range is on par with the strongest baseline, which is further discussed Sec.
5.3 ABLATION STUDIES
Attention injection. In Figure we explore the ef- fect of attention injection by generating videos (Stage I) with different blending weight a and replacing our spa- tial KV latent blending with three variants: 1) T-EMA: similar KV blending is adopted but in temporal atten- tion layers of SV3D, i.e., one frame blended with all views of the reference timestamp, which results in al- most identical (static) results. 2) S-Res: the residual term (skip connection) instead of KV latent is blended, which leads to collapse results. 3) S-Linear: KV blend- ing is used but only with the first frame. Without EMA, the diffusion model shows degraded generating capabil- ity for large motions departing from the reference frame (luigi in Figure 5 right). Moreover, we observe that the temporal consistency is highly sensitive blending weight For comparison, without any attention injec- tion strategy (a 0), views of different timestamps are generated independently. leading to dramatic temporal inconsistency in the back view of android (Figure left). Our proposed spatial KV blending with EMA ef- fectively improves the consistency when a is increased to 0.5. Please refer to Appendix D for the dynamic at- tenuation phenomenon when a 0.5.
color trans. color trans. (render with (render with
w/o color
Time 0
Time 1
Time
Figure Effects of color transforma- tion. Our color affine transformation ef- fectively disentangles the texture variation at different timestamps, enabling the ren- dering of color-consistent dynamics with arbitrary time-specific feature fr.
Color transformation. Figure shows the effective- ness of our proposed color transformation in Stage III Dynamic 3DGS typically models the inter-frame texture diversity as part of time-specife deformation. With color affine transformation, man- age to disentangle unwanted color inconsistencies and render semporal-consistrr texture details from whichever timestamp select.
Multiscale renderer. Figure 7 shows the effectiveness of our multiscale renderer of Stage II. We show the training and test PSNR during optimization in the left panel. can be observed that the multiscale renderer plays the role of regularizer that effectively avoids model overfitting (lower training PSNR and similar test PSNR). The qualitative result in the right panel illustrates that this design avoids overfitting to the noise introduced by the video diffusions.

---

Published as a conference paper at ICLR 2025
Refinement
GAN
Score Distiliation Refinementt
iterativee
Temporal Refinement
Our Refinoment
Discriminator
Refinement
Extraction
Figure 8: Ablation on different refinement methods. The leftmost column shows the image ren- dered by 4D model after the second stage optimization for the case tiger-quitar. Panels (a)-(f) demonstrate different refinement methods aimed at addressing the semantic defects. However, only the one-pass refinement (ours) successfully adds facial details while keeping the original structure.
Refinement strategies. Figure 8 illustrates ablations of refinement by comparing the visual details before and after applying various refine- ment techniques. (a) Adversarial training: many previous works (Chen et al. 2024 Roessle et al., 2023) leverage a GAN discriminator to opti- mize neural fields. However, observe that although the diseriminator loss converges quickly, the Gaussian points gradually diverge from the object surface, resulting in rendered images turning black after several iterations. (b) SDS (score distillation refinement): ): SDS seeks a single mode for text-aligned 4D representation, leading to un- successful refinement. (c) Iterative refinement: InstructN2N (Haque et al., 2023) iteratively updates the supervised dataset (cach image is refined for multiple times, different from our pas- refine- ment) for 3D) scene editing. In our task, the diverse outputs from diffusion model result in blurred 4D model under pixel-wise supervision. (d) Textured mesh extraction (Tang et al. 2024b): exper- iments show that meshes extracted from 3D Gaussians are not watertight and smooth (Tang et al.) 2024b, Huang et al., 2024a), leading to incoherent appearance. (e) Video temporal refinement (Ren et al. 2023]: SVD prior alone is insufficient for structure preservation and detail refinement. (f) One-pass refinement (Ours): in this way, the refined (supervised) images strike balance between detail restoration and preservation of structural integrity and consistency. This approach introduces reasonable details in noisy semantically defective regions. Temporal consistency is maintained despite trame-b--frame inference, as input images are similar (two renderings of the same object). The similarity of inputs and predefined text prompt ensure consistent noise directions and results in converging outputs in the image-to-image diffusion process (Haque et al., 2023). Quantitative ablation (Table B) demonstrates that this stage preserves temporal consistency while significantly enhancing per-frame image quality.
Table 3: Ablation study
Mathod
CD-FVD,
wio Stage III 179.3
W. Stage III
142.34
459.10
CONCLUSION AND DISCUSSION
Conclusion. In this paper, we propose F.4DD, novel framework for 4D generation from sin- gle image. This approach departs from previous score-distillttio-naass methodologies, promising not only intrinsic immunity against problems like over-saturation but also capabilities for consistent visual details and dynamics. We first equip the video diffusions with training-free attention injec- tion strategy explicitly generate consistent dynamics and multi-views of the given object. Then coarse-to tonee-tt-iinee fine 4D optimization scheme is introduced to further address practical challenges in syn- thesized videos. Qualitative and quantitative results demonstrate that EG4D produces 4D objects with more realistic and higher-quality appearance and motion compared with the baselines.
Limitations & Future work. One limitation is that our framework can not generate high-dynamic motion due to the limited capability of the base image-to-video model (Blattmann et al.) 2023a) and the consistency-motion trade-off in our attention injection strategy. Another problem lies in the multi-view diffusion model (Voleti et al. 2024). which currently struggles to apply precise camera pose conditioning, leading to unsatisfactory reconstruction. One solution for dynamics is to leverage more advanced video diffusions to generate high high-quality and high-dynamic video frames. Future work could also incorporate adaptive camera pose techniques (Fu et al. 2024 Smith et al. (024) in 4D reconstruction to further improve the robustness.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
We thank all anonymous reviewers and area chairs for their valuable comments. The work described in this paper was substantially supported by GRF grant from the Research Grants Council (RGC) of the Hong Kong Special Administrative Region, China [Project No. CityU 11208123).
REFERENCES
Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2D diffusion into 3D, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04968, 2023.
Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang. Gordon Wetzstein, Leonidas J. Guibas, Luc Van Gool, and Radu Timofte. 3D-Aware video generation. TMLR, 2023.
Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. TC4D: Trajectory-condittone text- to-4D generation. arXiv preprint 2rXiv:2403.17922 2024a.
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4D-fy: Text-to-4D generation using hybrid score distillation sampling. In CVPR, 2024b.
Jason Bailey. The tools of generative art, from flash to neural networks. Art in America, 2020.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align Your Latents: High- HigghRReoluttion Video Synthesis With Latent Diffusion Models. In CVPR, 2023b.
T Brooks, B Peebles, C Homes, W DePue, Y Guo, L Jing, D Schnurr, Taylor, T Luhman, E Luh- man, et al. Video generation models world simulators. OpenA/ Blog, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning follow image editing instructions. In CVPR, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Shengqu Cai, Duygu Ceylan, Matheus Gadelha, Chun-Hao Paul Huang, Tuanfeng Yang Wang, and Gordon Wetzstein. Generative rendering: Controllable 4D-guided video generation with 2D diffusion models. arXiv preprint 132.v:2312.014000 2023.
Ang Cao and Justin Johnson. HexPlane: fast representation for dynamic scenes. In CVPR, 2023.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay. Sameh Khamis, et al. Efficient geometry- 3D generative adversarial networks. CVPR, 2022.
Yiwen Chen, Chi Zhang, Xiaofeng Yang, Zhongang Cai, Gang Yu, Lei Yang, and Guosheng Lin. IT3D: Improved text-to-3D generation with explicit view synthesis. In AAAI, 2024.
11

---

Published as a conference paper at ICLR 2025
François Darmon, Lorenzo Porzi, Samuel Rota-Bulle Bulò, and Peter Kontschieder. Robust gaussian splatting, arXiv preprint arXiv:2404.02211, 2024.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.0566. 2023.
Sara Fridovich-Keil. Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei Efros, and Xiaolong Wang. COLMAP-free
3D gaussian splatting. In CVPR, 2024.
songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In CVPR, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. AnimateDifft Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024a.
Zhiyang Guo, Wengang Zhou, Li, Min Wang, and Houqiang Li. Motion-aware 3D gaussian splatting for efficient dynamic scene reconstruction. arXiv preprint 11Xiv:2403.11447, 2024b.
Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-NeRF2NNRRF: Editing 3D scenes with instructions. In ICCV, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Jonathan HO, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, and David J. Fleet. Video: High definition video generation with diffusion models. arXiv preprint arXiv:22 10.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint 20X4v.220.003458, 2022b.
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character anima- tion. In CVPR, 2024.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2D gaussian splatting for geometrically accurate radiance fields. ACM TOG, 2024a.
Yi-Hua Huang, Yang-Tian -ian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. SC-GS: Sparse-controlled gaussian splatting for editable dynamic scenes. In CVPR, 2024b.
Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. Animate3d: Ani- mating any 3d model with multi-view video diffusion. arXiv preprint arXiv:2407. 1398, 2024a.
Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4D: Consistent 360 dynamic object generation from monocular video. In ICLR, 2024b.
Bernhard Kerbl, Georgios Kopanas, Thomas Leinkuehler, and George Drettakis. 3D gaussian splat- ting for real-time radiance field rendering. ACM TOG, 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2Video-Zeno: Text-to-image diffusion models are zero-shot video generators. In ICCV, 2023.
Ruilong Li. Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. TAVA: Template-free animatable volumetric actors. In ECCV, 2022.
12

---

Published as a conference paper at ICLR 2025
Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Platanio- tis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-tempont consistent 4d generation via video diffusion models. arXiv preprint arXiv:2405.16645 2024.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yi Yu Liu, and Tsung-Yr Yi Lin. Magic3D: High-resolution text-to-3D con- tent creation. In CVPR, 2023.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your Gaussians: Text-to-4D with dynamic 3D gaussians and composed diffusion models. In CVPR, 2024.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306. 16928, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: Zero-shot one image to 3D object. arXiv preprint arXiv:2303. /3328, 2023b.
N. Max. Optical models for direct volume rendering. IEEE TVCG, 1995.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08777. 2021
Zijie Pan, Zeyu Yang, Xiatian Zhu, and Li Zhang. Efficient44: Fast dynamic 3d object generation from single-view video. arXiv preprint arXiv 2401,08742, 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In ICLR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICMI, 2021.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dream- Gaussian4D: Generative 4D gaussian splatting. arXiv preprint arXiv:2312.17142 2023.
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al. L4gm: Large 4d gaussian reconstruction model. arXiv preprint arXiv:2406. 10324, 2024.
Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, and Matthias Nießner. GANeRF: Leveraging discriminators to optimize neural radiance fields. ACM TOG, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022b.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. In ICML, 2023a.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint arXiv:2311.1704 2023b.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3D generation. In ICI.R 2024.
13

---

Published as a conference paper at ICLR 2025
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a- Video: Text-to-vider generation without text-video 2022a.
data. arXiv preprint arXiv:
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without textovideo data. arXiv preprint arXiv:2209. /4792, 2022b.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Videe Text-to-video generation without text-video data. ICLR, 2023a.
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4D dynamic scene generation. arXiv preprint arXiv:2301. 2023b.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. StyleGAN-V: A continuous video generator with the price, image quality and perks of StyleGAN2. CVPR, 2022.
Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. FlowMap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404. 15259, 2024.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang. Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310. 16818, 2023.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054. 2024a.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. DreamGaussian: Generative gaussian splatting for efficient 3D content creation. In ICLR, 2024b.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16. pp. 402-419, Springer, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302. 13971, 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXiv: 18/2.01717, 2018
Cyrus Vachha and Ayaan Haque Instruct-GS2GE Editing 3D gaussian splats with instructions, 2024.
witpuuttt..gggggg.......... 1o/
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from single image using latent video diffusion. arXiv preprint arXiv:2403.12008 2024.
Ziyu Wan, Despoina Paschalidou, Ian Huang, Hongyu Liu, Bokui Shen, Xiaoyu Xiang, Jing Liao, and Leonidas Guibas. CAD: Photorealistic 3D generation via adversarial distillation. In CVPR, 2024.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- DifieDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. NeurIPS, 2023.
14

---

Published as a conference paper at ICLR 2025
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024a.
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. ReconFusion: 3D reconstruction with diffusion priors. In CVPR, 2024b.
Zhongkai Wu, Ziyu Wan, Jing Zhang, Jing Liao, and Dong Xu. RaFE: Generative radiance fields restoration. arXiv preprint 4rXiv:2404.03654, 2024c.
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dy- namic 3d content generation with multi-frame and multi-view consistency. arXiv preprint mrXiv:2407.174700 2024.
Dejia Xu, Hanwen Liang. Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4D: LLM-guided compositional 4D scene generation. arXiv preprint (tti::2403.1999/) 2024.
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene repre- sentation and rendering with 4d gaussian splatting. In ICLR, 2024a.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-tovvidoo diffusion models with expert transformer. arXiv preprint arXiv:2408.0600 2024b.
Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3D gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 13 13101, 2023.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, and Fan Yang. NUWA-XL: Diffusion over Diffusion for eX- tremely Long Video Generation. In ACL, 2023a.
Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4DGen: Grounded 4D content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225. 2023b. 2023b.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409. 02048, 2024.
Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. arXiv preprint arXiv:2403. 14939, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. lc-light github page, 2024.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate Animating image 4D dynamic scene. arXiv preprint 211.::331..14633, 2023.
Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text- and image-guide 4D scene generation. In CVPR, 2024.
Zhizhuo Zhou and Shubham Tulsiani. SparseFusion: Distilling view-conditioned diffusion for 3D reconstruction. In CVPR, 2023.
15

---

Published as a conference paper at ICLR 2025
Appendix
In this appendix, we first present additional related works on 3D refinement (Appendix Aj. Then we provide detailed network specifications (Appendix B...ext. To ensure reproducibility and facilitate fair perceptual studies, we describe the experimental settings in detail (Appendix C Finally, we
include extended ablation studies (Appendix D) and additional results (Appendix E demonstrate the robustness and superiority of our methods across various settings.
A MORE RELATED WORKS
3D refinement with generative priors To deal with view-inconsistenge and low quality prob-
lems, many works (Wu et al. 2024c Roessle et al. 2023 Chen et al,, 2024 Wu et al.] 2024b]
Haque et al. 2023: Vachha & Haque 2024/Zhor & Tulsiani 2023) take advantages from gen-
erative priors, e.g. adversarial training Goodfellow et al., 2014) and score distillation sampling
(SDS) (Poole al. 2023) optimize the 3D representation. GANeRF (Roessle et al. (2023) refines
the rendered images with an image-conditional generator and leverages the re-rendered image con- straints guide the NeRF optimization in the adversarial formulation. InstructNcRFFNNRFF (Haque) et al., 2023) uses the text-conditioned image generator, InstructPix2pix (Brooks et al., 2023). to edit the image rendered by pre-trained NeRF in an iterative manner and updates the underlying 3D representation with the edited images. ReconFusion (Wu et al. 2024b) uses the diffusion priors, Zero-123 (Liu al. 2023b), as a drop-in regularizer to enhance the 3D reconstruction performance, especially for sparse-view scenarios. In contrast to directly optimizing the implicit representation, another line researches (Tang et al. 2024b Ren et al. (023) first extracts the explicit textured mesh, and then refine the texture in U V--spac with diffusion prior and differentiable rendering. In particular, DreamGaussian4D leverages SVD as image-to-ve tiveeoo prior to enhance the texture tem- poral consistency. In paper, in consideration of the artifacts generated in video diffusion, extend the refinement techniques to the 4D representation.
B NETWORK DETAILS
In this section, we unpack the core network design in Figure
Attention injection. In Sec. 4.1 we exploit the attention injection strategy to alleviate the tem- poral difference between multi-view diffusion models. Figure illustrates its network details: in each spatial attention layer, we replace the self-attention by simultaneously considering the current zt and previous visual information with EMA.
Deformation field with color transformation. In Sec. 4.2 use color affine transformation to model the temporal texture variation. Figure [00 shows the detailed architecture of that. We first query the time-specific feature f. from the learnable HexPlane (Cao & Johnson 2023) with the canonical Gaussian positions After that, the geometric deformations of Gaussian properties (μ location, r rotation, and 8 scale) are predicted with lightweight decoder. Additionally, we use the affine color transformation to model the temporal texture variations. Finally, these deformed Gaussians are rendered into an image.
C ADDITIONAL EXPERIMENTAL SETTINGS
C.I OPTIMIZATION DETAILS
We report the optimization of 4D Gaussian splatting for the purpose of reproduction. Basically, we follow the training recipe from 4DGS (Wu et al. 2024a) in the coarse 4D reconstruction stage. In the semantic refinement stage (Stage III), we fine-tune 4DGS for 5k steps with Adam optimizer. The initial learning rate is set to le-4 with exponential decay. The weight À in diffusion refinement loss is set to 0.5. Our implementation is primarily based on the PyTorch framework and tested on a single NVIDIA RTX 3090 GPU.
16

---

Published as a conference paper at ICLR 2025
Condition
Linear Projection
Cross Cross-Atterinn Ces--Atttetoon
Convolution
Linear
FFN
latent
K
Projection
Lincar
Projection
SV3D UNet
Spatial Attention
Temporal Attention
Figure 9: Network details of Attention Injection. denotes the EMA blending operator men- tioned in Sec. 4.1 2x is the multi-view latent current timestamp 1, and 2; is the blended latent. Previous visual information is injected into the current latent by modifying the original spatial self- attention mechanism.
HexPlane
J
F
Deformation Decoder
Gaussian Renderer
Affine Transformation
Ce
Canonical
Gaussians
Deformation Field
Figure 10: Network architecture of our 4D representation. t1, F, represents the canonical Gaus- sian properties: 3D location, rotation and scale from the coarse stage training in 4DGS (Wu al. 2024a). The time-specific local feature ft is queried from the HexPlane (Cao & Johnson [023). where the subscribe t means the time-specific property. Different from vanilla 4DGS, we employ additional color affine transformation to obtain the time-specific color ct. The geometric deforma- tions are predicted by a lightweight decoder. Finally, the time-specific Gaussians are rendered to produce an image (right).
C.2 REPRODUCTION, DATA AND CODE
We reproduced our baselines, including Animate 124 İZhao et al. 2023). DreamGaussan44D (Ren cet al. 2023). and Consistent4D [Jiang et al.) 2024b). EfficientDD (Pan et al.) (024), using their official code. Additionally, have included the imput images and LVD-generates, videos in the supplementary materials. Apart from the data provided by Animate 124 and DreamGaussian44) we have added three more examples: android, chicken-basketbal. and penguin. Code is also available in the supplementary materials.
C.3 USER STUDY DETAILS
We provide details of the user preference study with two screenshots. Figure illustrates the guide- lines: each participant is asked to evaluate images and videos rendered by four different methods across five metries. Figure 22 shows the image and video samples presented to the participants. After comparing the images (Figure (2)a)) rendered by different models, participants select the method with the highest "reference image consistency" and "3D appearance". After watching the videos (Figure b)) rendered by different models, participants select the method with the highest "motion realism" and "motion range". Finally, they choose the method with the best overall quality. We presented several cases to 47 participants and compiled the statistics. For statistical significance, we make the assumption of multinomial distribution, and report the 2-sigma error bar (95.6% CI). We standard deviation for enror bar calculation.
17

---

Published as a conference paper at ICLR 2025
Method
Independent
S-Res
S-Linear
T-EMA
S-EMA (Ours)
CLIP-I f (Radford al. 2021
0,9323
0.9136
0.9654
0.9962
0.9925
Flow IntensiltT leed Deng 2020
2.912
1.102
2.756
Table 4: Quantitative ablation on attention injection. We evaluate temporal consistency using CLIP-I score between the first and subsequent frames (1 higher is better), and motion range using optical Flow Intensity ( larger indicates larger motion range when CLIP scores comparable). means no reasonable results predicted by the optical flow estimator on video with noisy background. Both "T-EMA' and 'S-EMA' improve temporal consistency, but while "T-EMA' results in nearly static output, 'S-EMA' maintains substantial motion range. Qualitative results are shown in Figure
Time 0
Time
Reference
w/o injection (a
+S-EMA (ar 0.1)
S--EMA (ar 0.3)
+-EMMA (ar 0.5)
+-EMA (a 0.9)
(a 0.7)
Figure 11: Qualitative sensitivity analysis EMA blending weight of attention injection. As the blending weight increases, the temporal consistency is significantly improved (similar white textures and consistent leg geometry). However, the overly high (> 0.5) blending weight leads to very small motion range. To balance the motion range and temporal consistency, we choose the EMA weight as 0.5. Video demonstration can be found in our supplementary materials.
D EXTENDED ABLATIONS
Quantitative ablation on attention injection. We conduct comprehensive experiments to evalu- ate different variants of attention injection. Please refer Figure in the main paper for qualitative results. Quantitative analysis is summarized in Table demonstrating the impact of different atten- tion injection variants on video temporal consistency and motion range. To quantify temporal con- sistency, we compute the CLIP-I score between the first frame and subsequent frames. Our results indicate that both 'T-EMA' and 'S-EMA' significantly improve temporal consistency (inter-frama similarity), achieving higher CLIP-I scores compared to other variants. For motion range assess- ment, we employ the flow intensity, calculated by average value of optical flow on adjacent frames. When CLIP scores comparable, larger flow intensity indicates larger range of motion. The op- tical flow estimated with RAFT (Teed & Deng 2020). means no reasonable results predicted by the optical flow estimator on video with noisy background. Notably, 'T-EMA' yields DINO score approaching 1, suggesting minimal object movement. Among all variants examined, our pro- posed 'S-EMA' uniquely achieves an optimal balance, maintaining high temporal consistency while preserving substantial motion range.
Sensitivity analysis for attention injection weight. Figure analyzes different EMA blending weights of attention injection in the spatial attention layers. it obvious that the increasing blend- ing weight benefits the temporal consistency in texture, e.g., similar white texture the back and consistent leg geometry. We also observe that overly high (> 0.5) blending weight significantly attenuates the object motion range. This trade-off can be better illustrated by the videos provided in the supplementary materials. Taking both motion range and temporal consistency into consid- eration, we choose = 0.5 as an appropriate blending weight without sacrificing the dynamics. Figure 12 demonstrates the impact of different blending weights. As the weight increases. CLIP-I score (image quality) improves while motion range becomes smaller, indicated by decreasing flow intensity. CD-FVD will not be better due the diminishing motion.
Number of Gaussians. In Figure 13 the multiscale renderer. Guo et al. (Guo et al.) 2024b) observed that visual overfitting often leads to redundant Gaussian splats in dynamic scene reconstruction, which is hard optimize and causes
show the number of Gaussians before and after adding
18

---

Published as a conference paper at ICLR 2025
2x
CD-FVD
Row intersity
GUP-1
0.90
60
0.88
0.0
0.2
0.4
0.5
0.8 1.0
blending weight
Figure 12:
Quantitative sensitivity analysis on EMA blending weight of attention injection. Higher blending weights improve temporal consistency but excessive values restrict motion range indicated by lower flow intensity. We select a=0.3, achieving well balance between motion range and temporal consistency, with the best CD-FVD score.
Number of Gaussians
7.5e+4
6.5e+4
5.5e+4
baseline
multiscale
4.5e+4
5k
10k
15k
20k
25k
30k
step
Figure 13:
Additional ablation on the multiscale renderer. With the multiscale rendering aug- mentation in Stage II (darkred), the number of Gaussians declines significantly.
unsatisfying rendering results. With the multiscale renderer, we observe significant decline of Gaussian points, in addition to the dropped training PSNR reported in Figure 7
Additional results for diffusion refinements. In Figure 14 the effectiveness of diffusion refinement is illustrated with zoomed-in details. It can be observed that the facial and hand details become finer and Gaussian noises are removed after the refinement stage.
E EXTENDED RESULTS
Dynamics our results. For the best demonstration our 4D model dynamics, please refer to the supplementary materials where you can find videos generated by our 4D model. Figure 15 has illustrated more examples beyond SVD-generated videos, and show the scalability and generaliz liz- ability of our framework. The panel (a) uses video rendered from Objaverse (Deitke et al. 2023)
dataset, a large-scale 3D dataset that also contains some animation models. Figure 15 (b) shows
the 4D generation results from in-the-wild videos from the Consistent4D benchmark: In panel (c),
we leverage the pose-conditiones character video generation model, AnimatcAnyone (Hu 2024). as video model in framework.
19

---

Published as a conference paper at ICLR 2025
View 1
View 2
Coarse 3D
+ Diffusion Prior
Coarse 3D
Diffusion Prior
Figure 14: Ablation on diffusion refinement. The left and right panels depict two different view of renderings with the case anya. The results after adding the diffusion refinement show finer facial and hand details with less noisy Gaussians.
Input
View
View
View
1ej Ranuls wwuusbaasdeeee on Objirarn data
(0) Resuht baser on Consistentlo i-the-wild cats
(c) leeulls besed
Figure 15: Extended results rendered by EG4D based semantic/large motion of synthetic/real- world objects. The input data includes (a) single-view rendering from Objaverse (Deitke et al. (2023) objects, (b) in-the-wild videos from Consistent4D (Jiang et 2024b). and (c) character motions generated by cose-conditioned video diffusion, AnynateAnyone (Hu 2024).
Qualitative comparison with Efficient4D. We compare our results with another baseline Effi- cient4D (Pan et al. 2024), which uses 4DGS (Yng et al. 2024a) reconstruction backbone. Con- sistent with quantitative results in main paper Figure the 4D results generated by our method have higher view consistency and temporal consistency.
20

---

Published as a conference paper at ICLR 2025
Input Image
Efficint4D
Input Image
Efficienc4D
Ours
Input Image
EfficientdD
Ours
Figure 16: Qualitative comparison with another baseline, Efficient4D
(Pan et al.
2024
Method
CLIP-I
PSNR
SSIM t
LPIPS 1
FVD.
CD-FVDJ
SV4D Xie et al. 2024
0.9459
22.57
0.852
0.196
138.81
31
EG4D (Ours)
0.9535
23.28
0.904
0.173
142.34
459.10
Table 5: Quantitative comparison with a training-based method, SV4D Xie et al. 2024).
Comparison with training-based methods. Recent works (Liang et al. 2024 Jiang et al. 2024a Xie et al.) (2224) have advanced multi-view video diffusion through training on large-scale datasets, demonstrating significant improvements in 4D generation quality. Notably, Ani- mate3D (Jiang et al., 2024a) extends AnimateDiff (Guo et al. 2024a) to generate spatiotemporally consistent multi-view videos of static 3D objects. We compare our method with SV4D (Xie et al.) 2024]. as shown in Figure [7 and Table While SV4D achieves better temporal consistency, approach exhibits superior image fidelity and view-consistency. This is evident in examples like luigi and zelda, where SV4D produces overly bright faces lacking detail and shading. This suggests that while SV4D performs well on its training set, it may have limited generalization capa- bility out-of-distribttion (O.O.D)) samples.
More discussion about training-based methods. Our framework offers two key advantages over training- based methods like Diffusion4D and SV4D: First, our approach training-free. leverag- ing off-the-shelf video and multi-view diffusion models without modifications. This allows rapid adoption of advances in either model type to generate 4D content efficiently, eliminating the need for expensive training large-scale 4D datasets. Second, our method maintains dataset indepen- dence and directly benefits from improvements video diffusion. Regarding motion of 4D object, advanced video diffusion models like XogVideoX (Yang al. 2024b) would enable more dynamic and diverse animations. For 3D content, multi-view diffusion for 3D scene, e.g. ViewCrafter Yu al. 2024), would provide possibility for 4D scene generation.
21

---

Published as a conference paper at ICLR 2025
Input Image
SYAD
OuT
Input Image
SV4D
Figure 17: Comparison with Training-base method, SV4D (Xie al., 2024). Benefited from the dataset-independency. we achieve higher view-consistency compared to SV4D in O.O.D data.
4D Generation with complex image prompt. Apart from the images/videos from Animate and Consistent4D benchmark, we experiment our methods in more complex image prompt from DreamCraft3D (Sun et al. 2023). For images with extreme lighting (Figure 18 first row) and com- plex image prompts (Figure 18 row 2-4), we find that our method can produce results with high view-consistency and image fidelity. However, we find hard to achieve stable results for some complex images. The main difficulties come from image- based video generation models: SVD or even the state-of-the-as produce videos with well-preserved identity and smooth motion. Generated videos, corresponded demo videos, and failure generated videos are all included the supplementary material.
video generation, CogVideoX-IBs
(Yang et al.) (024b),, can not always
Efficiency. Our framework takes approximately hour and 15 minutes on average for each 4D object generation in a single NVIDIA RTX 3090. Specifically, Stage I requires about 20 minutes for video and multi-view generation; Stage II, involving 4D Gaussian Splatting optimization, takes around 25 minutes: and the refinement process takes about 30 minutes. In previous works, Con- sistent4D (Tiang et al., 2024b) and Animate 124 (Zhao et al. 2023) take about 2.5 and hours, respectively, for 4D generation. Notably, DreamGaussian-4D (Ren et al. 2023) achieves extremely short optimization time of minutes. Diffusion4D (Liang et al., (224)) and SV4D (Xie et al. 20224) train diffusion network to generate the multi-viey view multi-frame image matrix. We have comparable inference time since both methods share similar pipeline of multi-vien view video generation and 4D rep- resentation optimization. However, they need to take huge computational expense for training. In contrast to optimization-bant
approaches, L4GM (Ren et al., 2024) uses feed-forward network to direct predict the Gaussian sequences within several minutes. Our optimization time falls between these, but our framework offers superior view consistency, 3D appearance, and motion quality. Since Stage appears be one of the efficiency bottlenecks, future work should focus on incorporating efficient sampling for video diffusion models to boost speed.
Multi-view results of our results. Figure shows the multi-view results of 4D model. which is supplement of Figure 4 Due to the page limit of the main paper, we only show two views of the 4D model there, which is not enough to illustrate the 3D appearance of our model. To this end, render model in more views: 0 90" 1350, 180", 225", and 270", The rendered multi-view images show that our method can produce images with high 3D consistency and satisfactory quality.
More visual comparisons. Figure 24 provides additional visual comparisons with our baselines, continuing from Figure in the main paper. We use three additional cases: luigi, anya, and chí siten-bbaseetba The first two columns show animation results from the same view,
22

---

Published as a conference paper at ICLR 2025
Input Image
4D Rendering
Figure 18: 4D generation results with complex image prompts. The first row is luigi relighted by LC-hight (Zhang et al. 2024). and 2-4 rows are image prompts from DreamCraft3D (Sun et al. 2023). Since SVD (Blattmann al. (2023a) fails to generate valid video for bear-arma case, we have opted to utilize Vidcoo- -IV 1- (Yang et al. 2024b) as an alternative.
Text Prompt:
"ninja, white background, standing, toy, cartoon, 3D model, high quality"
View
View
Generated Image
View
View 4
Figure 19: Text-to-4D results. We feed a text prompt (left top) into SDXL Podell et al. (2024) to generate a ninja image (left bottom). This image can be transformed into 4D objects with our framework, presenting indirect text-to-4D application. The right panel shows multi-view renderings of the 4D model.
while column 3 to 5 display three different views. The last column presents a zoomed-in image of the final rendered view. Multi-view videos for visual comparison can be found in the supplementary.
More applications. Benefiting from our explicit generation, we can easily adapt EG4D to both text-to-4D and video-to-4D tasks. Figure shows the generation results the text-to-4D. We first feed example text prompt into SDXL (Podell et al., (2024) to get the high-resolution image. Then this image is transformed into a 4D model with our framework. Figure 20 shows the results of the video-to-4D. We just skip the dynamic generation step and start with our view synthesis pipeline.
23

---

Published as a conference paper at ICLR 2025
View 1
View
Input Video
View 3
View 4
Figure 20: Video-to-4D results. Our framework can be seamlessly extended to video-to-4D gen- eration. The right panel shows the renderings our 4D model from four viewpoints. This bird video is taken from Consistent4D (Jiang et al. 2024b).
First all, thank you all participating!
Our is generate 4D model from given image, and then render at arbitrary view/time.
Please compare the generation results produced by different methods and answer the following questions.
First, please compare the images produced by four methods and select one method that you think provide the best results.
Which method's results have better consistency with the given image?
Focus on consistency instead of quality
Which method's results have the best 3D appearance?
Focus esthetics and viewcoonisteency
Then, please compare the videos produced by those methods.
Which method produces the most natural motion?
Which method produces the largest range of motion?
Finally,
Please select the method that shows the best overall quality!
Figure 21: Screenshot our user study guidelines. Each participant is asked evaluate the images and videos rendered by 4 different methods with 5 metrics, i.e., reference view consistency, 3D appearance, motion realism, motion range, and overall quality.
Case (Image)
(Video)
Mechod 1:
Mothod 1:
Irpu: Image
Inst
Method 2:
π
Method 2:
Mathod 3:
Method 3:
Mathod 4:
Method
(a) Screenshot of the image evaluation.
(b) Screenshot of the video evaluation.
Figure 22: Screenshot of our user study content. Each participant is provided with several images and videos rendered by different methods.
24

---

Published as a conference paper at ICLR 2025
Input Image
View
Figure 23: Multi-view results of our models. This figure is supplement of Figure 4 in the main
paper. The 6 columns show the images rendered by our model in different views: 0" 90" 135 180", 225", and 270". Multi-view renderings demonstrate the geometry/texture consistency and promising quality of our 4D representation.
25

---

Published as a conference paper at ICLR 2025
Animate124
Input Image
Consistent4D
DG4D
Ours
Consistent4D Animate124
Input Image
DG4D
Ours
Animate124
Input Image
ConsistentAD
DG4D
Ours
Figure 24: More comparison examples with the SOTA results in three cases luig anya and chicken bhicken-baskethan baskeetball (better zoom in).
26


Reviews:
[{"review_id": "0V8Pobf7Zt", "reviewer": "Reviewer_gHhE", "summary": "The paper introduces EG4D, a multi-stage framework designed for generating 4D objects from a single image input without relying on score distillation sampling (SDS). EG4D employs three key stages: view generation using video diffusion models, coarse 4D reconstruction using Gaussian Splatting, and refinement with diffusion priors. The framework addresses common issues like temporal inconsistencies, over-saturation, and the Janus problem, achieving superior performance over existing 4D generation models. The authors demonstrate the effectiveness of EG4D through both qualitative and quantitative analyses, showing significant improvements in 4D object realism and consistency.", "questions": "1. Could the authors provide more quantitative insights into the effects of the attention injection mechanism on temporal consistency?\n2. How sensitive is the model’s performance to variations in the blending weight (α) during attention injection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. EG4D introduces a unique multi-stage approach that successfully avoids issues associated with score distillation, such as over-saturation and Janus artifacts.\n2. The attention injection mechanism helps ensure temporal consistency.\n3. Quantitative and qualitative results demonstrate that EG4D produces high-quality 4D content, achieving superior alignment with the reference view and more realistic motion realism compared to baselines like DreamGaussian4D and Animate124.\n4. The evaluation is comprehensive. The evaluation metrics, including CLIP-I, PSNR, SSIM, FVD, and CD-FVD, along with a user study, offer robust evidence of the framework's advantages.", "weaknesses": "1. Lack of baseline methods. Recent approaches such as L4GM and efficient4D ( which also leverage 4D Gaussian Splatting) should definitely be included for comparison.\n2. Lack of Comparative Analysis. The attention injection technique is novel, but without comparisons to other methods, its effectiveness remains uncertain.\n3. Texture Inconsistencies. Despite color transformation, subtle color and texture shifts remain, questioning the robustness of temporal consistency.\n4. Limited Support for High-Dynamic Motion. The model struggles with high-dynamic motion, restricting its applicability in scenarios needing greater flexibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lTt1dN0Ax8", "reviewer": "Reviewer_pq5e", "summary": "The paper proposes a 4D generation framework that avoids the need of score distillation. Instead, multi-view videos are generated from one input image and a 4D representation is optimized directly.", "questions": "- For the main comparison results. Did the authors use the same video to compare the proposed method and Dreamgaussian4D? It seems that the generated object motions are very different. If not, is it possible to additionally compare a variant of DreamGausisan4D? In other words, can we use the foreground video rendered by the proposed method and reconstruct to 4D with the framework of DreamGausisan4D?\n\n- For the Mario case, as shown in the supplementary video, contains flickering artifacts on the face regions. Why is this happening? Is it because the refinement stage suffers from the Janus issue?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- A multiscale augmentation is proposed for rendering more details.\n- A training-free attention injection strategy is introduced to ensure consistency.\n- A Diffusion Refinement stage is included to refine the details.", "weaknesses": "- In Fig. 1, it seems that the multiscale renderer is not used for the \"Diffusion Refinement\" stage. What is the motivation for this change? Is it necessary?\n- What is the difference of the proposed Diffusion Refinement stage and that in DreamGaussian4D?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NUmacVF0Eg", "reviewer": "Reviewer_5QU4", "summary": "The paper proposes an explicit 4D object generalization method without using SDS loss. The method leverages SVD and SV3D to optimize the 4D representation from the generated videos. Experimental results demonstrate superior performance over previous baselines.", "questions": "Another way is to first generate 3D models and then animate the mesh [4]. I wonder if this line of methods are more effective and flexible.  Could you discuss the potential advantages and limitations of their approach compared to the 3D-then-animate pipeline. Specifically, the authors are  suggested they address aspects like computational efficiency, animation quality, and flexibility in handling different types of objects or motions.\n\n[4] Jiang, Yanqin, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. \"Animate3d: Animating any 3d model with multi-view video diffusion.\" arXiv preprint arXiv:2407.11398 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed pipeline is clearly motivated and reasonable.\n2.\tThe writing is clear and easy to follow, with a well-structured introduction and related work section\n3.\tExtensive experiments shows clear performance improvements over previous methods (Table 1, Table 2).\n4.\tThe authors provide a comprehensive ablation study, which helps justify the effectiveness of attention injection, color transformation, multi-scale renderer, refinement strategies.\n5.\tThe supplementary materials are sufficient.", "weaknesses": "1.\tAs shown in the video, the results are flickering and blurry. It seems that the quality is not as good as Diffusion4D [1] (see the examples of the project page of Diffusion4D).  It would be valuable to add a discussion and analysis about the proposed method with Diffusion4D (e.g., generation time, temporal consistency, image fidelity, 3D consistency).\n\n2.\tThe method is built on SVD and SV3D for 4D generation. It would strengthen the paper if the authors could discuss the different designs between SV4D [2] and proposed method. The authors are suggested to provide both qualitative and quantitative comparisons. In specific, the authors can quantitative comparison table between their method and SV4D on metrics like overall quality (view alignment, 3D consistency, multi-view consistency,  temporal consistency), animation flexibility (e.g. motion range, motion fidelity), and computational efficiency (e.g. FPS, generation time). \n\n3.\tI noticed the examples are some simple images. I wonder if the proposed can handle more complex images like examples in https://mrtornado24.github.io/DreamCraft3D/ and other cases (with multiple objects, complex backgrounds, or challenging lighting conditions).\n\n[1] Liang, Hanwen, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N. Plataniotis, Yao Zhao, and Yunchao Wei. \"Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models.\" arXiv preprint arXiv:2405.16645 (2024).\n[2] Xie, Yiming, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. \"Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency.\" arXiv preprint arXiv:2407.17470 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)