Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video
Authors: Yanqin Jiang
Li Zhang
Jin Gao
Weiming Hu
Yao Yao
Venue: ICLR 2024 poster (2024)
Keywords: Dynamic object generation
Primary area: generative models

Abstract: In this paper, we present Consistent4D, a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. Uniquely, we cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration. This is achieved by leveraging the object-level 3D-aware image diffusion model as the primary supervision signal for training dynamic Neural Radiance Fields (DyNeRF). Specifically, we propose a cascade DyNeRF to facilitate stable convergence and temporal continuity under the time-discrete supervision signal. To achieve spatial and temporal consistency of the 4D generation, an interpolation-driven consistency loss is further introduced, which aligns the rendered frames with the interpolated frames from a pre-trained video interpolation model. Extensive experiments show that the proposed Consistent4D significantly outperforms previous 4D reconstruction approaches as well as per-frame 3D generation approaches, opening up new possibilities for 4D dynamic object generation from a single-view uncalibrated video. Project page: https://consistent4d.github.io

Published as a conference paper at ICLR 2024
CONSISTENT4D: CONSISTENT 360 DYNAMIC OB- JECT GENERATION FROM MONOCULAR VIDEO
Yanqin Jiang" , Li Zhang", Jiin Gao! Weiming Hu 1.2.6 Yao Yao4.5
State Key Laboratory Multimodal Artificial Intelligence Systems (MAIS), CASIA
School of Artificial Intelligence, University of Chinese Academy of Sciences
"School of Data Science, Fudan University
"State Key Laboratory for Novel Software Technology, Nanjing University
i "School of Intelligence Science and Technology. Nanjing University
"chooo of Information Science and Technology. ShanghaiTech University ji angganqiin2221@1e ac. on, lizhangfde fudan edu, en, [jin. gicc wmhu cn, yaoyaoln .duudu. edd.cn
ABSTRACT
In this paper, present Consistent4D, novel approach for generating 4D dy- namic objects from uncalibrated monocular videos. Uniquely, we cast the 360- degree dynamic object reconstruction
4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration. This is achieved by leveraging the object-level 3D-aware image diffusion model su- pervision signal for training dynamic Neural Radiance Fields (DyNeRF). Specif- ically, we propose cascade DyNeRF to facilitate stable training convergence and temporal continuity given the time-discrete supervision signal. To achieve spatial and temporal consistency of the 4D generation, an interpolation-drive consistency loss is further introduced, which aligns the rendered frames with the interpolated frames from a pre-trained video interpolation model. Extensive ex- periments show that the proposed Consistent4D significantly outperforms previ- ous 4D reconstruction approaches as well as per-frame 3D generation approaches, opening up new possibilities for 4D dynamic object generation from single-view uncalibrated video. Project page: https://consistemndd.giii githuu.......ggg...i......o
Figure 1: Video-to-4D generation results of two input videos. We show the renderings at 2 view- points and 3 timestamps demonstrate the spatial-temporal consistency achieved by Consistent4D.
INTRODUCTION
Dynamic 3D content creation, is critical for variety of downstream applications, including virtual reality, augmented reality, autonomous driving simulation, and gaming. Existing 4D reconstruction methods typically rely synchronized multi-view videos (Li et al., 2022: Fridovich-Keil et al., 2023; Shao et al., 2023) or calibrated monocular videos (Li et al., 2021: Pumarola et al., 2021; Park et al., 2021b;a). These approaches, although effective, necessitate complex camera setups and rigorous camera calibrations, constraining their usability and scalability in real-workl settings. They also struggle with reconstructing unobserved regions.

---

Published as a conference paper at ICLR 2024
To adresss the aforementioned limitations, we instead generate 360 dynamic 3D object from single-view uncalibrated video captured by a static camera, streamlining data acquisition while maintaining high fidelity outputs. To tackle the challenging problem of 4D scene recovery from a single video, we harness visual priors from pre-trained large vision models (LVMss, casting 41) reconstruction generative process. This is inspired by human cognitive abilities, which intu- itively interpret 3D shape, appearance, and motion from a video clip using knowledge accumulated through life.
Inspired by recent advancements in 3D generation techniques (Poole et al., 2023; Wang et al., 2022; Chen al., 2023; Wang al., 2023; Deng al., 2022; Tang et al., 2023; Melas Deraa-Kyriazz et al., 2023), we introduce multi-view diffusion model (Liu et al., 2023) to provide spatially consistent supervision signal to the generation of dynamic 3D object. However, this supervision signal is time- discrete, posing challenges for maintaining temporal coherence in dynamic 3D object generation and consequently compromising the spatiotemporal consistency in overall 4D optimization.
In this work, present Consistent4D, a novel video-to-4D generation approach with focus spa- tial and temporal consistency. Our approach features a Cascade DyNeRF tailored for 4D scene rep- resentation in generation tasks, which facilitates temporal coherence under time-discrete supervison singal from a pre-trained multi-view diffusion model. To further improve both spatial and tempo- ral consistency, Interpolation-drivee Consistency Loss (ICL) is introduced, which minimizes the discrepancy between frames rendered by DyNeRF and frames interpolated by a pre-trained video interpolation model. The Ioss not only enhances consistency in 4D generation but also mitigates multi-face issues in 3D generation. Finally, an optional video enhancer refines occasional noise in 4D renderings.
We have extensively evaluated our approach on both synthetic videos rendered from animated 3D models and in-the-wild videos collected from the Internet. To summarize, contributions of this work include:
We propose video-to-4D framework for dynamic object generation from statically cap- tured monocular video. A Cascade DyNeRF, inherently inclined to temporally consistent outputs, is optimized by multi-view image diffusion model, with the option to refine its renderings using a video enhancer.
We introduce a novel Interpolation driven consistency loss improve the spatial and tem- poral consistency of the 4D generation, which also alleviates multi-face Janus problem in 3D generation.
We extensively evaluate our method on both synthetic and in-the-wild videos collected from the Internet, showing promising results for the new task of video-to-4D generation.
RELATED WORK
Recently, general- purepoe- 3D generation has been enabled by eext-to-image dif- fusion models pre-trained on Internet-scale data. DreamFusion (Poole et al., 2023) stands for the pioneering work of text-to-3D by using a 2D diffusion model, where the Score Distillation Sampling (SDS) loss is proposed to leverage the denoise diffusion process for Neural Radiance Field (NeRF) training. The follow-up works (Lin et al., 2023; Chen et al., 2023; Wang et al., 2023) further en- hance the visual quality of the generated object by using mesh representation, Variational Score DistillationVVSD), etc.
3D Generation
Alongside text-to-3D, image-to-3D is a popular alternative for 3D generation. Different from 3D reconstruction, which focuses on recovering 3D information from overlapped multi-viww images, image-to-3D 3D generation usually takes only one single image as input and relies on data priors (c.g., 2D diffusion) to generate invisible regions of the object (Melas-Kyriazi 2023; Liu et al., 2023). Most approaches in this domain simply convert input image to texts by using a large vision-language models and then reused the text-to-3D model for image-to-3D gener- ation Melas-Kyriazi et al., 2023; Tang al., 2023; Seo et al., 2023). One of the biggest challenges in 3D generation is multi-face Janus problem, arising from the lack of 3D awareness in the original 2D diffusion model. However, the recent image-to-31 work Zero (Liu et al., 2023) mitigates this issue by fine-tuning a 3D-aware image-to-imant model on a large-scale multi-view dataset, en- abling the novel view synthesis of the object from the input image. Our work employs Zero123
al., 2023; Tang et al.,
2

---

Published as a conference paper at ICLR 2024
and we propose an Interpolation-driven consistency Loss to further enhance spatial and temporal consistency.
4D Reconstruction Early works in 4D reconstruction, aka dynamic 3D reconstruction, are mainly object-level and adopt parametric shape models (Matthew Loper & Black, 2015; Vo et al., 2020) as representation. In recent years, Dynamic Neural Radiance Field(DyNcRF) become popular, and convenient dynamic scene reconstruction is enabled. These works can be classified into two cate- gories: a deformed scene is directly modeled as NeRF in canonical space with a time-dependent deformation (Pumarola et al., 2021; Park et al., 2021a;b; Wu et al., 2022b; Tretschk et al., 2021) or time-varying NeRF in the world space (Gao al., 2021; Li et al., 2021; Xian al., 2021; Fridovich- Keil et al., 2023; Cao & Johnson, 2023). These methods usually require multi-view, synchronized, and calibrated videos as input to the reconstruction algorithms, which tedious to collect Li et al. (2022); Shao et al. (2023). In other scenarios, monocular video is accepted as imput, but still requires accurate per-frame calibration and a moving trajectory for effective dynamic information recovery (Pumarola et al., 2021; Park et al., 2021a;b). In contrast, our approach utilizes single- view, uncalibrated video as input, significantly streamlining the data collection.
4D Generation 4D generation extends 3D generation the space+time domain and thus is more challenging. Early attempts focus on category-specific generation and parametric shape models is employed scene representation (Zuffi et al., 2017; 2018: Vo et al., 2020; Kocabas al.. 2020) They usually take images or videos as conditions and need category-specifi 3D templates or per-category training from collection of images or videos (Ren et al., 2021; Wu al., 2021; Yang et al., 2022; Wu et al., 2022a). Recently, MAV3D (Singer et al., 2023) has achieved general- purpose dynamic scene generation based on textual descriptions. follows the paradigm of Dream- Fusion (Poole et al., 2023) and extends it to the time domain by proposing three-stage training strategy. However, the quality generated scenes is limited due to low-quality video diffusion models. Instead, we propose the task of video-to-4D generation, leveraging the high-quality input video as a guiding constraint to enhance fidelity and ensure consistent rendering in the 4D output.
3 PRELIMINARIES
3.1 SCORE DISTILLATION SAMPLING FOR IMAGE-TO-3D
Score Distillation Sampling (SDS) is first proposed in DreamFusion (Poole et al., 2023) for text-to- 3D task. It enables the use of a 2D text-to-image diffusion model as a prior for optimization of a NeRF. We denote the NeRF parameters as 0, text-to-image diffusion model é, text prompt p, the rendering image and the noisy image as x and z, the SDS loss is defined as:
VoLSDS (6,x) = Er,e (w()) (Eo (Z.ip,T) e)
ix 20
(I)
where is timestamps in diffusion process, denotes noise, and w is weighted function. Intuitively, this loss perturbs x with a random amount of noise corresponding to the timestep т, and estimates an update direction that follows the score function of the diffusion model to move to a higher density region.
Besides text-to-3D, SDS is also widely used in image-to-3D tasks. Zero123 (Liu et al., 2023) is one prominent representative. It proposes a viewpoint-conditinn conditioned image-to-ine to-image translation diffusion model fine-tuned from Stable Diffusion (Rombach al., 2022), and exploits this 3D-aware image diffusion model to optimize a NeRF using SDS loss. This image diffusion model takes one image, denoted by I and relative camera extrinsic between target view and input view, denoted by (R, T),, as the input, and outputs the target view image Tout- Compared with the original text-to-image diffusion model, text prompt in Equation 1 is not required in this model cause the CLIP embedding of the input image and the relative viewpoint change replace the text prompt. Then Equation I could be re- re-written as:
dx
VoLSDS (6, =R,,e (t) (eg (zg; Inn. R, T, -
(2)
30
3

---

Published as a conference paper at ICLR 2024
Traitable
Image loss
Input video
Froez
Dirission Model
SDS loss
Booloward
Video Dahanoer
Temporal consistency
video
Model
Intorplation Module
Rendering
Fake
Disctmitaker
Spatial consistency
(a) Cascade DyNeRF
(b) SDS and Interpolation Consistency Optimization
(c) Post Video Enhancer
Figure 2: Schematic illustration of Consistent4D.. We introduce Cascade DyNeRF as our 4D repre- sentation, specially tailored to favor temporal continuity, ensuring stable training under time-discrete SDS from multi-view diffusion model. To compensate for temporal inconsistency arising from SDS, we propose novel Interpolation Consistency Loss, leveraging pre-trained video interpola- tion model, to enhance both spatial and temporal consistency 4D outputs. Additionally, we offer an optional video enhancer post-processe prosesrocessing step to refine any potential noise in 4D renderings.
3.2 K-PLANES
K-planes (Fridovich-Keiil et al., 2023) is simple and effective dynamic NeRF method which fac- torizes dynamic 3D volume into six feature planes (i.e., hex-plane), denoted as P (P.), where E (ry, y2, zz, xl, yl, z(}. The first three planes correspond to spatial dimensions, while the last three planes capture spatiotemporal variations. Each of the planes structured as a M1 M F tensor in the memory, where M represents the size of the plane and F is the feature size that en- codes scene density and color information. Let denote the timestamp of a video clip, given point in the 4D space, we normalize the coordinate to the range [0, M) and subsequently
p (z, project it onto the six planes using the equation f()). = (t)))))), where to the projection from space point to pixel the o'th plane. The plane feature f()). is extracted via bilinear in- terpolation. The six plane features are combined using the Hadamard product (i.e., element-wise multiplication), produce final feature vector follows:
f(p)
II
f(p)..
(3)
Then, the color and density of p is calculated c(p) c(f(p)) and d(p) d(f(p)), where and d denotes mips for color and density.
4
METHOD
In this work, we target generate 360 dynamic object from statically captured monocular video. To achieve this goal, we develop a framework consisting of a DyNeRF and an optional video enhancer, supervised by the pre-trained multi-view diffusion model and GAN, respectively. As shown in Figure 2, we design a Cascade DyNeRF which naturally favors temporal continuity, facilitating stable training under the time-discrete SDS loss from 2D diffusion model. To further guarantee spatial and temporal consistency, we propose novel Interpolation-drive Consistency Loss as the extra regularization for the DyNeRF. Inspired by pix2pix (Isola et al., 2017), we apply GAN to train lightweight video enhancer sional noise in 4D renderings. such a way, we obtain a DyNeRF from which we can render 360 view of the dynamic object, and the rendered results can be further enhanced by the video enhancer.
optional post processing step to refine the occa-

---

Published as a conference paper at ICLR 2024
In the following sections, will first introduce our design of the Cascade DyNeRF, and then illustrate the interolatiiondrivee Consistency loss. Video enhancer is described in the third section. At last, detail the training loss.
4.1 CASCADE DYNERF
Existing DyNeRF methods mainly assume the supervision signals are temporally coherent, however, this assumption does hold in our pipeline due the of the image diffusion models with no time notion. In order to minimize the impact of temporal discontinuity in the supervision signals, we are prone to 4D representations which naturally guarantee a certain level of temporal continuity. Therefore, we build our DyNeRF based on K-planes (Fridovich-Ken et al., 2023) which exploits temporal interpolation, an operator naturally inclined to temporal smoothing. Empirically, main- taining temporal coherence is possible when the time resolution of spatiotemporal planes is small, however, this results over-smoothed images where finer details are lost. In contrast, when the time resolution is large, the quality of the images is enhanced, but the continuity of images within the same time series diminishes. To achieve both temporal continuity and high image quality, we adjust the multi-scale technique from the original paper and introduced Cascade DyNeRF.
Let us denote the scale index by s. In original K-planes, multi-scale features are exploited by concatenation along feature dimension, then the color and density could be calculated
dp) = c(concat (LP()))))
d(p) = (concat((((p)))))
(4)
where the number of scales. our setting, simple concatenation hard to balance between image quality and temporal consistency. So propose leverage the cascade architecture and let the low-resolution planes learn temporally coherent dynamic objects with certain degree over- smoothing, and let the high-resolution planes learn the residual between the above results and the target ones. The final color and density are the addition of results from planes across the scale. That is,
dp) c(f(p))), d(p)" = (((())))
(5)
where k indicates the scale index. Note that SDS loss and other losses are applied to the rendering results of each scale guarantee that planes with higher resolution only learn the residual between results from previous scales and the target object. In this way, we can improve temporal consistency without sacrificing much object quality.
4.2 INTERPOLATION-DRIVER CONSISTENCY Loss
Video generation methods usually train an inter-frame interpolationi module enhance the tem- poral consistency between keyframes (Ho et al., 2022; Zhou et al., 2022; Blattmann et al., 2023). Inspired by this, exploit pre-trained Interpolation-drivenn Consistency Loss to enhance the spatiotemporal consistency of the 4D genera- tion.
video interpolation model and propose an
The interpolation model adopted this work is RIFE (Huang et al., 2022), which takes a pair of consccutive images as well as the interpolation ratio > (0 < 7 < 1) the input, and outputs the interpolated image. In our case, we first render a batch of images that are either spatially continuous or temporally continuous, denoted by {x} where J is the number of images batch. Let us denote the video interpolation model w. the interpolated image as &, then we calculate the Interpolation-driven Consistency Loss as:
xj (Xx1,X);771
(6)
Licr Σ - 1x11-8;112
where 7/ 3 and 2
This simple yet effective loss enhances the continuity between frames thus improving the spatiotem- poral consistency in dynamic object generation by a large margin. Moreover, we find the spatial ver- sion of this loss alleviates the multi-face problem in 3D generation tasks well. Please refer the
5

---

Published as a conference paper at ICLR 2024
experiment sections to quantitative and qualitative results. The Interpolation-driverr Consistency Loss and some other regularization losses are added with SDS loss in Equation 2, details which can be found in the experiment section.
4.3 CROSS-FRAME VIDEO ENHANCER
Sometimes image sequence rendered from the optimized DyNeRF suffers from artifacts, such as blurry edges, small floaters, and insufficient smoothness, especially when the object motion is abrupt or complex. To further improve the quality of rendered videos, we design a lightweight video en- hancer and optimize it via GAN, following pix2pix (Isola et al., 2017). The real images are obtained with image-to-image technique (Meng et al., 2021) using super-resolution diffusion model, and the fake images are the rendered ones.
To better exploit video information, We add cross-frame attention to the UNet architecture in pix2pix, i.e., each frame will query information from two adjacent frames. We believe this could enable better consistency and image quality. Denote the feature map before and after cross-frame- attention as F and F',, we have:
FJJ ,,,
2 = flatten Pj), K, = V, = flatten(concat(()-111 PD+1s
where Q, K. and V denotes query, key, and value in attention mechanism, and concat denotes the concatenation along the width dimension.
4.4 OPIMIZATION
We optimize the dynamic NeRF using SDS loss LSDS in Eq. and ICL loss LICL in Eq. 6. Besides, we apply reconstruction loss Lree and foreground mask loss L. for the imput view following Guo et al. (2023). 3D normal smooth loss C. (Guo et al., 2023) and orientation loss Cori (Verbin al., 2022) are utilized to achieve better geometry. Therefore, the final optimization objective for dynamic NeRF is calculated
CsDs A2CrCL + AaCree + 74hm + Aghn AaCori
For video enhancer optional post-processing step, the loss function the same as pix2pix (Isola al., 2017).
EXPERIMENT
We have conducted extensive experiments to evaluate the proposed Consistent4D generator using both synthetic data and in-the-wild data. The experimental setup, comparison with dynamic NeRF baselines, and ablations provided in the following sections.
5.1 IMPLEMENTATION DETAILS
Data Preparation For qualitative experiments, we collect in-the-wild and synthetic videos from Internet. For quantitative evaluation, we select and download seven animated models, namely Pistol, Guppie, Crocodie, Monster, Skull, Trump, Aurorus, from Sketchfab (ske, 2023) and render the multi- view videos by ourselves, as shown Figure 3 and appendix A.3. We render one input view for scene generation and 4 testing views for our evaluation. For each input video, we initially segment foreground object utilizing SAM (Kirillov et al., 2023) and subsequently sample 32 frames uniformly. The majority of the input videos span approximately 2 seconds, with some variations extending to around second or exceeding 5 seconds.
Training During SDS and interpolation consistency optimization, utilize zero trained by Deitke et al. (2023) as the diffusion model for SDS loss. For Cascade DyNeRF, we set = in most experiments except for the last in Table la, i.e., have loarse-level and fine-level DyNeRFs. The spatial and temporal resolution of Cascade DyNeRF are configured to 50 and 8 for coarse-level, and 100 and 16 for fine-level, respectively. We first train DyNeRF with batch size 4 and resolution

---

Published as a conference paper at ICLR 2024
image-level LPIPS CLIP
video-level
image-level
video-level
FVD
Cas-DyNeRE ICL Video enhancer LPIPS
CLIP
FVD
D-NeRF
0.51
0.68
2327.83
0.16
0.86
1303.31
K-planes
0.38
0.72
2295.68
4
0.16
0.87
1226.92
Zero
0.15
0.90
1571.60
0.15
0.88
1205.80
Ours (s=2)
0.16
0.87
1133.44
0.16
0.87
1133,44
Ours (s=4)
0.12
992.61
0.16
0.87
1114.85
(a) Comparison with other methods.
(b) Ablations.
Table I: Quantitative results on synthetic dataset.
64 5000 iterations. Then we decrease the batch size to 1 and increase the resolution 256 for the next 5000 iteration training. ICL is employed in the initial 5000 iterations with probability of 25%. The optimization of dynamic NeRF and video enhancer cost about 2.5 hours and 15 minutes on single V100 GPU. For more details, please refer to the appendix A.5.
Guppie
Trump
Aurorus
t=
t=t
t t=t
t t=
t2
t t=
t
t= t
videe video
Impart
D-NeRF
K-planes
Zero123
Ours
Comparison with dynamic NeRF methods and generative method. We render each dy- namic object from novel view at two timestamps. The temporal inconsistency in Zero123 high- lighted by the red box. but it's recommended to watch the video in supplementary material.
Figure 3:
5.2 COMPARISONS WITH OTHER METHIODS
To date, few methods have been developed for 4D generation utilizing video obtained from static camera, so we compare our method with approaches with 4D modeling capabilities, i.e., D- NeRF (Pumarola et al., 2021) and K-planes (Fridovich-Keil) et al., 2023), well approaches with 3d generation ability, i.c. Zero123 (Liu et al., 2023). For a fair comparison, video enhancer is not applied here.

---

Published as a conference paper at ICLR 2024
Quantitative Results To quantitatively evaluate the proposed video-4D generation method, pro- vide image-level metrics, LPIPS (Zhang et al., 2018) and CLIP (Radford et al., 2021), as well as video-level metric, Frechet Video Distance (FVD) (Unterthiner et al., 2018). LPIPS and CLIP are computed between testing and rendered videos per-frame way, reflecting single-frame quality. FVD is computed between video pairs, taking both kinle-frame quality and temporal coherence in the entire video into consideration. We report the scores averaged over the four testing views of seven objects Table. la (detailed metries on each object can found in the appendix). As shown in Table la, our dynamic 3D generation produces the best quantitative results over the other two dynamic NeRF methods on all metrics, which well aligns with the qualitative comparisons shown in Figure 3. Zero 123 (per-frame reconstruction) has advantages over our method in terms of image- level metric, however, lags behind the proposed method by a clear margin terms of video-level metric FVD, which indicates severe temporal incoherence in Zero 123 outputs.
Qualitative Results The outcomes of our method and those of other baselines are illustrated in Figure 3. Both D-NeRF and K-planes methods struggle to achieve satisfactory results in novel views, owing to the absence of multi-view information the training data. Zerol23, although outperforms ours in terms of image quality, suffers from severe temporal inconsistency, which could be observed via the video attached in the supplementary material. In contrast, our method manage to generate spatially and temporally coherent 4D object. Please refer the appendix A.1 for additional results.
5.3 ABLATIONS
We perform ablation studies for every component within our framework. For clarity, the video hancer is excluded when conducting all ablations except for its own. Quantitative results averaged on seven objects in the synthetic dataset are provided in Table 1b. Considering that the primary ob- jective of introducing Cascade DyNeRF and ICL loss is to enhance spatial and temporal coherence, advise readers to prioritize the video-level metric (FVD) over image-level metrics (LPIPS and CLIP) in evaluating this ablation study. The notable improvements observed in FVD scores under- score the efficacy of the proposed Cascade DyNeRF and ICL loss (as shown in the first four rows). The optional video enhancer improves the results slightly. Below, we present a qualitative analysis of Cascade DyNeRF and ICI. loss on in-the-wild monocular videos, deferring the video enhancer results to the appendix due to limited space.
w/o ICL
w/ ICL
w/o ICL
w/ ICL
preference rate(%)
24.5
75.5
success rate(5) (%)
19.3
28.6
(a) Video-to-4D.
(b) Text-to-3D.
Table 2: User study of interpolationddriver Consistency Loss.
Cascade DyNeRF In Figure 10 (see in appendix), we conduct an ablation study for Cascade DyN- eRF. Specifically, substitute Cascade DyNeRF with the original K-planes architecture, maintain- ing all other settings unchanged. In the absence of the cascade architecture, the training proves to be unstable, occasionally yielding incomplete or blurry objects, as demonstrated by the first and second objects in Figure 10. In some cases, while the model manages to generate a complete object, the moving parts of the object lack clarity, exemplified by the leg and beak of the bird. Conversely, the proposed Cascade DyNeRF exhibits stable training, leading to relatively satisfactory generation results.
Interpolation-driven Consistency Loss The introduction of Interpolation-drinn Consisteney Loss (ICL) stands as significant contribution of our work. Therefore, we conduct extensive experiments to investigate both its advantages and potential limitations. Figure 4a illustrates the ablation of both spatial and temporal Interpolation-drivenn Consistency Loss (ICL) in the video-to-4D task. Without ICL, the objects generated exhibit spatial and temporal inconsistency, face/foot issue in the blue jay and T-rex, and the unnatural pose of the seagull. Additionally, color discrepancies, such the black backside of the corgi, are also noticeable. Employing either spatial or temporal ICL mitigates the multi-face issue, and notably, the use of spatial ICL also alleviates the color defect problem. Utilizing both spatial and temporal ICL concurrently yields superior results.
evidenced by the multi-
8

---

Published as a conference paper at ICLR 2024
t=t
t=
t=
t=
t=t
t=t
t=4
t=b
tope vidto
Temppral ICL
Spetial ICL
51
ICE
(a) Video-to-4D. For each dynamic ohject, render it from novel view for timestamps with textureless rendering for each timestamp. For clarity, we describe the input videos as follows (from left to right): blue jay pecking, T-rex roaring, corgi smiling, seagull turning around.
KLL
chimpanze looking (hrough Ielescope
photo corgi standing up drinking boba
DSLR photo of playing a drum set
red eyed tree trog. 1ow poly
(b) Text-to-3D, For each 3D model, we render it from two views with textureless rendering for each view and remove the background focus the actual 3D shape.
Figure 4: Ablation of Interpolation-ddrienn Consistency Loss.
In the user study shown in Figure 2a, which averages the choices of 20 participants for 20 generated objects w/ and w/o ICL., 75% prefer the results w/ ICL.
We also test ICL loss text-to-3D tasks. We collect all prompts related to animals from the official DreamFusion project page, totaling 230, and compare the success rate of DreamFusion implemented w/ and w/o the proposed ICI. loss. The success rate comparison in Table 2b indicates results w/ ICL. always outperform results w/o it. Qualitative comparisons are presented in Figure 4b which indicates the proposed technique effectively alleviates the multi-face Janus problem and thus promotes the success rate. Details could be found in the appendix.
CONCLUSION
We introduce novel video-to-4d framework, named Consistent4D, aimed at generating 360 4D objects from a single-view uncalibrated video. We first develop a Cascade DyNeRF to facilitate sta- ble training under the discrete supervisory signals provided by image-to-image diffusion model. More crucially, we introduce an Interpolation poral consistency in 4D generation tasks. At last, lightweight video enhancer is provided as optional post- processro step. Comprehensive experiments conducted on both synthetic and in-the- wild data demonstrate the effectiveness of our method.
Consistency Loss to enhance spatial and tem-

---

Published as a conference paper at ICLR 2024
Acknowledgements. The authors would like to thank the anonymous reviewers for their valu- able comments and suggestions. This work was supported in part by the Beijing Natural Science Foundation (Grant No. JQ22014, (.223003), the Natural Science Foundation of China (Grant No. U22B2056, 62192782, 62036011, U2033210, 62102417), the Project of Beijing Science and tech- nology Committee (Project No. Z231 100005923046). Jin Gao was also supported in part by the Youth Innovation Promotion Association, CAS.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Deepfloyd. https://hugginggacce co. loyd, 2023.
Sketchfab. https://sketchhfbb cot.:/..oomddddmomdoddd 2023.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. 2023.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint, 2023.
Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, al. Nerdi: Single-view nerf synthesis with language-guider diffusion as general image priors. In arXiv preprint, 2022.
Sara Fridovich-Ken Keil, Giacomo Mcanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In arXiv preprint, 2023.
Chen Gao, Ayush Saraf, Johannes Kopf, and fa-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021.
Yuan-Chen Guo, Ying- Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Hai Zhang. threestudio: A unified framework for 3d content generation. https: //te://gittttt thub threestudic. 2023.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint, 2022.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV, 2022.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint, 2023.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, 2020.
Tianye Li. Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. CVPR, 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang. Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image to 3d object. In arXiv preprint, 2023.
11

---

Published as a conference paper at ICLR 2024
Javier Romero Gerard Pons-Moll Matthew Loper, Naureen Mahmood and Michael Black. Smpl: A skinned multiperson linear model. In ACM transactions on graphics, 2015.
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360 {ideg] reconstruction of any object from single image. In arXiv preprint, 2023.
Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In CVPR, 2021a.
Keunhong Park. Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, and Steven M Seitz. Hypemerf: A higher-dimensional represetationn for topologically varying neural radiance fields. In arXiv preprint, 2021b.
Ricardo
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2023.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Zhongzheng Ren, Xiaoming Zhao, and Alex Schwing. Class-agnostic reconstruction of dynamic objects from videos. 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let diffusion model know 3d-consistency for robust text-to-3d generation. In arXiv preprint, 2023.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In CVPR, 2023.
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. 2023.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- it-3d: High-fidelity 3d creation from single image with diffusion prior. In arXiv preprint, 2023.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In ICCV, 2021.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint, 2018
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In CVPR. IEEE, 2022.
Minh Vo, Yaser Sheikh, and Srinivasa G Narasimhan. Spatiotemporal bundle adjustment for dy- namic 3d human reconstruction in the wild. 2020.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Ych, and Greg Shakhmarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. 2022.
12

---

Published as a conference paper at ICLR 2024
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: Sigh-fidelity and diverse text-to-3d generation with variational score distillation. 2023.
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. In arXiv preprint, 2021.
Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. In arXiv preprint, 2022a.
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D22 nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. 2022b.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In CVPR, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In cvpr, 2018.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint, 2022.
Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In CVPR, 2017.
Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black. Lions and tigers and bears: Capturing non- rigid, 3d, articulated shape from images. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 ADDITIONAL VISUALIZATION RESULTS
In Figure 5, we present the result of our method on four in-the-wild videos. For clarity, we describe the input videos as follows: robot dancing. squirrel feeding. toy-spiderman dancing, toy-rabbit deforming. Due to limited space, the reviewers are strongly recommended to watch the video in the attached files to see various visualization results.
A.2 ABLATION ON CROSS-FRAME VIDEO ENHIANCER
In Figure 11, we show the proposed cross-frame video enhancer could improve uneven color distri- bution and smooth out the rough edges, shown in almost all figures, and remove some floaters, as indicated by the cat in the red and green box.
A.3 DATA USED IN VIDEO-TO-4D QUANTITATIVE EVALUATION
L
CLIFT
PAb
129%
PVD
1205
03P1
PVR
(UPF
1995
CUIF
PVD.
1998
CLEPT
PIG
C1IPT
K.
8:81
6.34
19330
1.
835
An225
316
8.5
93355
0.1
208.9
06
013
03.
118 16
8:16
6:1
wastad)
8:3
119)
40133
20131
418
116
96435
206636
135
3611
1810
616
99241
Table 3: Details of video-to-4D quantitative comparison.
Sin three dynamic objects are shown in Figure 3, we only visualize the rest four here, shown in Figure 6. The observation similar to the results the main paper. Additionally, we provide the details of quantitative comparison in Table 3.
A.4 THE NUMBER OF FRAMES
For simplicity, we sample each input video to 32 frames in all experiments. However, we find input videos without sampling sometimes give slightly better results, shown in Figure
A.5
IMPLEMENTATION DETAILS
A.5.1 CASCADE DYNERF
Initialization We follow Maige3D (Lin al., 2023) initialize the dynamic NeRF, Specifically, the blob scale and standard deviation of the density are set as 10.0 and 0.5. The activation function softplus.
Optimization We optimize the Dynamic NeRF using Equation 8, where A = 0.1, A2 2500,
Ag = 500, X₁ 50. 5 = 2.0, and Ag initially and increased 20 linearly until 5000 iterations. When applying ICL loss, we sample consecutive temporal frames at intervals of one frame and sample consecutive spatial frames at angular intervals of -15 in azimuth. Reconstruction loss and foreground mask, alternate with ICI. loss and SDS loss to optimize the model. When calculating SDS loss, the guidance scale the diffusion model set as 5, and the maximum/minimum percent of noise added the rendering images decreases linearly from 0.98/0.8 at the beginning of the training to 0.25/0.2 at the medium the training, and then kept unchanged.
A.5.2 VIDEO ENHANCER
For video enhancer architecture. we follow pix2pix (Isola al., 2017) except for that we modify the uner256 architecture to weightee version, with only three up/down layers and one cross- frame attention layer. Our codebase for video enhancer is the official GitHub repository of pix2pix, and we adopt all the default settings in their code except for Unet architecture, the learning rate and the training epochs, with the last two already mentioned the main paper. For Unet(our video enhancer), the feature dimensions for the unet layers are set 64, 128, and 256. Besides, inject
14

---

Published as a conference paper at ICLR 2024
videt
Novel view
Novel view 2
t=
1
t=t2
1
t=t
t=
t=t
t=
t=t
t=t
t=t
t=t
t=t
t=
Figure 5: Visualization results of our method. All four input videos are in-the-wild videos. The novel views presented are 22.5" and 112.5 away from the input view, respectively. The results of our methods include RGB, normal map and depth map (from left to right).
15

---

Published as a conference paper at ICLR 2024
Crocodile
Pistol
Skull
Monster
Inipu video
D-NeRF
K-plenes
Zeo123
Ours
Figure 6: Data and comparison results for video-to-4D quantitative evaluation.
sampling
wlo sumpling
sampling
weo sampling
Figure 7: Ablation video frame sampling. Videos w/ sampling contain 32 frames. Videos w/o sampling contain 72 and 39 frames for Aurorus and minions, respectively. The results of our methods include RGB and normal map (from left to right).
16

---

Published as a conference paper at ICLR 2024
wio
Novel View
DSLR photo of shiba inu playing golf wearing tartan golf clothes and hat
zoomed out DLR photo of kangaroo sitting beach playing the accordion
Figure Video-to-4D failure cases.
Figure 9: Text-to-3D failure cases.
cross-attention layer in the inner layer of the unet enable the current frame to query information from adjacent frames. For generating real images, we use DeepFloyd-II stage II (dee, 2023) in an image-to-imane way (Meng al., 2021) with denoising strength set as 0.35. Since this model is diffusion model designed for single-image super-resolution, its outputs are images with improved quality yet in lack of temporal coherence. The input image, i.e., the rendered image, is resized to 64 x 64 and the output resolution is 256 256. The prompt needed by the diffusion model is manually set, i.e., we use the "a as the prompt, which is the category of the dynamic object. For example, the prompts for dynamic objects in Figure 11 are bird, car, minions. The prompt cloud also be obtained from image or video caption models, or large language models.
A.5.3 TEXT-TO-3D DETAILS
We choose Threestudio built by (Guo et al., 2023) as the codebase since the best public im- plementation we could find. DeepFloy-IF (dee, 2023) is employed as the diffusion model, and all default tricks in Threestudio are utilized. The hyper-parameters for results w/ and w/o ICL, such as batch size and learning rate, are kept consistent between the implementations w/ and w/o ICL, except for those related to ICL. We train the model for 5000 iterations, the first 1000 iterations with batch size 8 and resolution 64, and the rest 4000 with batch size and resolution 256. The learning rate is 0.01 and the optimizer is Adam, the same as the default setting in Threestudio. The ICL loss is applied in the first 1000 iterations with probability 30% and weight 2000.
A.6 FAILURE CASES
Video-to-4D Since the video-to-4D task in this paper is very challenging, our method actually has many failure cases. For example, we fail to generate the dynamic object when the motion complex or abrupt, as shown in Figure 8. In Figure 8, the dog's tail disappears in the second image because the tail is occluded in the input image when t = tz. The frog, which is jumping up and down fast, gets blurry when f1.
Text-to-3D When applying ICL text-to-3D, we find some multi-face cases that could not be alleviated, and we show them in Figure 9.
A.7 LIMITATIONS
Although the proposed method achieves promising results for 360 dynamic object generation, our method has the following limitations: 1) Our method relies on pre-trained diffusion model, and this limits the generalization ability of our method. Particularly, since the diffusion model adopted this work trained on synthetic dataset, our model might have worse performance when the input image/video is from the real-world. 2) The performance of our model relies on the quality of input video. We find when the input video noisy, our model might not be able to generate the dynamic object in the video. 3) The training of model costs more than hours per object, and the long training time might present a challenge for practical deployment.
17

---

Published as a conference paper at ICLR 2024
t=t
t=t
t=
t=
t=t
t=
Input video
Ours K-planes
Novel view 1
Ours Cas DyNeRF
Ours K-planes
Novd view 2
Ours Cas- NeRF
Figure 10: Ablation of Cascade DyNeRF.
1=
t=
t=
1=4
t=
t=b
videt
Novel view
video enhancer
w/ vidoo enhancer
Figure 11: Ablation of Video enhancer. Please zoom in to view the details.
18


Reviews:
[{"review_id": "7eUMXWLZtU", "reviewer": "Reviewer_6R15", "summary": "This paper models Monocular Dynamic Reconstruction problem as a consistent 4D generation. Given an input image sequence the authors optimise for Dynamic Radiance field using score distillation sampling form a pre-trained diffusion model while enforcing spatial and temporal view synthesis consistencies via an video interpolator. A cascade version of K-palnes is used for modelling dynamic radiance fields and a video enhancer module is utilised to improve quality.", "questions": "1. Suggestion on paper's structure: The paper attempt to focus on too many incremental changes within individual components the 4D reconstruction pipeline uses. Some of these changes make no significant difference in the final results e.g. video enhancer though. Trying to pack and justify these incremental changes is difficult in limited space to which author did not do justice. I would highly recommend leaving some of these details out to appendix and incorporate more critical explanation of the pipeline -- which I reiterate is sufficiently novel for publication once explained clearly. Things such as loss function used to train the entire system must be included in the main body of the paper and clearly written. I would also highly recommend avoiding a story telling narrative currently used in the paper where a \n series of failed/suboptimal experiments are listed one after the other without letting the reader know what is eventually used. Instead, explaining what the proposed pipeline and methodology is, followed by reiterating novel changes will be more clear.  Incremental changes in K-plane with feature concatenations, cascade DyNerf justification followed by by statement that Cascade DyNeRF alone is not enough for spatiotemporal consistency, so we resort to extra regularization, please see in the next section\" confuses reader regarding what is novel. \n\n2. The paper uses multiple off the shelf deep networks within an iterative optimisation framework to train for radiance file, a video enhancer and GAN. Technical details for most of which is omitted even from the appendix making reproducing the experiment impossible. Authors should also comment on reconstruction times and compute required to generate the results.\n\n3. While the reconstructions looks impressive, I am not convinced with the quantitative evaluations presented in the paper. First, It is not fair to compare the proposed approach with Dynamic NeRF literature given that the baselines are designed to work on videos with sufficient camera motion only and not use many foundation models. As the proposed method introduces the 4D consistency via video interpolation module in 3D generation frameworks, would it not be more suitable to quantitatively evaluate the results with a baseline like Zero123? Ablations of ICL gives a flavour of the same but it is only qualitative. I would strongly recommend the 4D consistency and accuracy of the method is quantitatively evaluated on the synthetic sequence.\n\n4. Besides, given that the authors use the image  reconstruction loss used in NeRF (I struggled to verify the same), I disagree that traditional novel view synthesis losses (PSNR,SSIM) should be avoided. In fact, I do wonder why some the generated novel views for example T-rex roar have Seagull turning around have texture don't have texture consistent with the input views. Perhaps the authors can elaborate. In fact a through 4D reconstructions evaluation requires not only good novel view synthesis (consistent with input appearance) but a good quality key-point tracking as well as accurate structure. Given that the authors have used synthetic datasets some of this can be evaluated as well. I understand that the paper focuses on the challenging scenario of stationary camera for 4D reconstructions but in principle the methodology can be evaluated on any sequence Dynamic NeRFS are evaluated on and encourage authors to do so. \n\nSome of informal writing and notational inconsistency requiring explaination can be find below. These are non exclusive:\n\na. “DyNeRF methods mainly assume the supervision signals are temporally coherent, however, this assumption does not hold in our task,” The author never formally defines what is \"temporal coherence\" However the task paper solve is same as that of Dynamic NeRF. i.e. reconstructing deforming scene.  Perhaps authors are tring to allude to the challenges in incorporating diffusion models as priors which have no notion of time?\n\nb. “In order to minimize the impact of temporal discontinuity in the supervision signals, we are prone to DyNeRF methods with low-rankness”, Are authors suggesting they prefer to use a framework like K-planes – which imposes rank constraints - as the suitable dynamic NeRF framework? It is unclear to me why low rank will specifically help the proposed approach beyond being a useful prior for 4D reconstruction is general case. Please specify.\n\nc. Equation 6: Does the indices j and batch size J are being confused in the equation? I suppose that a sensible interpolation loss will take image x_1 and x_J as input with \\gamma_j and return the image \\hat{x_j}? Also, while the text describe the input images to be indexed 1 and J the equation itself use x_0 and x_j? The author does not differentiate between spatial and temporal constancy. Is the spatial consistency enforces 3D consistency of rendered views using virtual cameras at a single time or something else?\n\nd. “The spatial and temporal resolution of Cascade DyNeRF are configured to 50 and 8 for coarse-level, and 100 and 16 for fine-level, respectively. We first train DyNeRF with batch size 4 and resolution 64 for 5000 iterations. Then we decrease the batch size to 1 and increase the resolution to 256 for the next 5000 iteration training” I am not certain what the resolution means in the second statement. Technical details in this paragraph are very hard to understand in general and can be explained better.\n\ne. “we also apply foreground mask loss, normal orientation loss, and 3D smoothness loss.” These losses should be explained in appendix (i prefer within the body of the paper) and cited in the paper if they are coming from literature directly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper assembles latest advances in dynamic NeRFs and image guided 3D generation to create novel pipeline for consistent 4D reconstruction where state of art frame interpolation techniques play key role in enforcing consistency on single view 3D generation. The contribution has been evaluated via ablation study independently. The pipeline is sufficiently innovative for and results are impressive enough to be published in ICLR.\n\nAdditionally, few architectural innovations in terms of modelling Dynamic NeRFs can benefit the field focusing on view synthesis via geometric cues as well.", "weaknesses": "1. Writing of the paper can improve a lot. Paper's readability is significantly hampered due to informal writing style, usage of obscure terminology and grammatical errors. Crucial details are left out leaving reader to fill gaps by extrapolating prior art. Also, notations are misplaced in a few places. (I did not check all but see questions section for the details on these)\n2. Some of the experimental evaluation, especially quantitative evaluation is not appropriate and given synthetic datasets are available on which authors evaluate the proposed method already, the same can be used for more appropriate Comparisions and ablations.\n3.  Due to lack of details, over complicated setup involving too many neural networks and very unclear writing, the experiments presented in the paper are not reproducible in my opinion. See details and suggestions in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g4JxQBEcb6", "reviewer": "Reviewer_ZZjx", "summary": "The paper proposes Consistent4D, a novel framework for generating 4D dynamic objects (i.e. 360-degree views over time) from a single monocular video captured by a static camera. The work presents a novel approach for generating consistent 4D objects from monocular videos, demonstrating the promise of leveraging generative models for this challenging problem. The consistency loss helps improve both video-to-4D and text-to-3D generation.", "questions": "What is the computational cost of your approach, and is it suitable for real-time applications? Can you provide benchmarks on the computational resources required?\nHave any user studies been conducted to assess the qualitative aspects of the generated 4D objects? If not, what was the rationale behind this choice?\nWhat are the main limitations of the proposed method, and in what scenarios does it tend to fail? How do these limitations affect the practical deployment of the method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces an innovative approach to 4D object generation from monocular videos, focusing on generation instead of reconstruction, and uniquely utilizes generative models as priors. A novel consistency loss driven by video interpolation enhances spatiotemporal coherence. The method shows superior performance over baseline DyNeRF methods through extensive evaluation on synthetic and real datasets, demonstrating high-quality 4D outputs. The paper is well-structured and clearly articulated, with detailed method descriptions and thorough component ablations, effectively using tables and figures to convey results. This research represents a significant advancement in 4D content generation from monocular videos, with the potential of the introduced consistency loss to improve other generative 3D tasks. The paper is recognized as a strong contribution to the field, particularly for its consistency loss technique that could influence further generative 3D research.", "weaknesses": "The paper's potential weaknesses include a heavy reliance on pre-trained models that could limit flexibility in different scenarios, lack of detailed validation for generalizability and robustness which is crucial for assessing the method's practical applicability, and insufficient discussion on computational efficiency and scalability that are vital for real-time applications. Moreover, the absence of comprehensive quantitative and qualitative assessment metrics may impede a full understanding of the method's performance, especially when direct comparisons with state-of-the-art techniques and benchmarking are not thoroughly presented. Lastly, a detailed discussion of the technical limitations and potential failure cases would be beneficial for readers to understand the applicability and boundaries of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2CvrqmVbJ", "reviewer": "Reviewer_Yv9V", "summary": "The paper introduces \"Consistent4D,\" a novel framework designed to generate 360° 4D dynamic objects from uncalibrated monocular video footage, a task with significant implications in the field of computer vision. This innovative approach is notable for its elimination of the traditional requirements for multi-view data collection and camera calibration, thus simplifying the 3D reconstruction process from dynamic scenes. The framework consists of two main components: a Dynamic Neural Radiance Field (DyNeRF) that creates a coherent structure from sparse input data, and a video enhancer that refines the output by addressing color inconsistencies and removing artifacts.\n\nThe authors validate their method through extensive experiments on various challenging datasets, including in-the-wild videos, demonstrating its robustness and versatility. The results indicate that Consistent4D can produce high-quality reconstructions with detailed textures and consistent geometry across different viewpoints and time frames. By showcasing successful reconstructions alongside cases where the method falls short, the paper provides a transparent view of the framework's current capabilities and potential areas for future improvement. \n\nOverall, the contributions of this paper have the potential to significantly impact applications that rely on high-fidelity 3D dynamic object reconstruction, such as virtual reality, autonomous driving simulations, and digital content creation.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、 The paper introduces a novel framework that significantly deviates from traditional multi-camera setups for 3D reconstruction, which is a unique approach in the field. The combination of Dynamic Neural Radiance Fields (DyNeRF) with a video enhancement process supervised by a 2D diffusion model and a GAN is an original contribution. This integration of techniques for addressing the 360° 4D dynamic object generation from monocular video footage is highly innovative.\n\n2、The paper is structured in a manner that logically presents the problem, proposed solution, and validation, which enhances clarity and ease of understanding. The use of visual aids, such as figures and diagrams, is well-executed, providing a clear representation of the framework's functionality and the results obtained.", "weaknesses": "1、The figure 2 in this paper looks a little confusing, in terms of the input，output and loss propagation.\n\n2、 More comparisons are needed in this setting, are there other datasets which can be evaluated on this tasks. Lack of training time and inference time comparison with other methods.\n\n3、The detailed usage of the diffusion model in the method part are needed for better understanding this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7cimaA30vy", "reviewer": "Reviewer_B4LH", "summary": "This paper tackles the task of generating/reconstructing 4D objects from statically captured monocular videos: it utilizes L2 photometric loss and score-distillation sampling loss from pre-trained diffusion models. To push the quality, the authors exploit a cascaded mechanism and K-planes; to encourage consistency, the authors introduce interpolation-driven consistency loss. Qualitative and quantitative results demonstrate the effectiveness of the proposed approach.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow. The interpolation loss is interesting and seems effective.", "weaknesses": "### 1. Generative baselines\n\nIn the paper, the authors compare the proposed approach to D-NeRF and K-planes, which are regression-based methods. I think the authors should add at least one generative-based baseline to convince the effectiveness. Concretely, I think a valid baseline is to run a single-view reconstruction pipeline per frame [a] and see whether they can be consistent.\n\n[a] https://github.com/threestudio-project/threestudio#zero-1-to-3-\n\n### 2. About ablations\n\nCurrently, the only ablation in the paper is about whether to use interpolation consistency loss or not. Such ablation is only conducted with a user study.\n\nI think there are quite a few ablations missing such that it is unclear which module contributes to the performance. To name a few (all could be provided with quantitative numbers):\n\na. How does each of the following loss contribute to the performance: foreground mask loss, normal orientation loss, and 3D smoothness loss\n\nb. What if we remove the enhancer in Sec. 4.3.\n\nc. Currently, the interpolation consistency loss is used with 25% probability during training. What if we have more or have less? It would be more informative if there could be a performance curve for this.\n\nd. The weights for different losses differ quite a lot: the weight for reconstruction loss is 500 while the one for SDS is only 0.01. How the weight for SDS will change the performance?\n\n### 3. About Baselines D-NeRF and K-planes\n\nFrankly speaking, I am quite surprised about the inferior results of D-NeRF or K-planes in Fig. 3. Can the authors provide some discussion about this?\n\nAs mentioned in Bulletin 2: the proposed weight for SDS loss is only 0.01 while the weight for reconstruction loss is 500. Essentially, this is just a regression-based framework. Can authors clarify:\n\na. Do we apply the same reconstruction loss, including foreground mask loss, normal orientation loss, and 3D smoothness loss? If not, then the comparison seems misleading.\n\nb. The difference between D-NeRF and K-planes: is the only difference is that D-NeRF uses MLP while K-planes use the plane representation?\n\n### 4. Typo\n\nNot sure whether this is a typo: should Eq. (6) be $\\hat{x}_j = \\phi (x_0, x_J, \\gamma_j)$, i.e., the input is $x_J$ instead of $x_j$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxov3xBqgh", "reviewer": "Reviewer_YMYg", "summary": "The authors present a method by which a monocular, stationary video is turned into a 360 dynamic video of an object.  In particular, the authors first reconstruct a k-plane dynerf using SDS.  This is done in a hierarchical way so that low frequency spatio-temporal signal can be encoded first so that temporal continuity is preserved and then high frequency details are added afterwards.  An interpolation module is used to define losses between observed frames to further improve temporal continuity.  Finally, a video enhancement module uses a GAN to improve visual quality.", "questions": "Can the authors provide more justification for the video enhancer?  There are plenty of implementation details of the SR module in A.4, but I still would like to know what the main goal is in terms of training the GAN if you already have the SR module.\n\nHas the cascaded dynerf been evaluated for s > 2?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I think this is the first time that I've seen DyNerf combined with Dreamfusion.  The results look reasonable and the method doesn't appear overly convoluted.  The prior work section is sufficient to get the gist of what's going on and most of the bells and whistles have ablation studies.", "weaknesses": "I didn't really understand how the \"real\" images for the video enhancement GAN were generated.   There's some sort of super resolution network, but is that all the video enhancement network aims to do?  I didn't really understand why not just run this SR network on the generated images if you already have the network?  Is this a distillation step to accelerate the SR operation?  Or is the SR module only good for single image SR, but lacks temporal consistency?\n\nThe results in the supplemental have some pretty noticeable artifacts.  I think connecting DyNerf/k-planes + SDS is a pretty reasonable thing to do, but I'd hope for higher quality given the cascading approach.  In particular, the authors chose s=2 and don't increase this value.  It would be interesting to know how more layers would impact quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)