Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yanqin Jiang
authors: Li Zhang
authors: Jin Gao
authors: Weiming Hu
authors: Yao Yao
reviews: 
Comments: The authors present a method by which a monocular, stationary video is turned into a 360 dynamic video of an object.  In particular, the authors first reconstruct a k-plane dynerf using SDS.  This is done in a hierarchical way so that low frequency spatio-temporal signal can be encoded first so that temporal continuity is preserved and then high frequency details are added afterwards.  An interpolation module is used to define losses between observed frames to further improve temporal continuity.  Finally, a video enhancement module uses a GAN to improve visual quality.
Strengths: I think this is the first time that I've seen DyNerf combined with Dreamfusion.  The results look reasonable and the method doesn't appear overly convoluted.  The prior work section is sufficient to get the gist of what's going on and most of the bells and whistles have ablation studies.
Weaknesses: I didn't really understand how the "real" images for the video enhancement GAN were generated.   There's some sort of super resolution network, but is that all the video enhancement network aims to do?  I didn't really understand why not just run this SR network on the generated images if you already have the network?  Is this a distillation step to accelerate the SR operation?  Or is the SR module only good for single image SR, but lacks temporal consistency?

The results in the supplemental have some pretty noticeable artifacts.  I think connecting DyNerf/k-planes + SDS is a pretty reasonable thing to do, but I'd hope for higher quality given the cascading approach.  In particular, the authors chose s=2 and don't increase this value.  It would be interesting to know how more layers would impact quality.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper tackles the task of generating/reconstructing 4D objects from statically captured monocular videos: it utilizes L2 photometric loss and score-distillation sampling loss from pre-trained diffusion models. To push the quality, the authors exploit a cascaded mechanism and K-planes; to encourage consistency, the authors introduce interpolation-driven consistency loss. Qualitative and quantitative results demonstrate the effectiveness of the proposed approach.
Strengths: The paper is easy to follow. The interpolation loss is interesting and seems effective.
Weaknesses: ### 1. Generative baselines

In the paper, the authors compare the proposed approach to D-NeRF and K-planes, which are regression-based methods. I think the authors should add at least one generative-based baseline to convince the effectiveness. Concretely, I think a valid baseline is to run a single-view reconstruction pipeline per frame [a] and see whether they can be consistent.

[a] https://github.com/threestudio-project/threestudio#zero-1-to-3-

### 2. About ablations

Currently, the only ablation in the paper is about whether to use interpolation consistency loss or not. Such ablation is only conducted with a user study.

I think there are quite a few ablations missing such that it is unclear which module contributes to the performance. To name a few (all could be provided with quantitative numbers):

a. How does each of the following loss contribute to the performance: foreground mask loss, normal orientation loss, and 3D smoothness loss

b. What if we remove the enhancer in Sec. 4.3.

c. Currently, the interpolation consistency loss is used with 25% probability during training. What if we have more or have less? It would be more informative if there could be a performance curve for this.

d. The weights for different losses differ quite a lot: the weight for reconstruction loss is 500 while the one for SDS is only 0.01. How the weight for SDS will change the performance?

### 3. About Baselines D-NeRF and K-planes

Frankly speaking, I am quite surprised about the inferior results of D-NeRF or K-planes in Fig. 3. Can the authors provide some discussion about this?

As mentioned in Bulletin 2: the proposed weight for SDS loss is only 0.01 while the weight for reconstruction loss is 500. Essentially, this is just a regression-based framework. Can authors clarify:

a. Do we apply the same reconstruction loss, including foreground mask loss, normal orientation loss, and 3D smoothness loss? If not, then the comparison seems misleading.

b. The difference between D-NeRF and K-planes: is the only difference is that D-NeRF uses MLP while K-planes use the plane representation?

### 4. Typo

Not sure whether this is a typo: should Eq. (6) be $\hat{x}_j = \phi (x_0, x_J, \gamma_j)$, i.e., the input is $x_J$ instead of $x_j$?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: The paper introduces "Consistent4D," a novel framework designed to generate 360° 4D dynamic objects from uncalibrated monocular video footage, a task with significant implications in the field of computer vision. This innovative approach is notable for its elimination of the traditional requirements for multi-view data collection and camera calibration, thus simplifying the 3D reconstruction process from dynamic scenes. The framework consists of two main components: a Dynamic Neural Radiance Field (DyNeRF) that creates a coherent structure from sparse input data, and a video enhancer that refines the output by addressing color inconsistencies and removing artifacts.

The authors validate their method through extensive experiments on various challenging datasets, including in-the-wild videos, demonstrating its robustness and versatility. The results indicate that Consistent4D can produce high-quality reconstructions with detailed textures and consistent geometry across different viewpoints and time frames. By showcasing successful reconstructions alongside cases where the method falls short, the paper provides a transparent view of the framework's current capabilities and potential areas for future improvement. 

Overall, the contributions of this paper have the potential to significantly impact applications that rely on high-fidelity 3D dynamic object reconstruction, such as virtual reality, autonomous driving simulations, and digital content creation.
Strengths: 1、 The paper introduces a novel framework that significantly deviates from traditional multi-camera setups for 3D reconstruction, which is a unique approach in the field. The combination of Dynamic Neural Radiance Fields (DyNeRF) with a video enhancement process supervised by a 2D diffusion model and a GAN is an original contribution. This integration of techniques for addressing the 360° 4D dynamic object generation from monocular video footage is highly innovative.

2、The paper is structured in a manner that logically presents the problem, proposed solution, and validation, which enhances clarity and ease of understanding. The use of visual aids, such as figures and diagrams, is well-executed, providing a clear representation of the framework's functionality and the results obtained.
Weaknesses: 1、The figure 2 in this paper looks a little confusing, in terms of the input，output and loss propagation.

2、 More comparisons are needed in this setting, are there other datasets which can be evaluated on this tasks. Lack of training time and inference time comparison with other methods.

3、The detailed usage of the diffusion model in the method part are needed for better understanding this paper.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper proposes Consistent4D, a novel framework for generating 4D dynamic objects (i.e. 360-degree views over time) from a single monocular video captured by a static camera. The work presents a novel approach for generating consistent 4D objects from monocular videos, demonstrating the promise of leveraging generative models for this challenging problem. The consistency loss helps improve both video-to-4D and text-to-3D generation.
Strengths: The paper introduces an innovative approach to 4D object generation from monocular videos, focusing on generation instead of reconstruction, and uniquely utilizes generative models as priors. A novel consistency loss driven by video interpolation enhances spatiotemporal coherence. The method shows superior performance over baseline DyNeRF methods through extensive evaluation on synthetic and real datasets, demonstrating high-quality 4D outputs. The paper is well-structured and clearly articulated, with detailed method descriptions and thorough component ablations, effectively using tables and figures to convey results. This research represents a significant advancement in 4D content generation from monocular videos, with the potential of the introduced consistency loss to improve other generative 3D tasks. The paper is recognized as a strong contribution to the field, particularly for its consistency loss technique that could influence further generative 3D research.
Weaknesses: The paper's potential weaknesses include a heavy reliance on pre-trained models that could limit flexibility in different scenarios, lack of detailed validation for generalizability and robustness which is crucial for assessing the method's practical applicability, and insufficient discussion on computational efficiency and scalability that are vital for real-time applications. Moreover, the absence of comprehensive quantitative and qualitative assessment metrics may impede a full understanding of the method's performance, especially when direct comparisons with state-of-the-art techniques and benchmarking are not thoroughly presented. Lastly, a detailed discussion of the technical limitations and potential failure cases would be beneficial for readers to understand the applicability and boundaries of the proposed method.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair