Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel approach for text-to-3D generation based on pre-trained text-conditioned diffusion models (Imagen in this paper). More specifically, the authors learn a neural radiance field by rendering at random views and use the pretrained diffusion model as the objective function to back-prop gradients to the radiance field. The authors also analyzed the difficulties of directly applying the denoising objective as the loss function and proposed a "simpler" form related to "probability density distillation."
strength_and_weaknesses: Strength: The method is pretty novel and effective. The analysis of the instability of the loss function is technically sound, and the solution makes the model effective in optimizing the high-quality 3D content based on the text prompt.

Weakness:
(1) the proposed text-to-3D generation is a bit tricky to evaluate. Since the model is per-scene optimization with a pretrained DM and text prompt, both the quality and text input will affect the synthesized output. Also, as it is difficult to have real 3D ground truth to compare the generation quality (in comparison to 2D generation) 
(2) while the generated results correspond to the text input well and sometimes generalize to very novel scenes, I think it is only thanks to the original power 2D diffusion model. The generated 3D, however, seems to have severe mode collapse issues compared to 2D models (blurry, dark, and unnatural high-contrast colors). Is it because the objective function encourages that? What might be the possible way to avoid such a generation further? How can we generate multiple 3D scenes with the same text?
(3) the model seems to require a lot of computational resources to optimize for a single scene. Will the same approach be generalized to synthesize the radiance field without optimization directly?

clarity,_quality,_novelty_and_reproducibility: The paper writing is clear and easy to follow. The proposed method is also novel and can be an interesting contribution to the diffusion and 3D synthesis community. Also, the algorithm is clear and easy to implement. However, since it relies on a strong pre-trained text-to-image diffusion model, it will be difficult to reproduce the same level of quality without access to the original model.

summary_of_the_review: This submission creates a novel way of generating 3D content without needing 3D or 2D image data. Compared to similar existing work (e.g., dreamfield),  the usage of the diffusion model makes it possible to generate highly realistic samples. Also, the proposed fix of using "probability density distillation" loss for learning with a pretrained DM is technically sound, and also has great potential to be applied to other distillation tasks. Therefore, despite some possible weaknesses listed above, it should be a solid to be appear at ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a Text-to-3D algorithm by using a pre-trained 2D text-to-image diffusion model. 
It modifies the denoising diffusion loss for better supervision signal. 
It also introduces a probability explanation of the loss called density distillation.
Using this loss in a DeepDream-like procedure, the method optimizes a randomly-initialized NeRF via gradient descent such that its 2D renderings from random angles achieve a low loss. 
strength_and_weaknesses: Strength:

+ The method allows flexible control of the resulting 3D model of the given text, e.g. it can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment.  

+ The approach requires no 3D training data and no modifications to the image diffusion model.

+ The 3D rendering process is well-considered such that many local minimums can be avoided.

+ The paper is well written, and one can quickly grasp the main idea and many technical details.

+ The paper presents a novel way to sample parameter space instead of pixel space.


Weakness:

- On P4, the paper says that it minimizes the diffusion training loss: $L_{Diff}$. As shown in Eq. 1, it is $\epsilon_\phi$ that is used to compare with $\epsilon$. However, in Eq. 2, the variable becomes $\hat{\epsilon}_\phi$. Please explain the discrepancy here.

- On P6, above Eq. 5, the paper says that it performs alpha-composition from the back of the ray to the camera. This is quite unusual, as in NeRF we usually go the opposite way, i.e. from near camera to far, such that objects near the camera are first added, and the further objects can be occluded. 

- There seems no specular term in Eq. 7. However, some specular effects can be seen in Figure 4, e.g. on the armor. I wonder where those effects come from.

- On P6, the description of the second MLP for scene generation is too brief. How important is it for the final synthesis performance? Why does this MLP only take ray direction as input? Why does it only produce RGB color instead of 3D geometry as well?

- Some important results are missing from the ablation study: 1) instead of adding components one by one, the study should consider removing each component separately, such that it is easier to understand which component is the most relevant; 2) the comparison between the new SDS loss and the previous CLIP loss in DreamField is missing.

- Minor issue: In Table 1, it is better to add a dash in the CLIP names, for example, CLIP-B/32.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. The algorithm is novel. See more comments in Strength And Weaknesses.
summary_of_the_review: The paper is well-written. The algorithm is novel. See more comments in Strength And Weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel method to generate 3D assets from text descriptions.
The proposed method utilizes a pretrained diffusion model to generate a denoised 2D image from text and noisy input.
The noisy input is rendered with a NerF model and known noise. The NerF model is trained by minimizing the noise residual between known noise and predicted noise.
The experimental results and the galleries in the supplementary material show the proposed method works quite well.

strength_and_weaknesses: ++ The proposed method should be useful for 3D asset generation, and the results are very promising.

++ The loss from Score Distillation Sampling opens a new way to use the 2D diffusion model as prior for NerF model optimization.

++ The diffusion model is pretrained and can be directly used in the text-to-3D task. 

There are also several concerns. 
-- In Fig. 1, some images are without a symbol indicating the prefix, and there is no "wide angle zoomed out..."

-- CLIP model is not explained and cited in Sec. 1 while Sec. 4 shows the reference of CLIP.

-- The ground truth noise $\epsilon$ in Eq. 1 is not defined or explained before use. What does $\epsilon$ look like?

-- It is better to define the KL operation in the main text near Eq. 4, letting the reader have a smoother understanding.

-- Why use Imagen in the proposed method?  How is the performance if using other diffusion models?
 

clarity,_quality,_novelty_and_reproducibility: Clarity: The method is described clearly, while some variables or operations are not well defined in the main text. 

The quality and novelty are very good, it is a big step to generate 3D models from text without 3D labeling.

The reproducibility is possible with diffusion models other than Imagen, but may not be totally reproducing the results due to different diffusion model qualities.


summary_of_the_review: The paper presents a novel solution for 3D assets generation from text, and the results show the proposed method work very well.
Some descriptions can be improved to make readers a smoother understanding.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper uses a NeRF backbone with CLIP-based text guidance to synthesize 3D objects. Although most components are borrowed from its prior work, Dream Fields, the paper focuses on the sampling algorithm, and shows superior performance on both qualitative and quantitative results.
strength_and_weaknesses: Strengths:
1. The visual quality is obviously phenomenal in generative modeling research, although the formulation is not generative modeling. Under the umbrella of creative content synthesis, it becomes interesting whether do we really need a real-time generative model, or simply a faster (1.5 hours in this work is still too long) inversion method.

Weaknesses:
1. This is personal, but I do not like the writing strategy of the paper. Many of the core concepts in this paper are borrowed from Dream Fields, but the abstract and introduction are written in a way as if the concept is first proposed in this paper. And the differentiation between DreamFusion and Dream Fields is written in obscure sentences. I would suggest the authors take a deeper look into how to present the differences.
2. In Figure 5., the re-implementation version of Dream Fields is visually much worse than the original version, despite the quantitative scores seeming to be fine. I would suggest the authors adding more visual comparisons between these two methods in the Appendix.

Questions:
1. Current 3D generative models mostly rely on a canonical coordinate to align objects, and therefore fail on categories without aligned canonical coordinates (such as trees or scenes). Does Dream Fusion have a similar limitation?
clarity,_quality,_novelty_and_reproducibility: As described in weaknesses, I believe the paper needs some effort in clarifying the differences between DreamFusion and Dream Fields.
summary_of_the_review: This paper is a clear acceptance, but there are some minor issues in the writing. And I see the TPUs are used in this paper, despite unlikely, but it would be greatly appreciated if the authors can make the codes publicly available soon, as an effort towards an open and welcoming research community.
review_title: null

Decision: Accept: notable-top-5%