Text-to-3D with Classifier Score Distillation
Authors: Xin Yu
Yuan-Chen Guo
Yangguang Li
Ding Liang
Song-Hai Zhang
XIAOJUAN QI
Venue: ICLR 2024 poster (2024)
Keywords: 3D modeling
generative models
Primary area: generative models

Abstract: Text-to-3D generation has made remarkable progress recently, particularly with methods based on Score Distillation Sampling (SDS) that leverages pre-trained 2D diffusion models. While the usage of classifier-free guidance is well acknowledged to be crucial for successful optimization, it is considered an auxiliary trick rather than the most essential component. In this paper, we re-evaluate the role of classifier-free guidance in score distillation and discover a surprising finding: the guidance alone is enough for effective text-to-3D generation tasks. 
We name this method Classifier Score Distillation (CSD), which can be interpreted as using an implicit classification model for generation. This new perspective reveals new insights for understanding existing techniques. We validate the effectiveness of CSD across a variety of text-to-3D tasks including shape generation, texture synthesis, and shape editing, achieving results superior to those of state-of-the-art methods. Our project page is https://xinyu-andy.github.io/Classifier-Score-Distillation

Published as a conference paper at ICLR 2024
TEXT-TO-3D WITH CLASSIFIER SCORE DISTILLATION
Xin Yu'x Yuan-Chen Guo
Yangguang Li Ding Liang' Song-Hai Zhang' Xiaojuan Qi!
The University of Hong Kong Tsinghua University VVST
ABSTRACT
Text-to-3D generation has made remarkable progress recently, particularly with methods based on Score Distillation Sampling (SDS) that leverages pre-trained 2D diffusion models. While the usage of classifier-free guidance is well acknowl- edged to be crucial for successful optimization, it is considered an auxiliary trick rather than the most essential component. In this paper, we re-evaluate the role of classifier-free guidance in score distillation and discover surprising finding: the guidance alone is enough for effective text-to-3D generation tasks. We name this method Classifier Score Distillation (CSD), which can be interpreted as using an implicit classification model for generation. This new perspective reveals new in- sights for understanding existing techniques. We validate the effectiveness of CSD across variety of text-to-3D tasks including shape generation, texture synthesis, and shape editing, achieving results superior to those of state-of-the- art methods. Our project page is https://xinyu-andy wittaaannddggggdiiiiiee wtt.buu....//lsssfieeeeeeeee cisoor--istlllationn
INTRODUCTION
3D content creation is important for many applications, such interactive gaming, cinematic arts, AR/VR, and simulation. However, it is still challenging and expensive to create a high-quality 3D asset as requires a high level of expertise. Therefore, automating this process with generative models has become an important problem, which remains challenging due to the scarcity of data and the complexity of 3D representations.
Recently, techniques based on Score Distillation Sampling (SDS) (Poole et al., 2022; Lin et al., 2023: Chen al., 2023; Wang et al., 2023b), also known Score Jacobian Chaining (SJC) (Wang et al., 2023a), have emerged as major research direction for text-to-3D generation, as they can produce high-quality and intricate 3D results from diverse text prompts without requiring 3D data for training. The core principle behind SDS is to optimize 3D representations by encouraging their rendered images to move towards high probability density regions conditioned on the text, where the supervision is provided by pre-trained 2D diffusion model (Ho et al., 2020; Sohl-Dickstein et al., 2015; Rombach et al., 2022; Saharia et al., 2022; Balaji et al., 2022). DreamFusion (Poole al., 2022) advocates the use of SDS for the optimization of Neural Radiance Fields (NeRF). (Barron et al., 2022; Mildenhall et al., 2021). Subsequent research improve the visual quality by introducing coarse-to-fine optimization strategies (Lin et al., 2023; Chen al., 2023: Wang al., 2023b), efficient 3D representations (Lin et al., 2023; Chen et al., 2023), multi-view-consiiteen diffusion models (Shi al., 2023; Zhao et al., 2023), or new perspectives on modeling the 3D distribution (Wang et al.. 2023b). Despite these advancements, all these works fundamentally rely on score distillation for optimization.
Although score distillation is theoretically designed to optimize 3D representations through proba- bility density distillation (Poole al., 2022: Oord al., 2018), as guided by pre-trained 2D diffusion models (i.e., Eq. (4)), a practical gap emerges in its implementation due to the widespread reliance on classifier-free guidance (CFG) (Ho & Salimans, 2022). When using CFG, the gradient that drives the optimization actually comprises two terms. The primary one is the gradient of the log data den- Ps (x, y), estimated by the diffusion models to help move the synthesized images x to high-density data regions conditioned on text prompt y. which is the original optimization ob- jective (see Eqs. (4) and (5)). The second term is the gradient of the log function of the posterior,
sity, i.c., 1ogps
"This work is in collahoration with VAST. Corresponning author.

---

Published as a conference paper at ICLR 2024
"o DSLR photo of borbecue gril cooking sausages and burger patties"
DSLR photo robot and cincssur DSLR photo of drum set made of playing chess, high reselution"
DSLR photo cdible typewriter mode out of vegetobles"
cheese"
"a DSLR photo of pig playing drum sel*
zeomed out DSLR photo of chimporase wearing headphones"
zoomed out D.SLR photo of food cotchhing a fly with its tongue"
rest with few white eggs and one golden egg"
D-SLR photo of a ned pickup truck criving scrces stream"
DSLR photo chinpanzee dressed like Henry VIII king of Englond
DSLR photo of kodla wearing a party hat and blowing out birthday
D.SLR photo wearing virtual reclity headset in reneissance oil pointing Corovaggie"
(a) NeRF generation from text-prompts
"a wide ongle D.LL photo of humonoid banano sitting desk doing homework"
zoomed out D.SLR photo of panda weaning chef's hat and leeesiin briadd dough countertop"
"a DSLR of forging zoomed DSLR photo of astronout chopping vegetables surlit kitchen"
(b) Textured mesh generation from text-prompts
"Uniform jocket of the captain"
'Humareid with figer head, odorned Chinsse-style wielding traditional weopons"
"Rugged, vinhoge inspired liking boots with westhered loather finish"
"Cinika lives in the forest"
(c) Synthesize texture for given meshes from text-prompts
Figure 1: Illustrative overview of our method's capabilities. (a) Generation of a Neural Radiance Field (NeRF) from text inputs, trained using a 64x64 diffusion guidance model. (b) Subsequent refinement leads to high-quality textured meshes using a 512x512 diffusion guidance model. (c) Texture synthesis on user-specified meshes, resulting highly realistic and high-resolution detail.
2

---

Published as a conference paper at ICLR 2024
ie., log Pa (y/xx), which can be empirically interpreted Salimans, 2022). We elaborate more on this in Sec. 3. The combination of these two components determines the final effect of optimizing the 3D representation. Notably, although the use of CFG (i.e., the second term) is widely recognized as vital for text-to-3D generation, it is considered an auxiliary trick to help the optimization, rather than the primary driving force.
implicit classification model (Ho &
In this paper, we show that the classifier component of Score Distillation Sampling (SDS) is not just auxiliary but essential for text-to-3D generation, and using only this component sufficient. We call this paradigm Classifier Score Distillation (CSD). This insight fundamentally shifts our understanding of the mechanisms underlying the success of text-to-3D generation based on score distillation. Specifically. the efficacy stems from distilling knowledge from an implicit classifier Pe (y) xx) rather than from reliance on the generative prior Po (xx(y). This finding further enables us to uncover fresh insights into existing design choices: 1) We demonstrate that utilizing negative prompts can be viewed as a form of joint optimization with dual classifier scores. This realization enables to formulate an annealed negative classifier score optimization strategy, which enhances generation quality while maintaining the result faithfulness according to prompts. 2) Building on understanding, we illustrate the utilization of classifier scores for efficient text-driven 3D editing. 3) The Variational Score Distillation technique (Wang et al., 2023b) can be viewed as an adaptive form of negative classifier score optimization, where the negative direction supplied by diffusion model trained concurrently.
CSD can be seamlessly integrated into existing SDS-based 3D generation pipelines and applica- tions, such as text-driven NeRF generation, textured mesh generation, and texture synthesis. As demonstrated in Fig. I, our method produces high-quality generation results, featuring extremely photo-realistic appearances and the capability to generate scenes corresponding to complex textual descriptions. We conduct extensive experiments to evaluate the robustness of our approach and compare our method against existing methods, achieving state-of-the- results.
DIFFUSION MODELS
The diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021; Song & Er- mon, 2020) is a type of likelihood-based generative model used for learning data distributions. Given an underlying data distribution q(x), the model initiates a forward process that progres- sively adds noise to the data x sampled from (x).. This produces a sequence of latent variables
{xo x, xi,..,, xr). The process is defined as Markov Chain ((x1.r x) == I[= q(xx/xxxxx) (x-1) with Gaussian transition kernels. The marginal distribution of latent variables at time & is given by q(x, |x) Naax I).. This is equivalent to generating a noise sample x; using the equation Xx = 02x + oye, where ~ (O, I). Parameters ae and o, are chosen such that of + of = 1, and oe gradually increases from 0 1, Thus, ((x,) converges to Gaussian prior distribution (O I).
Next, reverse process (i.c.. the generative process) is executed to reconstruct the original signal from Xy. This is described by a Markov process Ps (XoTT) == p(xr) IITm Pa (xx) with the transition kernel Ps (x--1 xx) N (us (xx, (). If).. The training objective is to optimize ng by maximizing a variational lower bound of the log data likelihood. In practice, Hg is re-parameterized as denoising network es (Ho et al., 2020) and the loss can be further simplified Mean Squared
Error (MSE) criterion (Ho et al., 2020; Kingma et al., 2021):
Lia (+) (xx- (. (w((t) |lea (axx o1e; - c[12
where wt) is time-dependent weights
The objective function can be interpreted predicting the noise that corrupts the data x. Besides, is correlated to NCSN denoising score matching model (Ho et al., 2020; Song & Ermon, 2020), and thus the predicted noise is also related the score function of the perturbed data distribution g(x,), which is defined as the gradient of the log-density with respect to the data point:
Vx, log (x,) = -cs (xx:) /at-
(2)
This means that the diffusion model can estimate a direction that guides x, towards a high-density region of g(x)), which is the key idea of SDS for optimizing the 3D scene. Since samples in high-
3

---

Published as a conference paper at ICLR 2024
density regions of ((xx) are assumed to reside in the high-density regions of q(x(-1) (Song & Ermon, 2020), repeating the process can finally obtain samples of good quality from (xx).
Classifier-Free Guidance Text-conditioned diffusion models (Balaji et al., 2022; Saharia et al., 2022: Rombach al., 2022: Ramesh et al., 2022) generate samples x based on a text prompt y. which is also fed into the network as an input, denoted as co (, t). A common technique to improve the quality of these models is classifier-fre guidance (CFG) (Ho & Salimans, 2022). This method trains the diffusion model in both conditioned and unconditioned modes, enabling it to estimate both Xx, log y (x, (yly)) and Vx, log q (xx), where the network are denoted as 6 (Xx: y, () and eg (xs;; t). During sampling, the original score is modified by adding a guidance term, i.e., (x( y, t) + (x(x; y, + w [eo (x;: y. - es (xe: t)), where w is the guidance scale that con- trols the trade-off' between fidelity and diversity. Using Eq. (2) and Bayes' rule, we can derive the following relation:
[e y, t) (x(;;; t)] Xx, log 9 (xxlv) Xx, log q (xx) x Vx. logg (y/x+). (3)
Thus, the guidance can be interpreted as the gradient of an implicit classifier (Ho & Salimans, 2022).
3 WHAT MAKES SDS WORK?
Score Distillation Sampling (SDS) (Poole et al., 2022) is novel technique that leverages pretrained 2D diffusion models for text-to-3D generation. SDS introduces a loss function, denoted as LsDs, whose gradient is defined as follows:
LSLSD (o(0)) = Er,,. wi(()
Tt Vo (1 g(0; c)) lip (xclw)) at
(4)
where 0 is the learnable parameter of a 3D representation (e.g.. NeRF), and is a differentiable rendering function that enables obtaining the rendered image from the 3D scene 0 and camera c. The optimization aims to find modes of the score functions that are present across all noise levels throughout the diffusion process which is inspired by work on probability density distillation (Poole et al., 2022: Oord et al., 2018).
Eq. (4) can be simplified into: Er,,.c (es()(es(())) ()) D6 This formulation more intuitive than Eq. (4): The conditional score function e (x,;; (. y. indicates the gradient for updating x, to be closer to high data density regions (see Eq. (2)), allowing the parameter 0 to be trained using the chain rule. However, in practice, DreamFusion recommends adjusting the gradient 6g y, 4) to Ex(x;; y, not alter the overall objective because Er,,, (w(t) update is expressed as:
ax
since this modification has been found to improve convergence. This change does 0x
O. Consequently, the final gradient to 06
dx
VoLSDS = Er,,c ((t))(e(xx))))))
(5)
00
What Makes SDS Work? A crucial aspect of score distillation involves computing the gradient to be applied the rendered image during optimization, which we denote 5g () (x(x;; y, f) This encourages the rendered images to reside in high-density areas conditioned on the text prompt y. From Eq. (4), in principle, (x(;; y, t) should represent the pure text-conditioned score function. However, in practice, classifier-free guidance is employed in diffusion models with a large guidance weight w (e.g.,
100 in DreamFusion (Poole et al., 2022)) to achieve high quglituaitty results, causing the final gradient applied to the rendered image to deviate from Eq. (5). Specifically, 5, with CFG is expressed as:
8 (Xxi t) [Eo (Xei y, el +w [eg(Xri y, E (Xc;
(6)
geea
8
The gradient can be decomposed into two parts, i.c., 8² := 8gen + w 8chs According to Eq. (2) and Eq. (3), the first term 82th associated with Vx, log q (xx This term signifies the gradient direction in which the image should move to become more realistic conditioned on the text, which we refer to as the generative prior. gels is related Vx, log q representig the update direction required for the image to align with the text evaluated by an implicit classifier model, which emerges from the diffusion model (Ho& Salimans, 2022). We refer to this part as the classifier score.
4

---

Published as a conference paper at ICLR 2024
Thus, there is a gap between the theoretical formu- lation in Eq. (4) and the practical implementation in Eq. (6). To better understand the roles of different terms, we validate their contribution by visualizing the gradient norm throughout the training of NeRF via SDS with guidance scale w = 40. As shown in Fig. (a), the gradient norm of the generative prior is several times larger than that of the classi- fier score. However, to generate high-quality results, large guidance weight must be set, as shown in Fig. 2 (b). When incorporating both components, the large guidance weight actually causes the gradient from the classifier score to dominate the optimiza- tion direction. Moreover, the optimization process fails when relying solely on the generative compo- nent, as indicated by setting observation leads us to question whether the clas- sifier score is the frue essential component that drives the optimization. To explore this. we intro- duce Classifier Score Distillation (CSD), which em- ploys only the classifier score for optimization.
Norn of 6g
Norm of 65th
1003
2000
3000
4000
5000
Cytimiesicct skge
(a) The gradient norm of two terms during optimizazion (or = 10)
in Eq. (6). This
Rup Different results with difierence giidance weights
100
v-- 40
Figure (a) The gradient norm during op- timization. (b) Optimization results through different guidance weights.
CLASSIFIER SCORE DISTILLATION (CSD)
Consider text prompt y and parameterized image generator g(0). We introduce a loss function denoted as Classifier Score Distillation LESD, the gradient of which is expressed as follows:
0x
VeCcSD Er,c, (())(e((x5)))) -000)==((xx))
ex(xx;
30
Our loss fundamentally different from the previous principle presented in Eq. (4). According to Eq. (3), we use an implicit classification model that is derived from the generative diffusion models to update the 3D scene. Specifically, text-to-image diffusion models with CFG can generate both conditioned and unconditioned outputs, and we can derive a series of implicit noise-aware classifiers (Ho & Salimans, 2022) for different time steps t, based on Eq. (3). These classifiers can evaluate the alignment between the noisy image and the text y. Thus, the overall optimization objective in Eq. (7) seeks to refine the 3D scene in such a manner that rendered images at any noise level t align closely with their respective noise-aware implicit classifiers. We will use 8eh (x.; y, (x(( y, t) es (x;; 1)] to denote the classifier score for further discussions.
We note that some previous works have also used text and image matching losses for 3D generation, rather than relying on a generative prior, such as the CLIP loss (Jain et al., 2022; Michel et al., 2022). However, these methods are still less effective than score distillation methods based on the generative diffusion model. In the next section, we show that our loss function can produce high-quality results that surpass those using the SDS loss. Moreover, our formulation enables more flexible optimization strategies and reveals new insights into existing techniques. In the following, we first introduce improved training strategies based on CSD, then we develop the application for 3D editing and discuss the connections between CSD and Variational Score Distillation (VSD).
CSD with Annealed Negative Prompts We observe that employing negative prompts Yueg. text queries describing undesired images, can accelerate the training process and enhance the quality of the generated results. We provide an interpretation from the perspective of CSD. Specifically, with negative prompts, the SDS loss can be rewritten as:
guds = (xti y, t) el + w [Eo(Xri y/ t) - es (Xti Hoeg, t)]
(8)
Recall that (xx: t) == cs (xx;;,,) t) - (X;; t)], we arrive at the following expression: (x,;
(x+; yeng: t) = =xcs (x4: y, t) es (x;: t)] IE (x4: Uneg t) c (xx:t)
88 y. gels (xx: Yocg, t)
Therefore, the use of negative prompts can be seen as a dual-objective Classifier Score Distillation: It not only pushes the model toward the desired prompt but also pulls it away from the unwanted
5

---

Published as a conference paper at ICLR 2024
states, evaluated by two classifier scores. However, this may also cause trade-off between quality and fidelity, as the negative prompts may make the optimization diverge from the target text prompt y, resulting in mismatched text and 3D outputs, shown in Fig. 5.
Given CSD and our novel insights into the effects of negative prompts, we introduce an extended CSD formulation that redefines dals as follows:
gohs = 8% (xt: y,t) w2 8% (xx: Moog, t)
-w1 (x.: v, t) co <x(xx)t))] We [[Ex(xxi Yreg, t) - (x(;t))) =w1 (x,: y, t) + (w2-wi) w1) cØ (x(xx)) - w2 (x( Yueg, t)
(10)
We use different weights wi and wa for the positive and negative prompts, respectively, to mitigate the negative effects of the latter. By gradually decreasing W2, we observe that our modified loss function enhances both the quality and the fidelity of the texture results, as well as the alignment with the target prompt (refer to Fig. 5). Note that this simple formulation is based on our new interpretation of negative prompts from the perspective of Classifier Score Distillation, and it is not part of the original SDS framework.
CSD for Text-Guided Editing The CSD method also enables us to perform text-guided 3D edit- ing. Suppose we have a text prompt Starget that describes the desired object and another text prompt Meda that specifies an attribute that we want change in the original object. We can modify the loss function in Eq. (10) by replacing y and Yneg with Murggt and Yedat respectively:
geeiit
W1 deh (xi 3harget 1) W2 8eh (Xei Medits 8) =w Epp (Xri Mtarget t) + (w2 w1) eo (xx;t) w2 (xx; Blodit f).
(11)
As shown in Fig. 6, this method allows us to edit the rendered image according to the target descrip- tion Muaget while modifying or removing the attribute in Yedit- Text-to-3D generation can be seen as a special case of this method, where we set W2 = 0 and no specific attribute in the original scene is given be edited. Our method shares similar formulation as Delta Denoising Score (Hertz al., 2023) for image editing, which we discuss in the Appendix. Furthermore, we can adjust the weights W1 and w2 balance the alignment with the text prompt and the fidelity to the original scene.
Discussions and Connection to Variational Score Distillation (VSD) With CSD, the new varia- tional score distillation (VSD) (Wang al., 2023b) can be interpreted from the perspective of using a negative classifier score. VSD enhances SDS by replacing the noise term e with the output of another rext-conditioned diffusion model, denoted as Egt, which is simultaneously trained using the rendered images from the current scene states. When optimizing the 3D scene, the gradient direction applied to the rendered image is:
goud_ = =<<0(x; v, (x 6)] + w (xx(x)))) 1) c (x(;t)] =(w + 1) (x):,,))) ea (xx: ()] [ea- (x,53, - cx(xx;
(12)
Since Eg- is obtained by fine-tuning the conditional branch of a pre-trained diffusion model us- ing LoRA (Hu al., 2021), we can assume that <x(x;;t) so ex' (Xr;t) and then the term [cs- (x,; f) (x;; corresponds to predicting Vx, log ps- (y/ (x)), where Pa- is a shifted distri- bution that adapts to the images rendered from the current 3D object. Thus, Eq. (12) encourages the optimization direction to escape from Pœ- which can be interpreted as an ability to adaptively learn negative classifier scores instead of relying on predefined negative prompt. However, this also introduces training inefficiency and instabilities. In our experiments, we find that CSD combined with general negative prompts can achieve high-quality texture quality comparable to VSD.
EXPERIMENTS
We evaluate the efficacy of our proposed Classifier Score Distillation method across three tasks: text-guided 3D generation, text-guided texture synthesis, and text-guided 3D editing. We present qualitative and quantitative analysis for text-guided 3D generation in Sec. 5.2 and text-guided texture synthesis in Sec. 5.3. To further substantiate the superiority of our approach, conduct user studies for these two tasks. To showcase the capabilities of our formulation in 3D editing, illustrative examples provided in Sec. 5.4.

---

Published as a conference paper at ICLR 2024
angle photo skiing penguan wearing a puffy jacker
to soomed out DSLR photo of bulldozer made out of toy bricke"
zocmed our bsLR photo of 30 model of adorable cottage with thatched roof"
DreamFusion
Magic3D
Fantasia3D
PreaifiDDrammrr
Ours
Figure 3: Qualitative comparisons to baselines for text-to-3I generation. Our method generate 3D scenes that align well with input text prompts with realistic and detailed appearances.
"Elf with ethoneal. butterfly myetical
hondles the shose of etylized dragens"
Input mesh&pprompt
Magic
ProlificDreameer
Figure 4: Qualitative comparisons to baselines for text-guided texture synthesis on 3D meshes. Our method generates more detailed and photo-realistic textures.
5.1 IMPLEMENTATION DETAILS
Text-Guided 3D Generation We follow Magic3D (Lin al., 2023) to initially generate scene represented by Neural Radiance Fields (NeRF) using low-resolution renderings. Subsequently, the scene converted into a triangular mesh via differentiable surface extraction (Shen et al., 2021) and further refined using high-resolution mesh renderings by differentiable rasterization (Laine et al., 2020). For the NeRF generation, we utilize the DeepFloyd-IF stage-I model (Stability 2023), and for the mesh refinement, we use the Stable Diffusion 2.1 model (Rombach et al., 2022) to enable high-resolution supervision. For both stages, CSD is used instead of SDS.
Text-Guided Texture Synthesis Given a mesh geometry and text prompt, we apply CSD to ob- tain texture field represented by Instant-NGP (Müller et al., 2022). We employ ControlNets (Zhang & Agrawala, 2023) based on the Stable Diffusion 1.5 as our diffusion guidance since it can improve alignment between the generated textures and the underlying geometric structures. Specifically, apply Canny edge ControlNet where the edge is extracted from the rendered normal maps, and depth ControlNet rendered depth maps.

---

Published as a conference paper at ICLR 2024
Table 1: User study on two tasks. In both tasks, more users prefer our results.
Table 2: Quantitative comparisons to base- lines for text-to-3D generation, evaluated by CLIP Score and CLIP R-Precision.
Methods
Text (8)
Texture Synthesis (9)
Fusiin (Poole at., 2022)
30.3
Methock
CLIP
R-Precision (98)
Magic3D (Lin et al., 2023)
10.3
12.4
Score (1)
R81
RM8
10
Fintasu33 (Chen et al., 2023)
11.0
ProfifeDreamer (Wing al. 202
7.9
DecamPesioo (Poole al., 2022)
67.5
TEXTure (Rachundson et al., 2023)
11.0
Magic3D (Lin al. 2023)
Ours
59.4
57.7
Ours
78.6
81.8
94.8
96.3
5.2 TEXT-GUIDED 3D GENERATION
Qualitative Comparisons. We present some representative results using CSD in Fig. 1, includ- ing both NeRF generation (a) and mesh refinement (b) results. In general, CSD can generate 3D) scenes that align well with input text prompts with realistic and detailed appearances. Even only trained with low-resolution renderings (a), our results do not suffer from over-smoothness or over- saturation. We also find that our approach excels in grasping complex concepts in the input prompt, which is a missing property for many previous methods. In Fig. 3, we compare our generation re- sults with FreamFusion (Poole et al., 2022), Magic3D (Lin et al., 2023), Fantasia3D (Chen et al., 2023), and PreaifieDreamer (Wang et al., 2023b). For DreamFusion, we directly take the results provided on its official website. For the other methods, we obtain their results using the implemen- tations from threestudio (Guo et al., 2023). Compared with SDS-based methods like DreamFusion, Magic3D, and Fantasia3D, our results have significantly more realistic appearances. Compared with ProlificDreamer which uses VSD, our approach can achieve competitive visual quality while being much faster, i.e., hour on single A800 GPU as opposed to 8 hours required by Preaiee
Quantitative Evaluations. We follow previous work (Jain et al., 2022; Poole et al., 2022; Luo et al., 2023) to quantitatively evaluate the generation quality using CLIP Score (Hessel et al., 2022; Radford et al., 2021) and CLIP R- R-Precision (Park et al., 2021). Specifically, the CLIP Score mea- sures the semantic similarity between the renderings of the generated 3D object and the input text prompt. CLIP R-Precision measures the top-N accuracy of retrieving the input text prompt from set of candidate text prompts using CLIP Score. We generate 3D objects using 81 diverse text prompts from the website of DreamFusion. Each generated 3D object is rendered from 4 different views (front, back, left, right), and the CLIP Score is calculated by averaging the similarity between each rendered view and text prompt. We use the CLIP ViT-B/32 (Radford et al., 2021) model to extract text and image features, and the results are shown in Tab. 2. Our approach significantly outperforms DreamFusion and Magic3D in terms of CLIP Score and CLIP R- R-Precision, indicating better alignment between the generated results and input text prompts.
User Studies. We conduct a user study for more comprehensive evaluation. We enlist 30 par- ticipants and ask them to compare multiple outputs, selecting the one they find most aligned with criteria such as visual quality and text alignment. In total, we collect 2289 responses. The results, presented in Tab. (Text-to-3D), reveal that 59.49 99.4% of the responses prefer our results, demonstrating the superior quality of our approach.
5.3 TEXT-GUIDED TEXTURE SYNTHESIS
We select 20 diverse meshes from Objaverse (Deitke et al., 2022) for the texture generation task. We compare our generation results with those from Magic3D (Lin et al., 2023), Fantasia3D (Chen al., 2023), PromificDreamer (Wang et al., 2023b), and TEXTure (Richardson et al., 2023). We obtain the results of TEXTure using the official implementation and others using the implementations in threestudio. As illustrated in the Fig. 4, our method excels in generating photo-realistic texture details. Moreover, owing to our optimization based on implicit functions, the results we produce do not exhibit the seam artifacts commonly observed in TEXTure (Richardson et al., 2023). Our approach ensures both local and global consistency in the generated textures. We also conducted the user study, where 30 participants were asked and got a total of 537 responses. The result is presented in Tab. 1 (Texture Synthesis), where 57.7% of the responses prefer our results.
Here, we examine how different uses of the negative prompt can impact visual quality. As shown in Fig. 5, incorporating negative prompts can indeed enhance the visual quality. However, we observe that it may compromise the alignment with the original text prompt, i.e., the original text prompt describes the color of the clothes as blue, but the generated clothes turn out to be white. As we have illustrated, since the negative prompt serves as an additional classifier score to guide the direction
8

---

Published as a conference paper at ICLR 2024
"Bee catton hoodie with ned flower pattienaa"
CSD,
classifee
Figure 5: Ablation study negative prompts and annealed negative classifier scores.
"enice SGL hanaadd
OSLR phefe hamanid oucumber
cut DOLR phate
banana
nut mede moss, momeiaun
made flower, an wapll lighting, high cad high resoktio"
WDOER ofete shiba dethecanncht Figure 6: Demonstration of CSD in text-guided 3D editing Tasks. Our method effectively modifies attributes based on the given prompt while faithfully preserving the remaining features.
planh nurse"
policeman"
"ubii plonoel cat fartanelf cuttar hut"
of the update, it has the effect of weakening the influence of the original prompt. Fortunately, our formulation allows for dynamic adjustment of weights to balance these forces. As shown in Fig. 5, we set W1 = and reduce the weight we assigned to the negative classifier score, resulting in improved visual quality that remains aligned with the text prompt.
5.4 TEXT-GUIDED 3D EDITING
In Fig. we demonstrate that CSD can be further applied to edit the existing NeRF scenes. For each scene, based on Eq. (11), we employ the portion that requires editing as the negative prompt (e.g., "nurse"), while modifying the original prompt reflect the desired scene (e.g., "a plush toy
of corgi policeman"). We empirically set w1 1 and w2 0.5. Our approach yields satisfactory results, aligning edited objects with new prompts while maintaining other attributes.
CONCLUSION, DISCUSSION, AND LIMITATION
In this paper, introduce Classifier Score Distillation (CSD), novel framework for text-to-3I) generation that achieves state-of-the-art results across multiple tasks. Our primary objective is to demonstrate that the classifier score, often undervalued the practical implementations of SDS, may actually serve as the most essential component driving the optimization. The most significant implication of our results is prompt rethinking of what truly works in text-to-3D generation in practice. Building on the framework of CSD, we further establish connections with and provide new insights into existing techniques.
However, our work has certain limitations and opens up questions that constitute important direc- tions for future study. First, while our empirical results show that using CSD leads to superior generation quality compared to SDS, we have not yet been able to formulate objective that guides this optimization. Second, despite achieving photo-realistic results 3D tasks, find that the application of CSD to 2D image optimization results in artifacts. We hypothesize that this discrepancy may arise from the use of implicit fields and multi-view optimization strategies in 3D tasks. Investigating the underlying reasons is of significant interest in the future. Despite these limitations, we believe our findings play a significant role in advancing the understanding of text-to-3D generation and offer novel perspective for this field.
distribution-basee

---

Published as a conference paper at ICLR 2024
REFERENCES
Mohammadreza Armandpour, Huangjie Zheng. Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou, Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.0444 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miia Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324. 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 13873, 2023.
Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. Proceedings of the Neural Information Processing Systems (NeurIPS), 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects, 2022.
Yuan-Chen Guo, Ying- Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. httpp::/giihhee cow/threesteeeee ptojeetttiiippppjeee threestudio, 2023.
Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf:: Editing 3d scenes with instructions. arXiv preprint arXiv:2303.1278) 2023,
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint arXiv:2304.07090, 2023.
Jack Hessel, Ari Holzzaan, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. reference-free evaluation metric for image captioning, 2022.
Clipscore: A
Jonathan Ho and Tim Salimans. M.Xiv:2207.12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXix:1106009681 2021.
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422, 2023.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text eoxt-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 867-876, 2022.
Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Col- laborative score distillation for consistent visual synthesis. arXiv preprint 1.Xiv:2307.04787, 2023.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 34:21696-21707 2021.
10

---

Published as a conference paper at ICLR 2024
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Trans. Graph., 39(6):
194:1-194:14, 2020. doi: 10.114573414685.3477888 URL httt:t///llloog//100000 1145/ 3414685. 3417861.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pre- trained models, 2023.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12663-12673, 2023.
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13492-13502, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Gener- ating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 conference papers. pp. 1-8, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(:):1-15, 2022.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et Parallel wavenet: Fast high-fidelity speech synthesis. In International conference machine learning, pp. 3918-3926. PMLR, 2018.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fffth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 1), 2021.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint vrXiv:2209.14988 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-877 8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 22010:2200400125,, 1(2):3, 2022.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36433 2022.

---

Published as a conference paper at ICLR 2024
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Ruizhi Shao, Jingxiang Sun, Cheng Peng. Zerong Zheng, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Control4d: Dynamic portrait editing by learning 4d gan from 2d diffusion-buan editor. arXiv preprint arXiv:2305. 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetra- hedra: a hybrid representation for high-resolution 3d shape synthesis. Marc' Aurelio Ran- zato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.). Advances in Neural Information Processing Systems 34: Anmual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp- 6087-6101, 2021. URL http::::t//.......... http:///cooceeee.eeeeee.. neur 30a237d 8c5 wttpa/acccco..eeaaaaaaaa....o... (bb.:31111144440 Abstract html.
neur i pss. pccppaper/22// /hash/
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion, 2020.
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, 2021.
Ata Decpfloyd. https: :
htwgggggggfccc..ecooo/oooa httpps//auggiinggaaee
co co/DeepFloyd, 2023.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5481-5490. IEEE, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12619-12629, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. /6213, 2023b.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXhv:2302.05543, 2023.
Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, and Xin Yu. Efficientdreamer: High-fidelity and robust 3d creation via orthogonal-view diffusion prior, 2023.
Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint 8887 2023.
Jingyu Zhuang, Chen Wang. Lingjie Liu, Liang Lin, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. arXiv preprint 200iv:2306.1 134455306,,3455 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 RELATED WORKS
Text-Guided 3D Generation Using 2D Diffussin Models. In recent years, after witnessing the success of text-to-image generation, text-to-3D content generation has also made stunning progress. Early work utilizes CLIP (Radford et al., 2021) to guide text-to-3D generation (Jain et al., 2022; Mo- hammad Khalid et al.. 2022) or texture synthesize (Chen et al., 2022; Michel al., 2022). Recently, utilizing 2D diffusion models pre-trained on massive text-image pairs (Schuhmann et al., 2022) for text-guided 3D generation has become mainstream. As the pioneering work of this field, Dream- Fusion (Poole et al., 2022) proposes the Score Distillation Sampling (SDS) technique (also known as Score Jacobian Chaining (SJC) (Wang et al., 2023a)). which is able to "distill" 3D information from 2D diffusion models. SDS has been widely used and discussed in the following works (Lin ct al., 2023; Metzer et al., 2023; Chen et al., 2023; Wang et al., 2023b; Huang et al., 2023; Zhu & Zhuang, 2023; Shi et al., 2023), which attempt improve DreamFusion in many different ways. Magic3D (Lin et al., 2023) and Fantasia3D (Chen et al., 2023) investigate the possibility of opti- mizing the mesh topology instead of NeRF for efficient optimization on high-resolution renderings. They operate on 512 x 512 renderings and achieve much more realistic appearance modeling than DreamFusion which optimized on 64 64 NeRF renderings. MVDream Shi et al. (2023) allevi- ates the Janus problem of this line of methods by training and distilling multi- viel text-to-image diffusion model using renderings from synthetic 3D data. Dream'Time (Huang al., 2023) demon- strates the importance of diffusion timestep in score distillation and proposes a timestep annealing strategy to improve the generation quality. HiFA (Zhu & Zhuang. 2023) reroomuussess the SDS loss and points out that it is equivalent to the MSE loss between the rendered image and the 1-step de- noised image of the diffusion model. ProlificDreamer (Wang et al., 2023b) formulates the problem as sampling from 3D distribution and proposes Variational Score Distillation (VSD). VSD treats the 3D scene as a random variable instead of a single point as in SDS, greatly improving the genera- tion quality and diversity. However, VSD requires concurrently training another 2D diffusion model, making it suffer from long training time. 2D diffusion models can also be used for text-guided 3D editing tasks (Haque et al., 2023; Shao et al., 2023; Zhuang et al., 2023; Kim et al., 2023), and texture synthesis for 3D models (Richardson et al., 2023). In this paper, we provide a new perspec- tive on understanding the SDS optimization process and derive a new optimization strategy named Classifier Score Distillation (CSD). CSD is blug-nnd-pay replacement for SDS, which achieves better text alignment and produces more realistic appearance in 3D generation. We also show the astonishing performance of CSD in texture synthesis and its potential to perform 3D editing.
A.2 CSD FOR EDITING
Our method enables text-guided 3D editing. We observe that our formulation bears some resem- blance to the Delta Denoising Score (DDS) (Hertz et al., 2023). Here, demonstrate that these two fundamentally different formulations and discuss their relations.
Specifically, DDS is proposed for 2D image editing. Given an initial image x, initial prompt ý. and a target prompt assume that the target image to be optimized is x. It utilizes SDS to estimate two scores. The first score is calculated using VeLsDS(X, y, while the second is determined by VoLSDS (X,, 0. t). They observed that relying solely on the first score would lead to blurred outputs and loss of detail. Therefore, they decompose it into two implicit components: one is a desired direction guiding the image toward the closest match with the text, and another is an undesired component that interferes with the optimization process, causing some parts of the image to become smooth and blurry (Hertz et al., 2023). Their key insight is the belief that the score from VoLsDS (x, ÿ. t) can describe the undesired component. Thus, they perform editing by subtracting this score during optimization. Using notions, their editing objective is expressed as:
godlt = (x158,1) 85th 0, ()
(13)
Since the classifier-fre free guidance is still used here for both two SDS terms, we can look closer and formulate it as:
13

---

Published as a conference paper at ICLR 2024
"panda"
"Squirrel snowboarding"
DDS
DDS with e = 0 DDS wio &
CSD
Figure 7: Comparisons between DDS variants and CSD on 2D image editing tasks, where the panda edited to be squirell.
gdds = (xx(xi (x(: +w (xx(Xx) (e-
eo (x(; - (a:t))]----o----- es (xr;t) (14)
6ds
Note that the above gradient update rule actually consists of our classifier score 8 so it is interest- ing to see if the essential part to drive the optimization is still this component. To demonstrate this, do several modifications ablate the effect of gels on 2D image editing. As shown in Fig. 7. without a guidance weight, DDS cannot do effective optimization. Besides, we ablate the effect of only removing deh and find the optimization yet cannot be successful. Instead, solely relying on 8 can achieve superior results. Thus, we hypothesize the essential part of driving the optimization is gels, The insight here is also more straightforward than that DDS and we do not need the reference image to provide score that is considered an undesired component.
A.3 ADDITIONAL RESULTS
We provide an extensive gallery of video results the supplementary material. The gallary features comprehensive comparison between method and existing approaches. Specifically, the supplementary files include 81 results for the text-guided 3D generation task and 20 results for the text-guided texture synthesis task, along with comparisons with baseline methods. Please refer to geomet ry. html for comparisons on 3D generation and texture. html for comparisons on texture synthesis.
A.4 IMPLEMENTATION DETAILS
For the generation of NeRF in the first stage, we utilize sparsity loss and orientation loss (Verbin et al., 2022) to constrain the geometry. The weight the orientation loss linearly increases from to 100 in the first 5000 steps, while the sparsity loss linearly decreases from 10 during the same period. After 5,000 iterations, we replace the RGB outputs as normal images with a probability of 0.5 for surface refinement. The weight of the negative classifier score is gradually annealed from
14

---

Published as a conference paper at ICLR 2024
O to I as the shape progressively takes form. To mitigate the Janus problem, employ Perp-Neg (Armandpour et al., 2023) with loss weight set to 3 and additionally constrain the camera view to focus only the front and back views in the first 1,000 iterations. For prompts without clear directional objects, we omit Perp-Neg and use a larger batch size of 4. This extends training time by approximately 40 minutes. For the second stage of mesh refinement, we use normal consistency loss with a weight of 10,000 and replace the RGB outputs as normal images with a probability of 0.5 for surface refinement. The weight of the negative classifier score is annealed from 1 to 0.5. For both stages, we use negative prompts "oversaturated color, ugly, tiling, low quality. noisy". For texture generation, we optimize over 30,000 iterations. To achieve better alignment of geometry and texture, we use ControlNets guidance. Specifically, we employ canny-control and depth-control, where
applies canny edge detection the rendered normal image and uses the edge map as condition. For both conditions, we start with weight of 0.5 for the first 1000 steps, then reduce it to 0.2.
15


Reviews:
[{"review_id": "LwLayUpXiB", "reviewer": "Reviewer_oCeh", "summary": "This paper introduces a new score distillation scheme for text-to-3D generation, dubbed, Classifier Score Distillation (CSD). While the original Score Distillation Sampling (SDS) from DreamFusion subtracts random noise, CSD subtracts unconditional noise estimate (or noise estimation with negative prompts). With CSD, the author shows its effectiveness in text-to-3D generation and texture synthesis.", "questions": "- See Strengths; how the image-conditioned noise subtraction of InstructPix2Pix diffusion model in Collaborative Score Distillation paper can be related to classifier score distillation? Can Collaborative score distillation can be improved with classifier score distillation like approach?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- CSD is a simple yet effective method in transferring 2D diffusion prior to the 3D scene generation or editing. In contrast to prior state-of-the-art ProlificDreamer, it does not require fine-tuning of diffusion models, which may introduce training inefficiency and instabilities. \n\n- The qualitative and quantitative results show its effectiveness compared to prior methods. Also, this work presents a relationship between Delta Denoising Score which also used subtraction of noises in image editing tasks. I believe this is also related to the noise subtraction scheme in collaborative score distillation [https://arxiv.org/abs/2307.04787] paper, which the discussion will make the paper more complete.", "weaknesses": "- In general, I do not see a crucial weakness of this paper as it illustrates a simple method that improves the current text-to-3D generation. I believe providing detailed hyperparameter ablation study will make  the paper more informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEnrxaps1a", "reviewer": "Reviewer_8VxR", "summary": "This manuscript introduces a novel perspective on score distillation sampling (SDS). Classifier free guidance (CFG) can be interpreted as an implicit classifier based on the diffusion model that scores how much the image corresponds to the text. Empirically, SDS adds a CFG term to its gradient to ensure that the generation corresponds to the text prompt. However, by doing so, the gradients used in SDS in practice are dominated by this CFG term. This work proposes Classifier Score Distillation (CSD) which uses solely this CFG term to provide the gradients. This paper shows that CSD alone is sufficient to guide 3D generation. Furthermore, this work uses its CSD formulation to give a new interpretation of negative prompting with CFG and proposes a new negative prompting formulation that allows for explicit weights on both the positive and the negative directions.  This paper compares CSD both qualitatively and quantitatively to numerous baselines on multiple generation tasks showing SOTA performance. This work also shows CSD on editing tasks.", "questions": "- Does using CSD cause images to be less “realistic” since it removes the prior term of the loss? I.e. the generation will adhere to the text prompt very closely, but lead to an potentially unrealistic result?\n- Similarly, how is the diversity of generations using CSD for a given prompt? I would guess that there is less diversity than SDS since higher CFG weight typically reduces diversity.\n- What are the CFG weights used in the experiments section for the SDS on the baseline methods? It is specified that the DreamFusion results were obtained from its website implying a CFG of 100, but what about for the others? The default value in ThreeStudio appears to be 100 for methods using stable diffusion and 7.5 for Prolific Dreamer. Is that what was used for the experiments? If so, it might be helpful to add experiments showing existing SDS methods with very large CFG weights (i.e. 200, 500, 1000, etc.) and see how that compares to CSD.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel formulation of score distillation that gives an interesting new perspective.\n- CSD is general and can be used for any approaches using score distillation (text-to-3D, text-driven image editing) to improve results. It can also be seamlessly integrated into any existing score distillation approaches.\n- Thorough evaluation shows that CSD gives improvement over SDS both qualitatively and quantitatively.\n- The paper is well written, clearly motivating and explaining the intuition behind CSD.", "weaknesses": "Major:\n- This likely inherits the weaknesses of using a high CFG with standard SDS (I assume the following are true, but see questions for more details): less diversity of generations for a given prompt, less realistic generations, over saturated colors. [1]\n- If I understand correctly, empirically, this is not much different than using SDS with a large weight for CFG. It would be helpful to show comparisons to SDS with very large CFG weights. See questions for more details.\n\nMinor:\n- Figure 2a: It might be more clear to show both norms on the same scale. At first glance it can be confusing if you don’t notice the different scales.\n- Figure 2b: Consider including CSD here. It would be interested to see higher values for w as well since DreamFusion uses w=100.\n\nReferences: [1] Wang, Zhengyi, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. \"ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation.\" arXiv preprint arXiv:2305.16213 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LCi6qvQlJd", "reviewer": "Reviewer_pueN", "summary": "This paper presents a text-to-3D generation model by exploring classifier-free guidance in score distillation. Experiments are conducted on several text-to-3D tasks to evaluate the proposal.", "questions": "Please check the details in Weaknesses section, e.g., more clarification about implementation details and more experimental results.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "++ The main idea is simple yet effective for text-to-3D generation.\n\n++ It is good to include an in-depth discussion about SDS in section 3.\n\n++ Lots of promising qualitative results are shown to validate the effectiveness of proposal.", "weaknesses": "-- According to implementation details in section 5.1, this work uses two different pre-trained text-to-image models (DeepFloyd-IF stage-I model and Stable Diffusion 2.1). So is there any reason or ablation study for this design choice?\n\nIn addition, some baselines (like ProlificDreamer) only use the pre-trained text-to-image model of Stable Diffusion. It is somewhat no fair to compare this work with other baselines using different pre-trained models.\n\n-- The evaluation of text-guided 3D generation is performed over 81 diverse text prompts from the website of DreamFusion. However, I noticed that the website of DreamFusion (https://dreamfusion3d.github.io/gallery.html) contains lots of results (more than 81 prompts). So how to choose the 81 diverse text prompts? Any screening criteria behind?\n\nMoreover, this evaluation only uses CLIP ViT-B/32 to extract text and image features, while DreamFusion uses three models (CLIP B/32, CLIP B/16, CLIP L/14) to measure CLIP R-Precision. So following DreamFusion, it is better to report more results using more CLIP models.\n\n-- The experimental results are somewhat not convincing, since the comparison of quantitative results is inadequate and more detailed experiments/baselines should be included:\n\n1) For text-guided 3D generation, Table 2 only includes two baselines, while other strong baselines (Fantasia3D and ProlificDreamer) are missing.\n\n2) Section 5.2 only mentions the computational cost of ProlificDreamer and this work. It is better to list the computational cost of each run.\n\n3) For text-guided texture synthesis, a strong baseline [A] is missing for performance comparison. Moreover, only user study is performed for this task, and I am curious to see more quantitative comparison using the CLIP score or CLIP R-Precision.\n[A] Lei J, Zhang Y, Jia K. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition[J]. Advances in Neural Information Processing Systems, 2022, 35: 30923-30936.\n\n-- I am curious to see more results by plugging the proposed CSD into more baselines (like  DreamFusion and ProlificDreamer).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4EwFTpizR2", "reviewer": "Reviewer_FT6d", "summary": "This paper addresses the problem of text-to-3D, in which a 3D model of an object (represented by a NeRF) is produced given a text prompt describing the object. It shows that the classifier-free guidance part of the SDS loss is the main term driving the optimization of the NeRF, hence, proposing a new loss called Classifier Score Distillation (CSD). Furthermore, they also leverage the negative prompts to drive the rendered image away from low-quality region. In the experiment, the authors qualitatively show that the new CSD loss is easy to optimize as the SDS loss but bring the 3D model quality similar to the VSD (proposed in the Prolific Dreamer paper).", "questions": "1. What negative prompts did you use?\n2. How well does the CSD loss perform without the help of negative prompts. i.e., with the Eq. (7) only?\n3. Which 81 text prompts you chose to compute CLIP R-precision, why don’t you compute all the text prompts (415 text prompts) provided in the DreamFusion repo?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is easy to understand and well-written. \n2. The qualitative results are promising. \n3. The proposed loss is simple and easy to reimplement.", "weaknesses": "1. The main weakness of this paper is that its reproducibility. Since the method is simple enough that I can reimplement it in the code base of SDS loss in the threestudio framework. However, I try my best to replicate every provided detail of the results are not good as shown in the paper. They are more or less like SDS, not good as VSD loss as claimed. Therefore, it would be much better if the authors do not provide their implementation to verify during the rebuttal phase, otherwise, it greatly affects their contribution.  \n2. In our reimplementation, the Janus problem is very serious. \n3. Lack of quantitative comparison with SOTA approaches such as Prolific Dreamer, Fantasia3D....", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)