Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation
Authors: Susung Hong
Donghoon Ahn
Seungryong Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-3D
Diffusion Models

Abstract: Existing score-distilling text-to-3D generation techniques, despite their considerable promise, often encounter the view inconsistency problem. One of the most notable issues is the Janus problem, where the most canonical view of an object (\textit{e.g}., face or head) appears in other views. In this work, we explore existing frameworks for score-distilling text-to-3D generation and identify the main causes of the view inconsistency problem---the embedded bias of 2D diffusion models. Based on these findings, we propose two approaches to debias the score-distillation frameworks for view-consistent text-to-3D generation. Our first approach, called score debiasing, involves cutting off the score estimated by 2D diffusion models and gradually increasing the truncation value throughout the optimization process. Our second approach, called prompt debiasing, identifies conflicting words between user prompts and view prompts using a language model, and adjusts the discrepancy between view prompts and the viewing direction of an object. Our experimental results show that our methods improve the realism of the generated 3D objects by significantly reducing artifacts and achieve a good trade-off between faithfulness to the 2D diffusion models and 3D consistency with little overhead. Our project page is available at~\url{https://susunghong.github.io/Debiased-Score-Distillation-Sampling/}.

Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation
Susung Hong*
Donghoon Ahn"
Seungryong Kim
Korea University, Scoul, Korea
Abstract
Existing score-distiling text-to-3I) generation techniques, despite their consid- erable promise, often encounter the view inconsistency problem. One of the most notable issues is the Janus problem, where the most canonical view of an object (e-g., face or head) appears in other views. In this work, we explore existing frameworks for score-distilling text-to-3D generation and identify the main causes of the view inconsistency problem-the embedded bias of 2D dif- fusion models. Based these findings, propose approaches to debias the score- score-distillee diot-diitillation frameworks for view-consistent text-to-3D generation. Our first approach, called score debiasing, involves cutting off the score estimated by 2D diffusion models and gradually increasing the truncation value throughout the optimization process. Our second approach, called prompt debiasing, identi- fies conflicting words between user prompts and view prompts using language model, and adjusts the discrepancy between view prompts and the viewing di- rection of an object. Our experimental results show that our methods improve the realism of the generated 3D objects by significantly reducing artifacts and achieve good trade-off between faithfulness to the 2D diffusion models and 3D consistency with little overhead. Our project page available at https: // susunghhng, github. wo/.ebiaseeeeeeeeo Score Distillation SaaooonSaaplngg//aaa
Baseline
Debinsed (Ours)
Baseline
Debiased (Ours)
"a kangaroo wearing boxing gloves"
"a colorful toucan with large beak"
"a "aflamingo standing on one leg shallow
small kitte"
Figure 1: Comparison between the baseline (SJC [27] and ours (Debiased Score Distillation Sampling; D-SDS). Our debiasing methods qualitatively reduce view inconsistencies in zero-shot text-to-3D generation and the so-called Janus problem.
'Equal contribution. CCoresppnning author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Recently, significant advancements have been made in the field of zero-shot text-to-3D generation [8], particularly with the integration of score-distillation techniques [10, 27, 18, 13] and diffusion mod- els [9. 25, 24, 5, 21, 20, 23, 2] to optimize neural radiance fields [15]. These methods provide a solution for generating wide range of 3D objects from textual input, without requiring 3D supervi- sion. Despite their considerable promise, these approaches often encounter the view inconsistency problem. One of the most notable problems is the multi-face issue, also referred to the Janus problem [18], which is illustrated in the "Baseline" of Fig. 1. This problem constrains the applicability of the methods [10, 27, 18, 13], but the Janus problem rarely formulated or carefully analyzed in previous literature.
To address the problem of view inconsistency, we delve into the formulation of score-distillin text-to- 3D generation presented in [18, 27, 13, 10]. We generalize and expand upon the assumptions about the gradients concerning the parameters of 3D scene in previous works such DreamFusion [18] and Score Jacobian Chaining (SJC) [27]. and identify the main causes of the problem within the estimated score. The score can be further divided into the unconditional score and pose-prompt gradient, both of which interrupt the estimation of unbiased gradients concerning the 3D scene. Additionally, since a naive text prompt describes a canonical view of an image such as front view, prior text-to-3D generation works [22, 18, 27, 13, 10] append a view prompt (e.g., "front view", "side view", "back view", "overhead view", etc.) to the user's input, depending on the sampled camera angle, to better reflect its appearance from a different view. We present an analysis of the score with user prompts and view prompts and their effect on 3D, arguing that refining them is necessary for generating more realistic and view-consistent 3D objects.
Building on this concept and drawing inspiration from gradient clipping [14] and dynamic threshold- ing [21], we propose a score debiasing method that performs dynamic score clipping. Specifically, our method cuts off the score estimated by 2D diffusion models mitigate the impact of erroncous bias (Fig. 2 and Fig. 3). With this debiasing approach, we reduce artifacts in the generated 3D objects and alleviate the view inconsistency problem by striking a balance between faithfulness 2D models and 3D consistency. Furthermore, by gradually increasing the truncation value, which aligns with the coarse-tee to-fine nature of generating 3D objects [3, 15], we achieve a better trade-off for 3D consistency without significantly compromising faithfulness.
While the first attempt address the bias issue in the scores, further present prompt debiasing method. In contrast to prior works [18, 27, 10] that simply concatenate a view prompt and user prompt, method reduces inherent contradiction between them by leveraging a language model trained with a masked language modeling (MLM) objective [1], computing the point-wise mutual information. Additionally, we decrease the discrepancy between the assignment of the view prompt and camera pose by adjusting the range of view prompts. These enable text-to-image models [21, 20, 16] to predict accurate 2D scores, resulting in 3D objects that possess more realistic and consistent structures.
Background
Diffusion models. Denoising diffusion models [5, 23) generate images through progressive denois- ing process. During training, denoising diffusion probabilistic models (DDPM) [5] optimizes the following simplified objective:
[||e [|6--e(<e)
LOOPM :- BewNN(,.)),..
(1)
where es is a network of the diffusion model parameterized by o. t E (T,T - 1,... 1} is a timestep. Xn is an original image, and x, denotes perturbed image according to the timestep t. During inference, starting from x,, DDPM samples a previous sample x; from a normal distribution with probability density of pa (xx)
Some works fit DDPM into the generalized frameworks, e.g., non-Markovian [23], score-based [25, 9], etc. Notably, denoising diffusion models have tight relationship with score-based models [25, 9] in the continuous form. Furthermore, has been shown that denoising diffusion models can refactored into the canonical form of denoising score matching using the same network parameterization [9].

---

NeRF
Pose
Noise
Image Ae
View Prompt
Prompt Deblasing
"Back view
1. Calculate PMI for each pair
Back view
smillng
rabbit
Poggo ilpped
Score Deblasing
Pig log ((so)
Diffusion Model
Prompt Deblasing
P(Plu)
PMI(V, Earer
Score Debiasing
2. Remove tokens with PMI e threshold
"a smiling rabbit" User Prompt
Back view
smiling
rabbit
Clip
Figure 2: Illustration of our framework. We propose prompt and score debiasing techniques to estimate robust and unbiased gradients of the 3D parameters w.r.t. the viewpoints.
This formulation further facilitates the direct computation of 2D scores [24, 9] with the following equation:
Vx log 0x)
(2)
σ²
where Do is an optimal denoiser network trained for every a. With some preconditioning. a diffusion
model 6ø [5, 23, 17, 20] turns into denoiser Dø-
diffusion models have sparked increased interest in text-to-image gen- eration [19-21, 16]. Diffusion guidance techniques [2, 6, 16, 7] have been developed to enable the control of the generation process based on various conditions such as class labels [6, 2], text [7]. In particular, our work conditions text prompts with classifier-fre. guidance [6], which is formulated as follows given a conditional diffusion model tx,,
Recent advancements
captions [16], internal information
== e. (x,, t, t, + s (eg (x,, t, w) - cx(x,, t)),
(3)
where é the guided output, is the user-given text prompt (user prompt for brevity), and the guidance scale [6].
Score distillation for text-to-3D generation. Diffusion models have shown remarkable perfor- mance in text-to-image modeling [16, 19, 21, 20]. On top of this, DreamFusion [18] proposes the score-distillation sampling (SDS) method that uses text-to-image diffusion models to optimize neural fields [15], achieving encouraging results. The distillaaao sampling utilizes the gradient computed by the following equation:
Ozg 00
VgLSDS & Eo ((())
where z, denotes the t-step noised version of zy which is a rendered image from a NeRF network with parameters 0 [15]. and wtt) is scaling function only dependent on This gradient omits the Jacobian of the diffusion backbone, leading to tractable optimization in differentiable parameterizations [18].
On the other hand, in light of the interpretation of diffusion models denoisers, SIC [27] presents new approach directly using the score estimation, called perturb-and-average scoring (PAAS). The work shows that the U-Net Jacobian emerging in DreamFusion is not even necessary, well as forming strong baseline using publicly open Stable Diffusion [20]. The perturb-and-average score approximates score with inflated noise level:
V xy log P.20 (Zo) RJ (E--N( Du(zo + on;o) Za
5
where the expectation is practically estimated by Monte Carlo sampling. This score estimate is then directly plugged into the 2D-to-3D chain rule and produces:
Ozg
VoLPAAS e Ezo Vzo log 2.22 (2g)
(6)
00
3

---

Although the derivation is different from SDS in DreamFusion [18]. it is straightforward to show that the estimation VALPAAS the same as VoLsns with different weighting rule and sampler [9].
In general, frameworks distilling the score of text-to-image diffusion models [18, 10, 27, 13, 11, 22] achieve certain level of view consistency by concatenating view prompts (e-g., "hack view of")) with user prompts [18, 10, 27, 13, 11, 22J. Although this is an important part in score distillation, it is rarely diseussed. In the following section, we elucidate this altogether, uncovering the underlying causes of the Janus problem.
Score Distillation and the Janus Problem
SJC [27] defines the probability density function of parameters 0 of 3D volume (e.g., NeRF [15]) as an expectation of the likelihood of 2D rendered images Zg from uniformly sampled object-space viewpoints (Eq. 6 in [27]). Unlike this definition, our approach defines the density function of the parameters as a product of conditional likelihoods given a set of uniformly sampled viewpoints and user prompt This can be expressed as:
PAD
II ĀEΛ
PrD 20 Ro)),.);
where P2D and P3D denote the probability density of 2D image distribution and unnormalized density of 3D parametrizations, respectively. By using this formulation, we avoid using Jensen's 's inequality, in contrast to [27]. Applying the logarithm each side of the equation yields:
log (a) =
Σ log PoD X, AEA
(8)
By taking the gradient of log (sn (0), we can directly obtain Vo (og,((O)),, since the normalizing constant of paD is irrelevant to 0. Using the chain rule, we obtain:
V PoD Σ
V loggip(zol w) = Z ExEN Pogg2(222)))
AEA
(9)
Ozo
= ExEA V
Zo log PaD (zo A w)
00
where Z
|A| is constant, and Vx log Pan( w) is practically estimated by diffusion models [9]. Note that this definition generalizes SJC [27] and even VoLSDS DreamFusion[ 18), which can be easily seen the estimation of Eq. 9 with different weighting rule and sampler. This is further expanded by applying Bayes' rule as follows:
Ozo
Va log pap (0) = ExEA
1og (ZD )
+
no log P2D (A, w/ zo))
00
(10)
Uncordiitiona score
Pose prompi gradient
The first gradient term, reflecting the unconditional score modeled by 2D diffusion models [5, 25]. contains bias that affects images viewed from typical viewpoints during early optimization of 3D volume when Ze is noisy. This contributes to the Janus problem, as facial views are more prevalent in the 2D data distribution for some objects.
On the other hand, the pose-prompt gradient in Eq. 10 is guidance [25, 6, 2, 7] that drives the rendered image better represent specific camera pose and user prompt. The term is further expanded:
Vxg log PaD (A, wlzz) Vz log PeD(A))o) + Vag log (w/()) + Vz, log C,
(1))
where C' is defined Pio (Al Alzo)pmml//z) tional mutual information (PCMI). If viewpoint À and user prompt w are contradictory. i.e. Pan(Alas 2n) E Pan A (A|zo), then C approximates to O for every Simultaneously. the terms Vz, log P2D (X (0)) and Vz, log (ZD(wlzn, (w) have an adverse effect on the 3D scene, making the view- consistent optimization particularly challenging.
PAD(A w.ze) (A)o)) which represents the pointwise condi-

---

Methodology
4.1 Score debiasing
Motivation and overview. If the unconditional score, Vz, ae log (ZD( is biased towards some viewing directions, which likely in 2D data as mentioned in Sec. 3, can negatively affect the 3D consistency and realism of generated objects through the chain rule (Eq. 9). Moreover, large magnitudes in the user prompt gradient, 22, log Paw Zo), can also cause issues by introducing text-related artifacts that are not present in the image rendered from 3D field (see Fig. 1 and Fig. 3). Such artifacts include extra faces, beaks, and horns, which are unrealistic or inconsistent with the 3D object's structure.
High magnitude those two terms is typically observed when the perturbed-anc andeed---d--eeoeeedddde im- age by diffusion models significantly deviates from the rendered image in the corresponding pixels (Fig. 3). Hence, adjusting this gradient necessary to reduce the artifacts and improve the realism of the generated 3D objects. How- ever, the 2D bias that flows into the 3D field has hardly been formulated or adjusted for better optimization and 3D consistency.
The intuition behind the scale of the distilled score Vz, l.g2./2 (Za) can be mathematically elucidated by examining its relationship with the expectation term. Concretely, the distilled score serves as an approximation of the expected value of the difference between the distorted image D(z + om; σ) and the original rendered image Zo, normalized by the square of the noise scale. This expectation is evaluated with respect to the normal distribution, (000 I) from which the noise term n is sampled. Note that the noise not only facilitates the use of diffusion models but can also be interpreted as random perturbation applied the rendered image Zg.
Rendered image
2D score
(c) Generated result
Figure 3: Visualization of the magnitude of the estimated Vz, log Pog (zD(2e (zol>, w) during the opti- mization. This visualization demonstrates that erroneous 2D scores result in critical artifacts, e.g. additional legs, beaks, and horns in this figure.
In this context, the expectation term provides
measure of the sensitivity of the denoising process variations in the noise. In other words, the magnitude of the estimated score can be interpreted as the (scaled) deviation of the original rendered image 2o from the 3D field. Notably, NeRF-W [12] also provides mechanism for handling uncertainty by explicitly rendering the variance. On the contrary, we propose a novel and efficient method directly clamp the estimated score, effectively suppressing significant deviations that ignore either geometry or appearance, thereby addressing the intrinsic bias inherent in score-based models.
Dynamic clipping of 2D-to-3D scores. In light of the need to control the flow of 2D scores 3D volume (Sec. 1) and inspired by the clipping methods [14, 21J, we propose an effective method that truncates the scores mitigate the effects of bias and artifacts the predicted 2D scores:
Uz log PrD
elipped
Clip V log PaD (ze) ( w)), v/statie),
(12)
where Clip(x, c) max(min(1;; c). -c). This score clipping prevents artifacts such extra faces, horns, eyes, and cars from appearing on the 3D objects.
However, the application of naive score clipping creates large threshold-dependee tradeoff between 3D consistency and 2D fidelity: the lower the threshold, the more artifacts are removed, but at the expense of 2D fidelity. To cireumvent this, we introduce effective coarse-to-fine strategy [15, 3]:
dynamicc (1 T) Wttant + Telend
(13)
Vxg log PaD
clipped Clip V as log P2D (lo w), 3/dynamic),
5

---

where 7 (max step) In the early stages of optimization, we focus on the overall structure and shape, which do not require the large magnitudes of the 2D scores, while in later stages, we focus more on the details that require higher magnitudes. so we increase the threshold as the optimization progresses. We provide illustration in Appendix A.5 to show what the rendered image at each step looks like as the scene undergoes optimization.
(step)
4.2 Prompt debiasing
Motivation and overview. Text-to-3D generation methods that distill diffusion models [18, 27] achieve certain level of view consistency by concatenating view prompts (e.g., "back view of") with user prompts [18, 10, 27, 13, 11, 22]. This simple and effective method leverages the knowledge of large-scale text-to-image models.
However, we argue that the current strategy of creating a view-dependent prompt by simply concatenating view prompt with user prompt intrinsically problematic, as it can result in contradiction between them. This contradiction one of the causes that make diffusion models not follow the view prompt.
Therefore, in the following subsection, pro- pose identifying the contradiction between the view prompt and user prompt using off-the-shelf language models trained with masked language modeling (MLM) [1].
"Back view of dog
Additionally, instead naively assigning regu- lar regions for view prompt augmentations, in the next subsection, we reduce the discrepancy between the view prompt and object-space pose by adjusting the regions.
Identifying contradiction. The prompt gradi- ent term Vz log (w ze) may cancel out the pose gradient term Zz, log PID (X ze) needed for the view consistency of generated 3D ob- jects, as we can derive from Eq. 11. For exam- ple, if the view prompt is "back view of" and the user prompt is "a smiling dog", results in contradiction since an observer cannot the dog's 's smile viewing from the back. This causes instead follow word like "smiling" in prompt, as shown in Fig. 4.
"Back view of smillng dog"
Figure 4: Samples from Stable Diffusion [20] given a text prompt with contradiction. Despite "Back view of" is given in the prompts, the word "smiling" in the prompt makes diffusion models biased towards the front view of objects.
diffusion models not to follow a view prompt, but
In this regard, we propose a method for identifying contradictions using language models trained with masked language modeling (MLM). Specifically, let V represent set of possible view prompts, and let U be set of size 2, which contains the presence and absence a word in the user prompt for brevity. We then compute the following:
P(vlu) P(e) Ewer P(u)w))(w))
P(ulu)
(14)
where we technically model P(vlu) with masked language modeling by alternating the view prompts and normalizing them, and P(u) is a user-defined faithfulness. Note that Eq. 14 corresponds to
the pointwise mutual information (PMI), as PMI(U, u) P(r.a) = P(ela) P(u) and removing contradiction involves eliminating a word with low PMI value concerning the view prompts. In practice, a word from a user prompt is omitted if the value falls below a certain threshold.
P(e)
Reducing discrepancy between view prompts and camera poses. Existing methods [18, 27, 10, 13] utilize view prompt augmentations by dividing the camera space into some regular sections (e.g., front, back, side, and overhead in DreamFusion [18]). However, this approach does not match the real distribution of object-centric poses in image-text pairs; e.g., the front view may cover narrower

---

Average CLIP similarities (front view)
Average CLIP similarities (back view)
0.16
Baseline
0.155
Debiased (ours)
0.14
Similarity
0.150
Similarity
0.13
0.145
0.11
0.140
Baseline
0.10
Debiased (ours)
-90"
0
90
180*
-90"
0'
90*
180*
Angle
Angle
Figure 5: Average CLIP similarities of rendered images for each azimuth, calculated using view-augmented prompts. The shaded areas, starting from the left, represent the 90 regions for the front view and back view, respectively.
Method
A- I-PPPSSvG + A-L.IPSAAe: +
Baseline [27]
0,2054
0.1526
Debiased (Preserved)
0.1963
0.1450
Debiased (Ours)
0.1940
0.1445
Table 1: Quantitative evaluation. The best values are in bold, and the second best are underlined. Preserved means user prompts are preserved, i.e., P(u) = 1 for all u.
region. Therefore, make practical adjustments to the range of view prompts, such reducing the azimuth range the "front view" by half, and also search for precise view prompts [18, 27] that yield improved results.
Experiments
5.1
Implementation details
We build our debiasing methods on the nigh-performing public repository SJC [27]. For all the results, including SJC and ours, we run 10,000 steps to optimize the 3D fields, which takes about 20 minutes using single NVIDIA 3090 RTX GPU and adds almost no overhead compared to the baseline. We set the hyperparameters of SJC to specific constants [27] and do not change them throughout the experiments.
5.2 Evaluation Metrics
Quantitatively evaluating zero-shot text-to-3D framework is challenging due to the absence of ground truth 3D scenes that correspond to the text prompts. Existing works employ CLIP R- Precision [8, 18]. However, it measures retrieval accuracy through projected 2D images and text input, making it unsuitable for quantifying the view consistency of a scene.
Therefore, measure the view consistency of generated 3D objects quantitatively, we compute the average LPIPS [29] between adjacent images, which we refer to as A-LPIPS, We sample 100 uniformly spaced camera poses from upper hemisphere of fixed radius, all directed towards the sphere's 's center at an identical elevation, and render 100 images from 3D scene. Then, we average the LPIPS values evaluated for all adjacent pairs of images in the 3D scene, finally aggregating those averages across the scenes. The intuition behind this is that if there exist artifacts or view inconsistencies in generated 3D scene, the perceptual loss will be large near those points.
addition, to assess the faithfulness to the view-augmented prompt, we present graph that illustrates the average CLIP similarities of rendered images for each azimuth, determined by view-augmented prompts. This metrie is designed be high when the score-distillation pipeline effectively generates an accurate view of an object.

---

Stable DreamFusion
Baseline
Debiased (Ours)
"a baby bunny, sitting on top of stack of pancakes" "
"a majestic giraffe with long neck"
"a mug with big handle"
"a 3D modeling of a lion" "
"a tay helicopter" "
"a sleek and speedy cheetah in mid-stride"
Figure 6: Comparison between Stable-DreaFuuss [26, 18], SJC [27], and ours. The baseline is original SJC [27]. Our debiasing methods qualitatively reduce view inconsistencies in zero-shot text-to-3D and the so-called Janus problem.
Method
View consistency Faithfulness Overall quality
Baseline [27]
9.58%
16.07%
9.67%
Debiased (Ours)
90.42%
83.93%
90.33%
Table 2: User study.
5.3 Comparison with the baseline
Quantitative results. We present quantitative results from user prompts for the baseline [27], combined method, and the method without the removal of contradicting words from user prompt. Our method produces more consistent 3D objects than the baseline, as demonstrated in Table 1. Note that removing contradictions in prompts indeed leads to better results with respect to A-LPIPS. meaning that the generated objects overall each azimuth are consistent with our debiasing methods.
We also present adherence the view-augmented prompts in Fig. 5. The diagram illustrates shaded sections depicting the 90-degree front and back view zones, beginning from the left side. When comparing our unbiased outcomes to the baseline, we observe a clear and preferable pattern in CLIP

---

Baseline
Baseline
Prompt
+ Prompt
debiasing
debiasing
Score
Score
debiasing
debiasing
Figure 7: Improvement of view consistency through prompt and score debiasing. We start from the baseline (SJC (27]) and apply score debiasing and prompt debiasing sequentially for each prompt, "a smiling cat" and "a cute and chubby panda munching on bamboo", respectively.
similarities associated with the view new-augmented prompts. In this pattern, the similarity with the wwwgameeetedd prompts reaches its highest point in the desired region. In contrast, the standard method exhibits minor fluctuations in CLIP similarities as we examine different angles in relation to the view prompts, implying less faithfulness the viewing direction.
In addition the user study outlined in Table 2, which evalu- ates view consistency, faithfulness to user prompts, and overall quality, we also report the success rate the generation in Table 3. The success rate applies to 41 out of 70 prompts featuring countable faces. We marked as successful only those objects that do not exhibit the Janus problem, i.e., those with accurate number of faces. Our method significantly outperforms the baseline terms of success rate.
Success-Bass Bascline
Success-Ous
29.3%
68.3%
Table 3: Success rate.
Overall, the experiments corroborate that our debiasing methods improve the realism and alleviate the Janus problem of generated 3D objects, without requiring any 3D guide [22] or introducing significant overhead or additional optimization steps to the zero-shot text-to-3D setting.
Qualitative results. We present qualitative results in Fig. 6. In addition the results of SJC [27], which serves as the baseline for our experiments, we include those of DraleeDreamFuuion [26]. an unofficial re-implementation of DreamFusion [18] that utilizes Stable Diffusion [20]. The results demonstrate that our methods significantly reduce the Janus, or view inconsistency problem. For example, given user prompt "a majestic giraffe with long neck," the whole body is consistently generated using our debiasing method, compared to the baseline with the Janus problem. Additionally, notable example, when considering "a mug with big handle," " our method successfully generates a mug with single handle, while the counterparts generate multiple handles.
Additionally, show that method is not only applicable to SJC [27] with Stable Diffusion [20], but also to any text-to-3I frameworks that leverage score distillation, we present results on Dream- Fusion [18] with DeepFloyd-IF, and concurrent frameworks such Magic3D [10] and Prolific- Dreamer [28] in Appendix A.4, showcasing various outcomes.
5.4 Ablation study
Ablation on debiasing methods. We present ablation results in Fig. 7, where we sequentially added prompt debiasing and score debiasing on top of the baseline. This demonstrates that they gradually improve the view consistency and reduce artifacts as intended.
Using prompt debiasing alone can resolve the multi-face problem to some extent. In the case of the prompt "a smiling cat", prompt debiasing eliminates the word "smiling" from the prompt. As can be seen in column and column 3, the cat has more realistic appearance compared with the baseline. However, the cat retains an additional ear. Sometimes, such as in the instance with the panda, it can even generate new ear. Therefore, using prompt debiasing alone does not solve the problem of creating additional artifacts like ears. Applying score debiasing removes these extra ears in both cases, leading to more view-consistent text-to-3D generation in combination with prompt debiasing.
Ablation on dynamic clipping. To show some examples of the effect of dynamic clipping, compare the results with those of static clipping and no clipping in Fig. 8. It demonstrates that naive static clipping can struggle to find good compromise between 3D consistency and 2D

---

(a) No clipping
(c) Static clipping (yeseie 2.0)
Static clipping (ystuiic 8.0)
(d) Dynamic clipping (Msstar 2.0, Vend 8.0)
Figure 8: Dynamic clipping of 2D-to-3D scores. The given user prompt is "a monkey cating ramen". Using static clipping. there tough compromise between 3D consistency and 2D faithfulness. Dynamic clipping achieves a better tradeoff between pixelation and many artifacts in the result.
faithfulness, which means lowering the threshold can eliminate more artifacts like extra ears eyes to achieve better realism, but it also returns fairly pixelated and collapsed appearance, can be seen in (c). Conversely, employing dynamic clipping produces visually appealing outcomes with artifacts eliminated, closely resembling the consistency of static clipping at low threshold. Moreover, it preserves intricate shapes and details without any pixelation or degradation of the object's visual presentation.
6 Conclusion
In conclusion, we have addressed the critical issue of view inconsistency in zero-shot text-to-3D generation, particularly focusing the Janus problem. By dissecting the formulation of score- distilling text-to-31) generation and pinpointing the primary causes of the problem, dynamic score debiasing method that mitigates the impact of erroneous bias in the estimated score. This method significantly reduces artifacts and improves the 3D consistency of generated objects. Additionally, our prompt debiasing approach refines the use of user and view prompts to create more realistic and view-consistent 3D objects. Our work, D-SDS, presents a major step forward in the development of more robust and reliable zero-shot text-to-3D generation techniques, paving the way for further advancements in the field.
have proposed
Acknowledgements
This research was supported by the MSIT, Korea (ITPP-202-20200-0-01 ICT Creative Consilience program, RS-2023-00227592, Development of 3D Object Identification Technology Robust to Viewpoint Changes, No. 2021-0-00155, Context and Activity Analysis-based Solution for Safe Childcare), and National Research Foundation Korea
References
[1] Jacob Devlin, Ming-Wei Chang, Kenton I Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 180.0008 2018.
[2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 34:8780-8794, 2021.
[3] Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Rosenbaum. From to functa: Your data point is function and you should treat like one. JCML, 2022.
Yuan-Chen Guo. Ying-Tian Liu, Chen Wang, Zi-Xin Zou, Guan Luo, Chia-Hao Chen, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://githut com/ threestudio- wtteeestddooppjjee////eeo jetee//treestuuiii 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrIPS, 33:6840- 6851, 2020.
[6] Jonathan Ho and Tim Salimans. Classifier-fre diffusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
10

---

[7] Susung Hong, Gyuseong Lee. Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. arXiv preprini urX1v:2210.00939, 2022.
[8] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, pages 867-876, 2022.
[9] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. NewrlPS, 2022.
[10] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler. Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint Xiv:22211. 10440, 2022.
[11] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303. 11328, 2023.
[12] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021.
[13] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.0766 2022.
[14] Tomas Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012.
[15] Ben Mildenhall, Pratul Srinivasam, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis, Communicutions of the ACM, 65(1):99-106, 2021.
[16] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffuusion models. arXiv preprint arXiv:2112. 2021.
[17] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171. PMLR, 2021.
[18] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Drcamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988 2022.
[19] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[20] Robin Rombach, Andreas Blatmannn, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, pages 10684-10695, 2022.
[21] Chitwan Saharia, William Chan, Saurahh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. / 1487, 2022.
[22] Junyoung Sea, Wooseok Jang, Min-Seop Kwak, Jachoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.079.7 2023.
[23] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[24] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 32, 2019.
[25] Yang Song, Jascha bick-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In /CLR, 2020.
[26] Jiaxiang Tang. dtable-dreamfussonn: Text-to-3d with stable-diffusion. 2022.
com/ wttp://twww..ccmm/mas../////////
[27] Haochen Wang. Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774 2022.
11

---

[28] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.102 16213, 2023.
[29] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechuman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
12

---

Appendix
Baseline
Debiased (Ours)
"a flamingo standing on one leg shallow water"
"an elegant leacup with delicate floral designs"
"a majestic gryphon with the body of lion the wings of an eagle"
Figure 9: Results of the debiased ProlificDreamee (VSD) [28] framework. We utilize the VSD implementation, introduced in PremificDreamerr of threestudio [4]. In the baseline examples, we observe additional necks, handles, and faces. These artifacts mitigated in debiased examples.
Baseline
Debiased (Ours)
"a colorfiil toucian with large beok"
"a baby bunny, sitting on top of a stack pancakes"
Baseline
Debiased
(Ours)
playful and cuddly kitten eyes"
"a kangaroo wearing boxing gloves" "
Figure 10: Results the debiased DreamFusion [18] framework. We utilize the DreamFusion implementation of threestudio [4], which leverages DeepFloyd-IF.
A
More Results
A.1 Qualitative results
We present additional qualitative results in Figs. 12. These results clearly show that our methods alleviate the Janus problem, also known as view inconsistency.
In certain instances, even though the Janus problem is present, the images from each angle still display reasonable appearances due to smooth transitions between angles. To illustrate this, we present series of 10 sequential images arranged order the camera angles, right, back, and left of the object, in Fig. 12. In the baseline images, the front view appears in the back view or side view. However, after applying our debiasing methods, we observe significant improvement view consistency, resulting in more realistic representations.
13

---

Baseline
Debiased (Ours)
"a playful cuddly eyes"
"a codorful toncan with a large beak"
Figure 11: Results of the debiased Magic3D [10] framework. We utilize the Magic3D implementa- tion of threestudio [4].
"a majestic and powerful grizzly bear in the wild"
Baseline
Debiased (ours)
"a rabbit with long ears"
Baseline
Debiased (ours)
Comparison of method with baseline (SJC [27]) in 360.. In both cases, the baseline [27] exhibits the Janus problem, where the face appears in every view. Our debiased methods ensure proper view consistency in the 360 images.
Figure/ 12:
Furthermore, we provide another example of ablation study methods in Fig. 13. This analysis clearly demonstrates that both prompt debiasing and score debiasing techniques significantly contribute improved realism, reduction of artifacts, and achievement of view consistency.
A.2 Dynamic clipping of 2D-to-3D scores
We provide an additional example where we examine the outcomes of dynamic clipping in comparison static clipping and the absence of clipping, as shown in Fig. 14. In the case (a)), several artifacts appear in certain views. Using high threshold for static clipping yields similar outcome (row (b)). low threshold successfully removes artifacts, but also makes necessary objects, like icebergs, appear transparent (row (c)). Gradually reducing the threshold from high to low preserves the main object while eliminating artifacts (row (d)). Overall, this demonstrates that dynamic clipping reduces artifacts and enhances realism.
clipping (row
A.3
User study
We conducted user study to evaluate the view-consistency. faithfulness, and overall quality of the baseline and our debiased results. The results are presented in Table 2. According to the study, our
14

---

Baseline
Prompt debiasing
Score
debiasing
Figure 13: Improvement view consistency through prompt and score debiasing. The baseline is original SJC [27], and Prompt and Score denote prompt and score debiasing, respectively. The given user prompt "an unicorn with a rainbow horn." "
(a) No clipping
(b) Static clipping (v) staatic 8.0)
(c) Static clipping (Pastatic = 2.0)
(d) Dynamic clipping (Vstarrt 2.0, Wend 8.0)
Figure 14: Dynamic clipping of 2D-to-3D scores. The given user prompt is "a polar bear iceberg".
method surpassed the baseline in all human evaluation criteria. We tested 75 participants anonymously. and the format of instructions provided to the users was as follows:
1. Which one has more realistic 3D form? (above/below)
Which one is more consistent with the prompt? Prompt: (prompt) (above/below)
Which one has better overall quality? (above/below)
A.4 Results other text-to-3I frameworks
Our method is designed to enhance 2D score-based text-to-3D generation methods. While it has been mainly claimed to be applicable to the SJC [27] and DreamFusion [18] frameworks, the applicability of our approach extends beyond these models. This approach can be adapted for any text-to-3D
15

---

1000 step
2000 step
4000 stop
8000 slep
Resull
1000 step
2000 stop
4000 step
stop
Result
Bascline
Debiased (Ours)
Figure 15: Evaluation of rendered images during optimization. Note that the geometry of the object is mostly formed within the first 4,000 optimization steps, which is when the problem in the geometry is clearly identified.
generation method that relies on a score generated by text-to-image diffusion model and incorporates
view-augmented prompting [18, 10, 28]. These methods, including contemporary works such as Magic3D [10] and ProlificDreamer [28], are to some extent susceptible to the Janus problem. With a recent implementation of the text-to-3D frameworks, threestudio [4], we have provided results that demonstrate the applicability of our method to recent frameworks such as Magic3D (Fig. 11), DreamFusion (Fig. 10), and ProlificDreamer (Fig. 9). We use same seed a fair comparison and only apply score debiasing for this experiment. Notably, even in instances with complex geometries that are susceptible to challenges like the Janus problem (c.g., "a majestic griffon with a lion 's body and cagle's wings" "an elegant teacup with delicate floral patterns") ) the results show clear improvement when our method is applied.
A.5 Visualization optimization process and convergence speed
We present Fig. 15 demonstrate how the rendered image evolves each step during the first stage of Magic3D [10]. This experiment underscores the motivation for dynamic clipping of 2D-to-3D scores, the geometry determined in the early stages.
In addition, the 3D scenes for both the baseline and ours evolve similarly in terms of optimization steps, with ours being debiased. It indeed shows that the impact of gradient clipping convergence speed quite marginal; the number of optimization steps is comparable to that of the baseline models, and the convergence speed is nearly unchanged by approach (approximately 20 minutes for both SJC and ours).
B Limitations and Broader Impact
B.1 Limitations
Although our debiasing methods effectively tackle the Janus problem, the results produced by some prompts remain less than perfect. This is primarily due the Stable Diffusion's limited comprehension of ciewccnnditooned prompts. Despite the application of our debiasing methods, these inherent limitations result in constrained outputs for specific user prompts. Fig. 16 presents examples of such failure cases.
Baseline (SJC)
Debiased (Ours)
"a majestic tiger with piercing green eyes and bold stripes"
"a fluffy white bumry with pink ears twitchy nose
Figure 16: Failure cases. In prompts where Stable Diffusion has severely limited ability to generate view-conditioned images, the view consistency of the result is constrained.
16

---

B.2
Broader impact
Our strategy pioneers the realm of debiasing. It possesses the capability be integrated into any Score Distillation Sampling (SDS) technique currently under development, given that these methods universally deploy view prompts [18, 10, 27, 13, 11, 22].
Artificial Intelligence Generated Content (AIGC) has paved the way for numerous opportunities while simultaneously casting certain negative implications. However, it is important note that procedure not identified as having any deleterious impact since it is exclusively designed for the purpose of debiasing the existing framework.
C Implementation Details
C.1
Common settings
We base our debiasing techniques on the publicly available repository of SJC [27]. To ensure consistency, we conduct 10,000 optimization steps for both SJC and our methods to enhance the 3D fields. The hyperparameters of SJC are set to fixed values and remain unchanged throughout our experiments. For future research, we present the prompts we used in our experiments in Table 4, where some prompts are taken from DreamFusion [18]. Magic3D and SJC [27]. When comparing the use of these prompts, intentionally omit Stable Dreamfusion [26, 18] because its occasional tendency fail to generate an object and only generate backgrounds can significantly skew our evaluation metrics.
C.2 Score debiasing
In terms score debiasing, we gradually increase the truncation threshold from one-fourth of the pre-defined threshold to the pre-defined threshold, according to the optimization step. Specifically, we linearly increase the threshold from 2.0 8.0 for all experiments using Stable Diffusion [20] that leverage dynamic clipping of 2D-to-3D scores. When using Deepfloyd-IF, we adopt a linearly increasing schedule from 0.5 to 2.0, considering the lower scale of the scores.
C.3 Prompt debiasing
To compute the pointwise mutual information (PMI), we use the uncased model of BERT [1] to obtain the conditional probability. Additionally, we P(u) for words that should not be erroneously omitted. Otherwise, we set P(u) 1/2. To use a general language model for the image-related task, we concatenated "This image is depicting a" when evaluating the PMI between the view prompt and P(r.u)
user prompt. We first get u, pairs such that P(v)P(u) < 1. Then, given a view prompt, we remove words whose PMI for that view prompt, normalized across all view prompts, is below 0.95.
For the view prompt augmentation,
typically follow the view prompt assignment rule of DreamFusion [18] and SJC [27]. However, we slightly modify the view prompts and azimuth ranges for each prompt as mentioned in Sec. 4.2. For example,
assign azimuth range of 222.50, 22.5%] for the "front view." Also, we empirically find that using a view prompt augmen- tation ( roont view", "back view", "side view", "top view") without "of" depending viewpoint gives us improved results for Stable Diffusion v1.5 [20].
17

---

French fries
Medonald's dsss Iries
a. modeling
bahy bunny, sitting lop of stack of pancakes
a blue bird with a hard beak
a cherry
with pearl
a collic dog, high resolution
a croccdile
a dolphin swimming
ocean
drragon with a long tail
a DJainng standing on one log in shallow water
a frog wearing sweater
kangaroo wearing boxing gloves
monkey swinging through the trees
a mug with a big hundle
penguin
struwberry cake
0 polar bear
a polar
iceberg
rabbii with long
a beautiful mermaid with shimmering scales and flowing hair
a brightly colored tree trog
colorful porrot with curved beak and vibrant reathers
coloorful trucan with large beak
a coloorful and exotic parror with curved beak
acolorful
chumeleon
colosfal and vibrant poison dart frog perched leaf
a confddent busineswoman in a sharp suit with briefease in hand a cozy coffee mug with steam rising from the top
creepyy spider with long, spindly
a curious meerkal standing its hind legs
a curious and mischievous raccoon with striped tail a cute and chubby panda munching on bamboo
cute and cuddly sugar glider jumping from tree to tree a delicious suchi on plate
Learsooe dragon
indescent
Auffy white bunny with pink and twitchy
3 gracelul ballerina man-daace, with flowing tutu and pointed toes
8 majestic cagle
majestic elephant with hig ears and long trunk
majestic giraffe with long neek
a majestic gryphon with the body of a lion and the wings of an eagle
mijjestic lioness
majestic tiger with piercing green eyes and bold stripes
a majestic and powertul grizzly bear in the wild
a photo comfortable hed
pirale collie dog, high resolution
a playTul baby elephant spraying water with its trunk
a playlau dolphin leaping out of the water
playful cuddly kitten big eyes
q.quikky and mischievous raccoon with a striped tail
a a.refreshing and Langy grapefruit with juicy segments
a regal queen in crown elegant gown
relaxxd yogu practitioner in pose
8 ripe strawberry
a and graceful swimming deep
and speody cheetah mid-stride
a sleek and speedy cheetah racing across the savannah
a sleck and speedy falcon souring through the sky
and stcaliby panther
small kitten
a smiling cat
smeaky
helicopter
2i sports cur
an octopus in the ocean
with rainbow horn
am hitten apple
an elegant teacup with delicate floml designs
Table 4: Example prompts.
18


Reviews:
[{"review_id": "DomtbQ51pe", "reviewer": "Reviewer_8eVk", "summary": "This paper proposes debiased score sampling (D-SDS) for improving 2D diffusion-based 3D generation, targeting at the Janus problem. The method is composed of two parts: (i) score debiasing that cuts off scores from 2D diffusion and (ii) prompt debiasing that fixes the discrepancy between view prompts and object orientation. Experiments have been conducted from the baseline method SJC [Wang et al., 2023], where impressive improvements have been shown, especially in solving the Janus problem. A detailed ablation study has shown the value of each debiasing design.\n\n[Wang et al., 2023] Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation. In CVPR.", "questions": "I have no other questions besides those listed before. I am happy to increase the score if my concerns are addressed.", "limitations": "Since the method is based on SJC, it will strongly bound its performance.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work is well-motivated, targeting a relevant problem of the Janus problem in 3D generation. The discussions on the problem and technical derivations are solid, insightful, and interesting. Besides, the paper is well-written.\n- The proposed method is novel. I like the idea of debiasing scores and prompts, which makes good sense to me. Besides, the proposed method can be easily extended to other 2D diffusion-based methods.\n- Experiments are well-designed and presented. Qualitative, quantitative, and user study experiments are shown, and the results demonstrate the effectiveness of solving the Janus problem.", "weaknesses": "- Somewhat not essential. The proposed method is simple and effective. However, it seems the method is only applicable to these SJC/DreamFusion 2D diffusion score-based methods. This is not a fundamental solution for the Janus problem but is kind of like a temporary effective trick that alleviates problems.\n- Lacking complex examples. The presented results use relative prompts mainly for simple 3D objects. What about more complex 3D structures like the \"Temple of Heaven\" shown in SJC? Since one of the main difficulties of text-to-3D generation is for these more comprehensive geometries, it is important to show more of these cases. Is it another limitation of the proposed method?\n- Limited baseline. The proposed method is currently conducted to improve the SJC baseline. What about other methods? Is it possible to be applied to the more recent work ProlificDreamer [Wang et al., 2023]?\n- Minor suggestion: It seems that the Janus problem in 3D generation was first pointed out and termed for DreamFusion by Ben Poole before on [social media](https://twitter.com/poolio/status/1578045212236034048?s=20), it would be better to cite DreamFusion when first listing this issue in the paper. But it is okay if it is not cited for this term.  \n\n[Wang et al., 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. arXiv preprint.\n\n[Poole et al., 2023] DreamFusion: Text-to-3D using 2D Diffusion. In ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Applicability.** Our method is widely applicable to a variety of concurrent methods that use score distillation for text-to-3D models, including the very recent work on variational score distillation (VSD) in ProlificDreamer [Wang et al., 2023]. In general response, we have provided a PDF file including results which demonstrate the applicability of our method to recent frameworks such as Magic3D [Lin et al., 2023] (Fig. 3), DreamFusion [Poole et al., 2023] (Fig. 2), and ProlificDreamer (Fig. 1). We kindly direct the reviewer to the general response for more information.\n\n**Complex examples.** The proposed method is designed to be also applicable to complex 3D structures, not just simple ones. In addition, it should be clarified that the \"temple of heaven\" example the reviewer suggested might not be the most suitable candidate for evaluating the Janus problem due to its radially symmetrical geometry. In fact, many papers showcase this type of prompts because the results are hardly identified as encountering the problem. Rather than this, we are eager to showcase our results with complex geometry that are particularly prone to issues like the Janus problem (e.g., \"a majestic gryphon with the body of lion and the wings of an eagle'' or \"an elegant teacup with delicate floral designs'') using the very recent implementation of threestudio [Guo et al., 2023], which enables us to generate more intricate objects with the ProlificDreamer VSD framework. To demonstrate our method's effectiveness in addressing the Janus problem in complex geometries, we present generation results in Fig. 1 in the general response and compare them with baseline models.\n\n**Baselines.** While we agree that applying our method to other models such as ProlificDreamer would further demonstrate its versatility, we selected SJC [Wang et al., 2023] as our primary baseline because it was the state-of-the-art text-to-3D framework available when we submitted our work. Actually, ProlificDreamer was published on arXiv after the NeurIPS paper submission deadline, and they even mentioned the debiasing methods in their appendix as orthogonal methods that can be used in conjunction with their variational score distillation approach.\n\n**Citation.** We appreciate the reviewer's suggestion to credit Ben Poole for introducing the Janus problem in social media. We will revise the manuscript to include the citation for DreamFusion.\n\n[Poole et al., 2023] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023.\n\n[Wang et al., 2023] Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation, CVPR 2023.\n\n[Lin et al., 2023] Magic3D: High-Resolution Text-to-3D Content Creation, CVPR 2023.\n\n[Wang et al., 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, arXiv preprint.\n\n[Guo et al., 2023] threestudio: A unified framework for 3D content generation."}, {"review_id": "ehBZfq2RY4", "reviewer": "Reviewer_gQ7C", "summary": "This paper addresses the Janus problem appearing in Score-Distillation-based 3D generation methods, where the most canonical view of an object appears in other views. In particular, the author propose two components for debiasing the score distillation and the prompt used for the generation. First, the authors provide a theoretical discussion over SDS loss and the terms that contribute to the Janus problem and bad generation quality. Therefore, the authors propose gradient clipping of the score gradient in order to tackle the discussed problem. Furthermore, the paper discusses another issue of the existing methods, which is the potential contradiction between tokens of the text prompt with the added view guidance tokens. To prevent the contradiction, the authors propose to exploit the point-wise mutual information (PMI) technique to identify and remove the contradictions from the text prompt. The proposed method evaluated quantitatively and qualitatively and compared to existing baselines. In addition, an ablation study on different proposed components is provided.", "questions": "-- I am wondering how much the proposed gradient clipping affects the convergence speed of the optimization. Is there any considerable difference with the baselines regarding the optimization steps?", "limitations": "Limitations and the societal impact of the work has been discussed in the supplementary material.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- The paper is well-written.\n\n-- The provided discussion on the bias of score distillation loss and view-augmented prompts is valuable\n\n-- The novelty of the proposed components is sufficient.\n\n-- Based on the provided results, the proposed method seems effective in alleviating the Janus problem.", "weaknesses": "-- The efficacy of the proposed method is still highly limited by the bias of diffusion models. (although this is an issue specific to the proposed method).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Bias of diffusion models.** This relates to the limitations of our proposed method including other approaches, specifically regarding the bias of diffusion models. As we mentioned in our paper, if the 2D prior generates a severely deformed 3D field in the baseline, our method might not be able to fully alleviate it. However, it should be noted that our method is intended for debiasing to better distill the 2D diffusion score, not for directly incorporating an additional 3D geometry prior. Nonetheless, our method can be used universally as diffusion models and score-distilling text-to-3D frameworks evolve in the future, acting as an effective plug-and-play method. Ambitiously, we hope that our work provides a stepping stone towards mitigating these biases and combining a 3D prior with those 2D score-based models in future research.\n\n**Convergence speed.** Regarding the question about the impact of gradient clipping on convergence speed, we assure the reviewer that the convergence speed is nearly unchanged by our approach (approximately 20 minutes for SJC and ours). Our experiments showed that the number of optimization steps is comparable to that of the baseline models. With the same number of steps, we observed that the content is maintained while being debiased, with fewer artifacts (figures in the main paper and the supplementary material are always optimized with $10,000$ steps). Additionally, in the general response, we display Fig. 4 in the PDF file to show what the rendered image at each step looks like as the scene undergoes optimization. The 3D scenes for both the baseline and ours evolve similarly in terms of optimization steps, with ours being debiased. Besides, this experiment also underscores the motivation for dynamic clipping of 2D-to-3D scores, as the geometry is determined in the early stages.\n\n**Applicability.** Our method is widely applicable to a variety of concurrent methods that use score distillation for text-to-3D models, including the very recent work on variational score distillation (VSD) in ProlificDreamer [Wang et al., 2023]. In general response, we have provided a PDF file including results which demonstrate the applicability of our method to recent frameworks such as Magic3D [Lin et al., 2023] (Fig. 3), DreamFusion [Poole et al., 2023] (Fig. 2), and ProlificDreamer (Fig. 1). We kindly direct the reviewer to the general response for more information.\n\n[Poole et al., 2023] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023.\n\n[Lin et al., 2023] Magic3D: High-Resolution Text-to-3D Content Creation, CVPR 2023.\n\n[Wang et al., 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, arXiv preprint."}, {"review_id": "8Ldy49e91T", "reviewer": "Reviewer_9gp3", "summary": "This paper proposes two approaches to debias the score-distillation frameworks for view-consistent text-to-3D generation. The first approach is called score debiasing, involves cutting off the score estimated by 2D diffusion models and gradually increasing the truncation value throughout the optimization process. The second approach, called prompt debiasing, identifies conflicting words be tween user prompts and view prompts using a language model, and adjusts the discrepancy between view prompts and the viewing direction of an object. The proposed results is demonstrated by experiments.", "questions": "1. My main concern is about the evaluation. Since the experiments are only conducted with 70 prompts, could you report the success rate of generation, i.e., how many generated results of the 70 prompts does not have Janus problem? This may be more convincing than the metrics used in the paper.\n\n2. I have tried the score debiasing method in Stable-DreamFusion repo. However, based on my observation, the Janus problems are not alleviated. Is it true that the improvements is actually from Prompt Debiasing, and the improvement of Score Debiasing is quite marginal? If it is not true, could you provide an experiment solely on Score Debiasing and report the improvements of success rate of generation? (Success = Without Janus problem, Fail = With Janus problem.)\n\nI am glad to raise my score if the concerns are resolved.", "limitations": "1. Since the results are based on NeRF with latent space rendering, the quality of generated 3D results is lower than current SOTA. Could this method be applied to other 3D representations, like DMTet, which demonstrates better quality than NeRF as is shown in Magic3D? Could this method be applied to a NeRF that renders in rgb space which demontrates stronger 3D consistency?", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is clearly written.\n2. The proposed method is technically sound and insightful.\n3. Some experiments are conducted to demonstrate the idea.", "weaknesses": "1. The experiment is not sufficient, as discussed in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Success rate.** The reviewer raised a valid suggestion regarding the evaluation, especially concerning the success rate of generation. We concur that quantifying the number of generated results from the 70 prompts that do not exhibit the Janus problem would indeed provide compelling evidence. In light of this, as well as the user study in the supplementary material that evaluated the view consistency, faithfulness to user prompts, and overall quality, we have also conducted the evaluation the reviewer suggested, as presented below. The success rate applies to 41 out of 70 prompts, which feature countable faces, and we marked only those objects as successful that do not exhibit the Janus problem, i.e., those with the accurate number of faces. This experiment particularly corroborates our effectiveness in addressing the Janus problem, and we will include these results in a revised version of the paper.\n\n|               | Success-SJC | Success-Ours |\n|---------------|-------------|--------------|\n| **Success %** | 29.3%       | 68.3%        |\n\n\n**Effectiveness of score debiasing.** We acknowledge that the reviewer has experienced limited success in applying the score debiasing technique from the unofficial Stable-DreamFusion repository. It is worth noting that the effectiveness of score debiasing can vary depending on the baseline. As mentioned in our paper, if the 2D diffusion prior generates a severely deformed 3D field, our method might not fully address these issues. Nevertheless, to showcase the improvement achieved by applying score debiasing to the DreamFusion framework [Poole et al., 2023], we conducted an experiment only applying score debiasing (Fig. 2 in the general response). This experiment used the same method as proposed in DreamFusion and utilized DeepFloyd-IF, an RGB-pixel-based diffusion model similar to Imagen [Saharia et al, 2022]. It's important to note that, due to the scale differences between the RGB scores and latent scores, we adjusted our score debiasing threshold for RGB scores to one fourth of the value proposed in the paper. In the experiment with the high-performing framework, we see that the geometry issue decreases significantly when using only score debiasing.\n\n**Application on other 3D representations and NeRF in RGB space.** The method we have proposed, while tested with NeRF and latent space rendering, is fundamentally a technique applicable for a variety of text-to-3D generation frameworks using score distillation. As such, we believe it could indeed be applied to other 3D representations like DMTet or to a NeRF that renders in RGB space. To address this concern, in general response, we have provided a PDF file including results which demonstrate the applicability of our method to recent frameworks such as Magic3D (using DMTet) [Lin et al., 2023], DreamFusion (using NeRF in RGB space), and ProlificDreamer (using VSD) [Wang et al., 2023]. We kindly direct readers to the general response for more information.\n\n[Saharia et al., 2022] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, NeurIPS 2022.\n\n[Poole et al., 2023] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023.\n\n[Lin et al., 2023] Magic3D: High-Resolution Text-to-3D Content Creation, CVPR 2023.\n\n[Wang et al., 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, arXiv preprint."}, {"review_id": "jCVtkSfCy5", "reviewer": "Reviewer_nz3q", "summary": "The paper proposes two simple methods to solve the widely known Janus problem in zero-shot text-to-3D generation, which is an essential issue. The proposed methods are intuitive and simple. They are more like optimization tricks instead of sysmetic formulations. From the qualitative results, the improvement of the proposed methods is not obvious.", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)The motivation is persuasive because the Janus problem is very important in text-to-3D generation.\n2)The paper proposes two novel strategies of debiasing the score-distillation from 2D diffusion models, solving the Janus problem widely existing in zero-shot text-to-3D generation.\n3)The first strategy performs dynamic clipping of 2D-to-3D scores to eliminate the biases towards some viewing directions, which solves the problems of additional legs, beaks, and horns.\n4)The second strategy uses a pre-trained LLM to identify and remove the conflict words with the view points.", "weaknesses": "1)Technically, the novelty is a little weak.\n2)The paper proposes to debias scores and prompts of 2D text-image diffusion models. But these two debiasing methods are simply integrated for text-3D generation without elegant co-formulation.\n3)Even with sophisticated mathematical formulations, the actual debiasing score method is very intuitive and simple.\n4)The prompt debiasing method is also very simple and intuitive. What is the difference between the proposed method and using the CLIP similarity scores to remove conflict words? \n5)The qualitative comparisons in Figure 6 and 7 do not show much superiority of the proposed method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Qualitative results.** The reviewer mentioned that the qualitative comparisons in Figs. 6 and 7 do not show a clear improvement of the proposed method. However, we would like to note that in the supplementary material, we have provided more examples (Figs. 1 and 3) and images from 360-degree angles (Fig. 2) to better demonstrate the improvements in terms of the Janus problem. If the reviewer regards these comparisons with SJC [Wang et al., 2023] as unappealing, we have also provided a PDF file in the general response. This file includes the results of integrating our method into a variety of concurrent methods that use score distillation for text-to-3D models, such as Magic3D [Lin et al., 2023] (Fig. 3), DreamFusion [Poole et al., 2023] (Fig. 2), and ProlificDreamer [Wang et al., 2023] (Fig. 1), showing clear improvements in the Janus problem. We kindly direct the reviewer to the supplementary material and the general response for more information.\n\n**Novelty and simplicity.** The reviewer has expressed concerns regarding the perceived lack of novelty in our proposed methods. However, to the best of our knowledge, ours is the pioneering effort that directly addresses the Janus problem, and it is also the first to challenge the intrinsic bias present in scores and prompts within text-to-3D frameworks. Technically, we've leveraged a language model trained with an MLM objective and seamlessly integrated it into our framework for detecting contradictions, utilizing its probability output (Section 4.2). Additionally, the dynamic clipping of 2D-to-3D scores is not only deeply rooted in a coarse-to-fine strategy (Section 4.1) but has also demonstrated its effectiveness, as validated in the main paper Fig. 8, supplementary material A.2, and general response PDF Fig. 4. We believe that simplicity in research is not a detriment. When a straightforward method addresses an issue, it stands a better chance of being integrated into diverse frameworks due to its clarity, ease of implementation, and adaptability. However, we acknowledge the reviewer's concern and sincerely appreciate the thorough review.\n\n**Intuitiveness and simple integration.** We acknowledge that our two debiasing methods might seem simple and intuitive. In fact, in Section 4, we presented our methods in such an intuitive manner to clearly convey our motivation. We would like to emphasize the value of this intuitiveness and simplicity in their explanations; they are both explainable and adaptable wherever this intuition is applicable, notably in any text-to-3D framework that utilizes scores from text-to-image diffusion models. Moreover, our methodologies aim to solve separate problems stemming from a fundamental formulation in Section 3. Specifically, as discussed in Section 4.1, the score debiasing method offers a novel interpretation of the scores as outlined in Eq. 2, conceptualizing their magnitude as a (scaled) deviation from the rendered image of a 3D field. This approach directly suppresses deviations that ignore either viewpoint or geometry, addressing the intrinsic bias of score-based models detailed in Section 3. Concurrently, prompt debiasing arises from the need to reduce the contradiction between view prompts and user prompts, as detailed in Section 3. To address this effectively, we employed an MLM-pretrained model to calculate conditional probabilities, identify contradictions, and remove them from view-augmented prompts. Apparently, addressing both problems is necessary and complementary.\n\n**Prompt debiasing with CLIP.** The reviewer questions the difference between the proposed prompt debiasing method and the use of CLIP similarity scores to remove conflicting words. However, CLIP was designed for vision-language contrastive learning, not for calculating conditional probabilities between words. Therefore, it may not be straightforward to use CLIP similarity scores for this purpose. In contrast, our proposed method indeed calculates the desired conditional probability of a word's occurrence based on the formulation of masked language modeling. In fact, in the early stages of our research, we tried using CLIP similarity between word embeddings but found it unsuccessful, leading us to develop a more novel and sound method: the current prompt debiasing.\n\n[Poole et al., 2023] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023.\n\n[Wang et al., 2023] Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation, CVPR 2023.\n\n[Lin et al., 2023] Magic3D: High-Resolution Text-to-3D Content Creation, CVPR 2023.\n\n[Wang et al., 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, arXiv preprint."}]

Author rebuttal:
**General response.** We deeply appreciate the insightful feedback provided on our manuscript and have thoroughly examined all comments. The reviewers recognized the critical nature of the Janus problem in text-to-3D generation, underscoring the importance of our contribution to the field. The reviewers acknowledged the strong motivation behind our research, emphasizing the relevance of the Janus problem within text-to-3D generation (8eVk, nz3q). They also praised the robustness of our discussions, technical insights, and the manuscript's lucidity (8eVk, gQ7C, 9gp3). Our innovative methodology has been spotlighted, especially the debiasing of scores and prompts approach, which has been recognized for its originality and potential adaptability to 2D diffusion-based methods (8eVk). In addition, our comprehensive experiments, covering qualitative, quantitative, and user studies, is a testament to our solution's efficacy in tackling the Janus problem (8eVk, gQ7C).

**Applicability.** Our method is designed to enhance 2D score-based text-to-3D generation methods. While it has been mainly claimed to be applicable to the SJC [Wang et al., 2023] and DreamFusion [Poole et al., 2023] frameworks, the applicability of our approach goes beyond these models. It can be adapted for any text-to-3D generation methods that leverage a score from a text-to-image diffusion model and use view-augmented prompting, which are the dominant approaches in current text-to-3D methods using the scores of 2D diffusion models [Chen et al., 2023; Poole et al., 2023; Wang et al., 2023; Lin et al., 2023; Seo et al., 2023]. These methods, including Magic3D [Lin et al., 2023] and ProlificDreamer [Wang et al., 2023], are to some extent susceptible to the Janus problem. We believe this context renders our work a rather novel solution that tackles the prevailing Janus problem. Thanks to the recent implementation of the text-to-3D frameworks [Guo et al., 2023], we have provided a PDF file including results that demonstrate the applicability of our method to recent frameworks such as Magic3D (Fig. 3), DreamFusion (Fig. 2), and ProlificDreamer (Fig. 1). We hope that concerns about the applicability are resolved when looking at the results in the file.

[Poole et al., 2023] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023.

[Wang et al., 2023] Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation, CVPR 2023.

[Lin et al., 2023] Magic3D: High-Resolution Text-to-3D Content Creation, CVPR 2023.

[Wang et al., 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, arXiv preprint.

[Seo et al., 2023] Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation, arXiv preprint.

[Chen et al., 2023] Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation, ICCV 2023.

[Guo et al., 2023] threestudio: A unified framework for 3D content generation.

Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)