SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	A-LPIPSVGG ↓	A-LPIPSAlex ↓
Baseline [27]	0.2054	0.1526
Debiased (Preserved)	0.1963	0.1450
Debiased (Ours)	0.1940	0.1445
Table 1: Quantitative evaluation. The best values are in bold, and the second best are underlined. Preserved means user prompts are preserved, i.e.,  $P(u) = 1$  for all  $u$ .
region. Therefore, we make practical adjustments to the range of view prompts, such as reducing the azimuth range of the "front view" by half, and also search for precise view prompts [18, 27] that yield improved results.
5.1 Implementation details
We build our debiasing methods on the high-performing public repository of SJC [27]. For all the results, including SJC and ours, we run 10,000 steps to optimize the 3D fields, which takes about 20 minutes using a single NVIDIA 3090 RTX GPU and adds almost no overhead compared to the baseline. We set the hyperparameters of SJC to specific constants [27] and do not change them throughout the experiments.
Quantitatively evaluating a zero-shot text-to-3D framework is challenging due to the absence of ground truth 3D scenes that correspond to the text prompts. Existing works employ CLIP R-Precision [8, 18]. However, it measures retrieval accuracy through projected 2D images and text input, making it unsuitable for quantifying the view consistency of a scene.
Therefore, to measure the view consistency of generated 3D objects quantitatively, we compute the average LPIPS [29] between adjacent images, which we refer to as A-LPIPS. We sample 100 uniformly spaced camera poses from an upper hemisphere of a fixed radius, all directed towards the sphere's center at an identical elevation, and render 100 images from a 3D scene. Then, we average the LPIPS values evaluated for all adjacent pairs of images in the 3D scene, finally aggregating those averages across the scenes. The intuition behind this is that if there exist artifacts or view inconsistencies in a generated 3D scene, the perceptual loss will be large near those points.
In addition, to assess the faithfulness to the view-augmented prompt, we present a graph that illustrates the average CLIP similarities of rendered images for each azimuth, as determined by view-augmented prompts. This metric is designed to be high when the score-distillation pipeline effectively generates an accurate view of an object.
"a baby bunny, sitting on top of a stack of pancakes"
"amajestic giraffe withalong neck
"amugwithabighandle
" a 3D modeling of a lion
"a toy helicopter"
" a sleek and speedy cheetah in mid-stride
Figure 6: Comparison between Stable-DreamFusion [26, 18], SJC [27], and ours. The baseline is original SJC [27]. Our debiasing methods qualitatively reduce view inconsistencies in zero-shot text-to-3D and the so-called Janus problem.
Method	View consistency	Faithfulness	Overall quality
Baseline [27]	9.58%	16.07%	9.67%
Debiased (Ours)	90.42%	83.93%	90.33%
Table 2: User study.
5.3 Comparison with the baseline
Quantitative results. We present quantitative results from 70 user prompts for the baseline [27], our combined method, and the method without the removal of contradicting words from a user prompt. Our method produces more consistent 3D objects than the baseline, as demonstrated in Table 1. Note that removing contradictions in prompts indeed leads to better results with respect to A-LPIPS, meaning that the generated objects overall in each azimuth are consistent with our debiasing methods.
We also present adherence to the view-augmented prompts in Fig. 5. The diagram illustrates shaded sections depicting the 90-degree front and back view zones, beginning from the left side. When comparing our unbiased outcomes to the baseline, we observe a clear and preferable pattern in CLIP
Figure 7: Improvement of view consistency through prompt and score debiasing. We start from the baseline (SJC [27]) and apply score debiasing and prompt debiasing sequentially for each prompt, "a smiling cat" and "a cute and chubby panda munching on bamboo", respectively.
similarities associated with the view-augmented prompts. In this pattern, the similarity with the view-augmented prompts reaches its highest point in the desired region. In contrast, the standard method exhibits minor fluctuations in CLIP similarities as we examine different angles in relation to the view prompts, implying less faithfulness to the viewing direction.
In addition to the user study outlined in Table 2, which evaluates view consistency, faithfulness to user prompts, and overall quality, we also report the success rate of the generation in Table 3. The success rate applies to 41 out of 70 prompts featuring countable faces. We marked as successful only those
Success-Baseline	Success-Ours
29.3%	68.3%
Table 3: Success rate.
objects that do not exhibit the Janus problem, i.e., those with an accurate number of faces. Our method significantly outperforms the baseline in terms of success rate.
Overall, the experiments corroborate that our debiasing methods improve the realism and alleviate the Janus problem of generated 3D objects, without requiring any 3D guide [22] or introducing significant overhead or additional optimization steps to the zero-shot text-to-3D setting.
Qualitative results. We present qualitative results in Fig. 6. In addition to the results of SJC [27], which serves as the baseline for our experiments, we include those of Stable-DreamFusion [26], an unofficial re-implementation of DreamFusion [18] that utilizes Stable Diffusion [20]. The results demonstrate that our methods significantly reduce the Janus, or view inconsistency problem. For example, given a user prompt "a majestic giraffe with a long neck," the whole body is consistently generated using our debiasing method, compared to the baseline with the Janus problem. Additionally, as a notable example, when considering "a mug with a big handle," our method successfully generates a mug with a single handle, while the counterparts generate multiple handles.
Additionally, to show that our method is not only applicable to SJC [27] with Stable Diffusion [20], but also to any text-to-3D frameworks that leverage score distillation, we present results on Dream-Fusion [18] with DeepFloyd-IF, and on concurrent frameworks such as Magic3D [10] and Prolific-Dreamer [28] in Appendix A.4, showcasing various outcomes.
5.4 Ablation study
Ablation on debiasing methods. We present ablation results in Fig. 7, where we sequentially added prompt debiasing and score debiasing on top of the baseline. This demonstrates that they gradually improve the view consistency and reduce artifacts as intended.
Using prompt debiasing alone can resolve the multi-face problem to some extent. In the case of the prompt "a smiling cat", prompt debiasing eliminates the word "smiling" from the prompt. As can be seen in column 1 and column 3, the cat has a more realistic appearance compared with the baseline. However, the cat retains an additional ear. Sometimes, such as in the instance with the panda, it can even generate a new ear. Therefore, using prompt debiasing alone does not solve the problem of creating additional artifacts like ears. Applying score debiasing removes these extra ears in both cases, leading to more view-consistent text-to-3D generation in combination with prompt debiasing.
Ablation on dynamic clipping. To show some examples of the effect of dynamic clipping, we compare the results with those of static clipping and no clipping in Fig. 8. It demonstrates that naive static clipping can struggle to find a good compromise between 3D consistency and 2D
(a) No clipping
(b) Static clipping  $(\psi_{\mathrm{static}} = 8.0)$
(c) Static clipping  $(\psi_{\mathrm{static}} = 2.0)$
(d) Dynamic clipping  $(\psi_{\mathrm{start}} = 2.0, \psi_{\mathrm{end}} = 8.0)$
Figure 8: Dynamic clipping of 2D-to-3D scores. The given user prompt is "a monkey eating ramen". Using static clipping, there is a tough compromise between 3D consistency and 2D faithfulness. Dynamic clipping achieves a better tradeoff between pixelation and many artifacts in the result.
faithfulness, which means lowering the threshold can eliminate more artifacts like extra ears or eyes to achieve better realism, but it also returns fairly pixelated and collapsed appearance, as can be seen in (c). Conversely, employing dynamic clipping produces visually appealing outcomes with artifacts eliminated, closely resembling the consistency of static clipping at a low threshold. Moreover, it preserves intricate shapes and details without any pixelation or degradation of the object's visual presentation.
In conclusion, we have addressed the critical issue of view inconsistency in zero-shot text-to-3D generation, particularly focusing on the Janus problem. By dissecting the formulation of score-distilling text-to-3D generation and pinpointing the primary causes of the problem, we have proposed a dynamic score debiasing method that mitigates the impact of erroneous bias in the estimated score. This method significantly reduces artifacts and improves the 3D consistency of generated objects. Additionally, our prompt debiasing approach refines the use of user and view prompts to create more realistic and view-consistent 3D objects. Our work, D-SDS, presents a major step forward in the development of more robust and reliable zero-shot text-to-3D generation techniques, paving the way for further advancements in the field.
This research was supported by the MSIT, Korea (IITP-2022-2020-0-01819, ICT Creative Consilience program, RS-2023-00227592, Development of 3D Object Identification Technology Robust to Viewpoint Changes, No. 2021-0-00155, Context and Activity Analysis-based Solution for Safe Childcare), and National Research Foundation of Korea (NRF-2021R1C1C1006897).
[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8794, 2021.
[3] Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you should treat it like one. ICML, 2022.
[4] Yuan-Chen Guo, Ying-Tian Liu, Chen Wang, Zi-Xin Zou, Guan Luo, Chia-Hao Chen, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/threestudio, 2023.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
[6] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[28] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific-dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213, 2023.
[29] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
Debiased (Ours)
"aflamingo standing on one leg in shallow water"
"an elegant teacup with delicate floral designs"
"amajestic gryphon with the body of a lion and the wings of an eagle
Figure 9: Results of the debiased ProlificDreamer (VSD) [28] framework. We utilize the VSD implementation, introduced in ProlificDreamer, of threestudio [4]. In the baseline examples, we observe additional necks, handles, and faces. These artifacts are mitigated in our debiased examples.
"acolorful toucan with a large beak"
"ababy bunny, sitting on top of a stack of pancakes
"aplyful and cuddly kitten with big eyes
Figure 10: Results of the debiased DreamFusion [18] framework. We utilize the DreamFusion implementation of threestudio [4], which leverages DeepFloyd-IF.
"akangaroo wearing boxing gloves
A More Results
A.1 Qualitative results
We present additional qualitative results in Figs. 12. These results clearly show that our methods alleviate the Janus problem, also known as view inconsistency.
In certain instances, even though the Janus problem is present, the images from each angle still display reasonable appearances due to smooth transitions between angles. To illustrate this, we present a series of 10 sequential images arranged in order of the camera angles, right, back, and left of the object, in Fig. 12. In the baseline images, the front view appears in the back view or side view. However, after applying our debiasing methods, we observe a significant improvement in view consistency, resulting in more realistic representations.
Figure 11: Results of the debiased Magic3D [10] framework. We utilize the Magic3D implementation of threestudio [4].
"amajestic and powerful grizzly bear in the wild
"arabbit with long ears
Figure 12: Comparison of our method with baseline (SJC [27]) in  $360^{\circ}$ . In both cases, the baseline [27] exhibits the Janus problem, where the face appears in every view. Our debiased methods ensure proper view consistency in the  $360^{\circ}$  images.
Furthermore, we provide another example of ablation study on our methods in Fig. 13. This analysis clearly demonstrates that both prompt debiasing and score debiasing techniques significantly contribute to improved realism, reduction of artifacts, and achievement of view consistency.
A.2 Dynamic clipping of 2D-to-3D scores
We provide an additional example where we examine the outcomes of dynamic clipping in comparison to static clipping and the absence of clipping, as shown in Fig. 14. In the case of no clipping (row (a)), several artifacts appear in certain views. Using a high threshold for static clipping yields a similar outcome (row (b)). A low threshold successfully removes artifacts, but also makes necessary objects, like icebergs, appear transparent (row (c)). Gradually reducing the threshold from high to low preserves the main object while eliminating artifacts (row (d)). Overall, this demonstrates that dynamic clipping reduces artifacts and enhances realism.
A.3 User study
We conducted a user study to evaluate the view-consistency, faithfulness, and overall quality of the baseline and our debiased results. The results are presented in Table 2. According to the study, our
Figure 13: Improvement of view consistency through prompt and score debiasing. The baseline is original SJC [27], and Prompt and Score denote prompt and score debiasing, respectively. The given user prompt is "an unicorn with a rainbow horn."
(a) No clipping
(b) Static clipping  $(\psi_{\mathrm{static}} = 8.0)$
(c) Static clipping  $(\psi_{\mathrm{static}} = 2.0)$
(d) Dynamic clipping  $(\psi_{\mathrm{start}} = 2.0, \psi_{\mathrm{end}} = 8.0)$
Figure 14: Dynamic clipping of 2D-to-3D scores. The given user prompt is "a polar bear on an iceberg".
method surpassed the baseline in all human evaluation criteria. We tested 75 participants anonymously, and the format of instructions provided to the users was as follows:
1. Which one has a more realistic 3D form? (above/below)
2. Which one is more consistent with the prompt? Prompt: {prompt} (above/below)
3. Which one has better overall quality? (above/below)
A.4 Results on other text-to-3D frameworks
Our method is designed to enhance 2D score-based text-to-3D generation methods. While it has been mainly claimed to be applicable to the SJC [27] and DreamFusion [18] frameworks, the applicability of our approach extends beyond these models. This approach can be adapted for any text-to-3D
Figure 15: Evaluation of rendered images during optimization. Note that the geometry of the object is mostly formed within the first 4,000 optimization steps, which is when the problem in the geometry is clearly identified.
generation method that relies on a score generated by a text-to-image diffusion model and incorporates view-augmented prompting [18, 10, 28]. These methods, including contemporary works such as Magic3D [10] and ProlificDreamer [28], are to some extent susceptible to the Janus problem. With a recent implementation of the text-to-3D frameworks, threestudio [4], we have provided results that demonstrate the applicability of our method to recent frameworks such as Magic3D (Fig. 11), DreamFusion (Fig. 10), and ProlificDreamer (Fig. 9). We use the same seed for a fair comparison and only apply score debiasing for this experiment. Notably, even in instances with complex geometries that are susceptible to challenges like the Janus problem (e.g., "a majestic griffon with a lion's body and eagle's wings" or "an elegant teacup with delicate floral patterns"), the results show clear improvement when our method is applied.
A.5 Visualization of optimization process and convergence speed
We present Fig. 15 to demonstrate how the rendered image evolves at each step during the first stage of Magic3D [10]. This experiment underscores the motivation for dynamic clipping of 2D-to-3D scores, as the geometry is determined in the early stages.
In addition, the 3D scenes for both the baseline and ours evolve similarly in terms of optimization steps, with ours being debiased. It indeed shows that the impact of gradient clipping on convergence speed is quite marginal; the number of optimization steps is comparable to that of the baseline models, and the convergence speed is nearly unchanged by our approach (approximately 20 minutes for both SJC and ours).
B Limitations and Broader Impact
B.1 Limitations
Although our debiasing methods effectively tackle the Janus problem, the results produced by some prompts remain less than perfect. This is primarily due to the Stable Diffusion's limited comprehension of view-conditioned prompts. Despite the application of our debiasing methods, these inherent limitations result in constrained outputs for specific user prompts. Fig. 16 presents examples of such failure cases.
"a fluffy white bunny with pink ears and a twitchy nose"
Figure 16: Failure cases. In some prompts where Stable Diffusion has a severely limited ability to generate view-conditioned images, the view consistency of the result is constrained.
B.2 Broader impact
Our strategy pioneers the realm of debiasing. It possesses the capability to be integrated into any Score Distillation Sampling (SDS) technique currently under development, given that these methods universally deploy view prompts [18, 10, 27, 13, 11, 22].
Artificial Intelligence Generated Content (AIGC) has paved the way for numerous opportunities while simultaneously casting certain negative implications. However, it is important to note that our procedure is not identified as having any deleterious impact since it is exclusively designed for the purpose of debiasing the existing framework.
C.1 Common settings
We base our debiasing techniques on the publicly available repository of SJC [27]. To ensure consistency, we conduct 10,000 optimization steps for both SJC and our methods to enhance the 3D fields. The hyperparameters of SJC are set to fixed values and remain unchanged throughout our experiments. For future research, we present the prompts we used in our experiments in Table 4, where some prompts are taken from DreamFusion [18], Magic3D and SJC [27]. When comparing the use of these prompts, we intentionally omit Stable Dreamfusion [26, 18] because its occasional tendency to fail to generate an object and only generate backgrounds can significantly skew our evaluation metrics.
C.2 Score debiasing
In terms of score debiasing, we gradually increase the truncation threshold from one-fourth of the pre-defined threshold to the pre-defined threshold, according to the optimization step. Specifically, we linearly increase the threshold from 2.0 to 8.0 for all experiments using Stable Diffusion [20] that leverage dynamic clipping of 2D-to-3D scores. When using Deepfloyd-IF, we adopt a linearly increasing schedule from 0.5 to 2.0, considering the lower scale of the scores.
C.3 Prompt debiasing
To compute the pointwise mutual information (PMI), we use the uncased model of BERT [1] to obtain the conditional probability. Additionally, we set  $P(u) = 1$  for words that should not be erroneously omitted. Otherwise, we set  $P(u) = 1/2$ . To use a general language model for the image-related task, we concatenated "This image is depicting a" when evaluating the PMI between the view prompt and user prompt. We first get  $u, v$  pairs such that  $\frac{P(v, u)}{P(v)P(u)} < 1$ . Then, given a view prompt, we remove words whose PMI for that view prompt, normalized across all view prompts, is below 0.95.
For the view prompt augmentation, we typically follow the view prompt assignment rule of DreamFusion [18] and SJC [27]. However, we slightly modify the view prompts and azimuth ranges for each prompt as mentioned in Sec. 4.2. For example, we assign an azimuth range of  $[-22.5^{\circ}, 22.5^{\circ}]$  for the "front view." Also, we empirically find that using a view prompt augmentation  $v \in \{“front view”, “back view”, “side view”, “top view” \}$  without "of" depending on a viewpoint gives us improved results for Stable Diffusion v1.5 [20].
French fries
Mcdonald's fries
a 3D modeling of a lion
a baby bunny, sitting on top of a stack of pancakes
a blue bird with a hard beak
a cherry
a clam with pearl
a collie dog, high resolution
a crocodile
a dolphin swimming in the ocean
a dragon with a long tail
a flamingo standing on one leg in shallow water
a frog wearing a sweater
a kangaroo wearing boxing gloves
a monkey swinging through the trees
a mug with a big handle
a penguin
a piece of strawberry cake
a polar bear
a polar bear on an iceberg
a rabbit with long ears
a beautiful mermaid with shimmering scales and flowing hair
a brightly colored tree frog
a colorful parrot with a curved beak and vibrant feathers
a colorful toucan with a large beak
a colorful and exotic parrot with a curved beak
a colorful and vibrant chameleon
a colorful and vibrant poison dart frog perched on a leaf
a confident businesswoman in a sharp suit with briefcase in hand
a cozy coffee mug with steam rising from the top
a creepy spider with long, spindly legs
a curious meerkat standing on its hind legs
a curious and mischievous raccoon with a striped tail
a cute and chubby panda munching on bamboo
a cute and cuddly sugar glider jumping from tree to tree
a delicious sushi on a plate
a fearsome dragon with iridescent scales
a fluffy white bunny with pink ears and a twitchy nose
a graceful ballerina mid-dance, with flowing tutu and pointed toes
a majestic eagle
a majestic elephant with big ears and a long trunk
a majestic giraffe with a long neck
a majestic Gryphon with the body of a lion and the wings of an eagle
a majestic lioness
a majestic tiger with piercing green eyes and bold stripes
a majestic and powerful grizzly bear in the wild
a photo of a comfortable bed
a pirate collie dog, high resolution
a playful baby elephant spraying water with its trunk
a playful dolphin leaping out of the water
a playful and cuddly kitten with big eyes
a quirky and mischievous raccoon with a striped tail
a refreshing and tangy grapefruit with juicy segments
a regal queen in a crown and elegant gown
a relaxed yoga practitioner in a serene pose
a ripe strawberry
a sleek and graceful shark swimming in the deep sea
a sleek and speedy cheetah in mid-stride
a sleek and speedy cheetah racing across the savannah
a sleek and speedy falcon soaring through the sky
a sleek and stealthy panther
a small kitten
a smiling cat
a sneaky fox
a toy helicopter
a toy sports car
an octopus in the ocean
an unicorn with a rainbow horn
an bitten apple
an elegant teacup with delicate floral designs
Table 4: Example prompts.