LCGen: Mining in Low-Certainty Generation for View-consistent Text-to-3D
Authors: Zeng Tao
Tong Yang
Junxiong Lin
Xinji Mai
Haoran Wang
Beining Wang
Enyu Zhou
Yan Wang
Wenqiang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-3D
Janus Problem
Certainty
Diffusion
NeRF
Primary area: generative_models

Abstract: The Janus Problem is a common issue in SDS-based text-to-3D methods. Due to view encoding approach and 2D diffusion prior guidance, the 3D representation model tends to learn content with higher certainty from each perspective, leading to view inconsistency. In this work, we first model and analyze the problem, visualizing the specific causes of the Janus Problem, which are associated with discrete view encoding and shared priors in 2D lifting. Based on this, we further propose the LCGen method, which guides text-to-3D to obtain different priors with different certainty from various viewpoints, aiding in view-consistent generation. Experiments have proven that our LCGen method can be directly applied to different SDS-based text-to-3D methods, alleviating the Janus Problem without introducing additional information, increasing excessive training burden, or compromising the generation effect.

LCGen: Mining in Low-Certainty Generation for View-consistent Text-to-3D
Zeng Tao', Tong Yang", Junxiong Lin', Xinji Mai', Haoran Wang', Beining Wang', Enyu Zhou', Yan Wang f Wenqiang Zhang*
Shanghai Engineering Research Center of AI & Robotics, Academy for Engineering & Technology, Fudan University. Shanghai, China
Shanghai Key Lab of Intelligent Information Processing, School of Computer Science, Fudan University, Shanghai, China
Schooo of Computer Science, Fudan University, Shanghai, China EEngineering Research Center of AI & Robotics, Ministry of Education, Academy for Engineering Technology, Fudan University, Shanghai, China {ztao19, yanwang19 xqzhang) @fudan edu.
Janus Problem
No Janus Problem
PBF
azimuth
3atmuuh
Gradient of Lsps
clevation
elesstinn
Gradient of Ceert
cerbiisty
High Certainty
Low Certainty
Distribution
L.Gen
Sample
'A corgi"
elevation, seimuth
Figure 1: The main cause of the Janus Problem in SDS-based text-to-3D methods is their reliance on high-certainty 2D priors for representation, which can result in heads appearing from multiple viewpoints. To address this, we introduced the LCGen method, using low-certainty generation to align viewpoints with the optimization direction.
Abstract
The Janus Problem is common issue in SDS-based text-to-3D methods. Due view encoding approach and 2D diffusion prior guidance, the 3D representation model tends to learn content with higher certainty from each perspective, leading to view inconsistency. In this work, we first model and analyze the problem, visu- alizing the specific causes of the Janus Problem, which are associated with discrete view encoding and shared priors in 2D lifting. Based on this, we further propose the LCGen method, which guides text-to-3D obtain different priors with different certainty from various viewpoints, aiding in view-consistent generation. Exper- iments have proven that LCGen method be directly applied different SDS- based text-to-3D methods, alleviating the Janus Problem without introducing
"Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

additional information, increasing excessive training burden, or compromising the generation effect. Project page is [here]
1
Introduction
At the forefront of the digital domain, considerable advancements have been achieved in converting
textual prompts into realistic 2D images, signaling a new epoch of computational creativity [14
[
4
3 2
36
38 39, 30 31
22 17 18 However, transferring such achievements to the 3D domain
introduces additional layer of intricacy. While 3D generation technology [5] 29
g 24 32 is
becoming increasingly indispensable across various fields, from virtual reality to architectural design, traditional 3D content generation demands a substantial investment of time and expertise, and the difficulty in acquiring 3D data makes explicit text-to-3D modeling exceedingly challenging. In response, Score Distillation Sampling (SDS) 25 based on 2D lifting, has emerged to simplify and advance the 3D creative process 6 16
In text-to-3D tasks, 3D representation (such as NeRF (24) ) renders and outputs the corresponding visual image of given camera viewpoints. After that, SDS employs the guidance from priors embedded within pre-trained cext-conditioned 2D diffusion models to compute losses for images or latents, iteratively guiding the 3D representation to its optimality.
23
However, this paradigm carries certain risks, with the Janus Problem 25 standing out as a significant and common issue [33] 37 [4)]. This problem occurs when 3D models exhibit multiple, often conflicting viewpoints, resulting inconsistencies with the original text conditions. As illustrated in Fig. 1 faces appear at various positions within the same 3D object. This is an inevitable result stemming from the inherent nature of SDS-based text-to-3D approaches. Firstly, current based text-to-3D methods employ discrete viewpoint encoding. Camera perspectives are classified into regions, with each region sharing a uniform view and text guidance. Consequently, the images within each region share the same prior distribution, biasing the 3D representation towards locally optimal synthesis with the highest certainty, as shown in and Sec. 3 From global perspective, there is high probability of bias towards synthesizing heads at multiple biased positions. Secondly, diffusion models lack diverse 3D training data and thus a nuanced understanding of 3D space [19]. For different camera views, there is tendency to generate images with high certainty that emphasize the most characteristic features of the object (TT), such as the head [. These intrinsic shortcomings render the Janus Problem dominant challenge in the text-to-3D process.
Addressing Janus Problem remains a critical challenge. with numerous studies dedicated mitigating its effects [15] 21 (2 40 [2], like DreamControl [3] and Perp-Neg M. However, these methods either require extensive multi-stage fine-tuning or object-specific designs and do not address the fundamental causes of the Janus Problem. In response, we have innovatively modeled and analyzed the underlying causes of the text-to-3D Janus Problem and proposed a novel approach named Low Certainty Generation (LCGen) method.
Specifically, we first analyzed the causes of the Janus Problem through probability modeling. Under the paradigm of discrete viewpoint encoding, viewpoint inputs from the same region possess the same prior distribution. We modeled this distribution and analyzed the relationship between the occurrence of the Janus Problem at various biased positions and the distribution itself. By calculating probabilities, we derived the likelihood of biases occurring at specific positions, as illustrated in Fig. [ showing probability density peaks at different biases, indicating the Janus Problem.
Thus, addressing the text-to-3I Janus Problem necessitates reevaluating the relationship between views and distributions of guidance, advocating for their decoupling rather than adherence to a shared distribution. Direct explicit modeling of the distribution quite challenging, so we sought implicit distribution constraints based on certainty learning. Here, we define the probability C(x,-1/2;) in Eq. 7 estimated at the current timestep, as generation certainty in diffusion. We discovered that different viewpoint images exhibit varying levels of certainty during the diffusion denoising process. For areas rich in object features (such as the front), diffusion model tends exhibit higher certainty during the denoising process, and conversely lower elsewhere. This is linked to the data bias in pretrained diffusion models [3 (see Appendix. B). Leveraging this characteristic, we constrain the generation of 3D representations so that their input x, into diffusion model achieves certainty consistent with the viewpoint. Consequently, we obtain a decoupled data distribution for precise distribution localization of separated viewpoints, thus releasing the Janus Problem. Extensive data
2

---

analysis and visualization substantiate the scientific validity and effectiveness of LCGen. Our method can be integrated into various SDS-based text-to-3D methods, consistently mitigating the Janus Problem without compromising generative performance.
Our contributions as follows:
We model and analyze the Janus Problem in text-to-3D, identifying the fundamental reasons for its occurrence. Our findings indicate that the inevitability of the Janus Problem is associated with the SDS-based text-to-3D framework that employs discrete view encoding and 2D diffusion lifting.
We specifically develop LCGen, a method that decouples the distribution of viewpoint data from the perspective of generation certainty, thereby guiding precise view localization and effectively mitigating the Janus Problem.
We conduct extensive data analysis and experiments to demonstrate the scientific validity and effectiveness of LCGen. Our method is transferable and consistently mitigates the Janus Problem across various baselines without compromising the generative quality.
2 Background
2.1 Diffusion model
Diffusion models [10] [27] have proven a powerful class of generative models, particularly excelling in text-to-image synthesis. Building on the progress made with text-to-image diffusion models, new approaches have been developed to extend these capabilities to 3D content generation.
A diffusion model typically involves a forward process that gradually adds noise data sample and a reverse process that aims to reconstruct the original sample by progressively denoising it. Mathematically, the forward process is described as:
q( xx-1) = N( V1 B, 3,1).
where N is Gaussian distribution, 3r are variance terms increasing over time, and I is the identity matrix. The reverse process, which more pertinent to generative tasks, is modeled as:
po ( x+) uz 1), I,
where no (I,, f) represents the mean learned by the diffusion model Φ, and of are learned variances.
2.2 SDS-based text-to-3D
Score Distillation Sampling (SDS). Score Distillation Sampling (SDS) (25] is novel approach tailored for bridging the gap between 2D image generation and 3D model synthesis. The SDS process utilizes the gradients from pretrained 2D diffusion model & guide the generation of 3D model O. The key idea is to render 2D projections of a 3D model from various views and adjust the model parameters to maximize the agreement between these projections and the images generated by text-conditioned diffusion model. Given text prompt y, SDS can be described as:
Ox ae
VeLsus Er,e.c (())( c)
where E is the noise predicted by Φ x: is the rendered image of view c by θ S2 is weighting factor.
Variational Score Distillation (VSD). Building upon SDS, Variational Score Distillation (VSD) 34 introduces a probabilistic framework that treats the problem of text-to-3D synthesis as a distribution optimization task. VSD seeks create distribution over possible 3D shapes that likely under the given text condition, rather than finding a single deterministic shape. VSD can be expressed as:
VeLvsD Ee,,, ((t) v.) éo v)) dx 00
where éØ (wr, c, v) is the noise predicted by rendered images. This approach allows for exploring richer space of 3D geometries, potentially capturing more complex and diverse features consistent with the textual description.

---

3 Analysis of Janus Problem in Text-to-3D
In this Section, analyze the reasons for the Janus Problem produced by the SDDS-based text-to-3I) method. Through modeling, we have identified how the discrete view encoding method leads to shared distributions that cause the Janus Problem. For detailed derivation, please refer to Appendix C In response to this finding, we have developed the LCGen method, as described in Section
(b)
(d)
irverhend
PDa)
Σ p (b.0, (b. sin
view
back
by
back
0/4
2pi
Janus Problem
P(D)
bg
Side vien
side
side
((b))
siće
back
overbead
(05-5)
of
62
5o 5g
6.
6/
0.19
pi2
.12
Figure 2: Analysis of the Janus Problem in Text-to-3D. Due to the discrete encoding of viewpoints, there is a high probability of multiple heads appearing at different positions b on the sphere.
Our modeling focuses on two variables: the camera viewpoint parameter e and the position of the head in the 3D representation. Both variables can be considered points on a sphere S represented by (0, Φ). Taking the head position as an example, our modeling yields the probability P(b) of the head appearing at each position on the sphere. The modeling steps are as follows:
As shown in Fig. [a), in the SDS, the sphere S divided into different region intervals I. For each within I, the same viewpoint text is used, resulting in the same text condition and diffusion guidance (See Appendix. [B. We represent the guidance as a superposition of Gaussian distributions with the spherical position as the variable. In one-dimensional Gaussian distribution across 0 dimension, the probability p of generating a head at position under the viewpoint e is given by:
p² (b. c) = Σ w
1
cxp
(c- b b-
(5)
20
2TO²²
I2
where uF are the mixture weights for ith Gaussian component in I, 1z and of are the mean and variance of the i-th Gaussian component in I. Σ, wf = 1 to ensure that the total probability of the mixture distribution in I integrates to one over its domain.
As shown in Fig. b), integrating over the viewpoints [çu, c1] within I gives the probability P of generating head at position b within p=((b) = - (b, de.
Now, we extend from one dimension Gaussian distribution two dimensions (b. (c (-)(c- b 147) "c b nz{)) in Fig. 2() and consider spheri-
E, cal integration. the probability of generating a head position is given by: I E/
E, wf 1 exp
P(b) - Σ (1b)=/ I / (b.
Σ pF (b, Φ) sin(0) d0 do
(6)
where I represents different regions of the sphere, dA is the differential solid angle element in spherical coordinates, and sin( accounts for the area element on the sphere. Each integral fz calculates the contribution to P(b) from each region I.
Thus, we can use numerical integration techniques to obtain the probability of generating head at different positions b. As shown Fig. dd, probability peaks appear different positions This indicates that discrete viewpoint encoding may lead to the generation of heads at different positions in the 3D representation, known as the Janus Problem. Another head is equally likely to appear on the side and back, compromising the realism of the 3D representation. See details in Appendix C

---

LCGen: Low Certainty Generation
Text Guidance
corgi"
Pre-trained Diffusion
31 rendering
"from (I)
2
view"
Text Encoder
1.CGen
PE
Φ
Data flow
Back propagution
Input
Lenr (x
(t)
Camera parameters
Represeniation
I
Region Interval of
Lsns
Image from
Figure 3: Overview of LCGen. LCGen can be embedded into any SDS-based text-to-3D method, providing different guidance for various viewpoints by constraining the generation certainty, thus alleviating Janus Problem.
From the analysis in Sec. 3] it is evident that the discrete view-dependent text condition lead the 3D representation to manifest the Janus Problem at different positions b during the synthesis process. Therefore, addressing the relationship between and guidance presents a viable method for mitigating the Janus Problem. We propose LCGen, which leverages the certainty characteristics decouple distributions across different viewpoints, thereby mitigating the Janus Problem in the text-to-3D task. Specifically, for I, we constrain the guidance towards pf c, y) rather than pf (x, |, c.
LCGen. During the diffuusion denoising process, different e with different æ; the same object possess distinet certainty (z.)-1laz). We decouple these guide the synthesis process generate images that correspond more closely to the desired viewpoint.
In the diffusion model in SDS, each denoising step is considered probabilistic inference process from the current state 3, to the previous state -;-1. This process typically relies on the following assumptions: 1) The noise follows Gaussian distribution, which is estimated by the model at each step. 2) The mapping from x, to 2- can be represented using parameterized Gaussian process.
Assuming we have obtained the prediction of é, through the diffusion model, we can predict the estimation the certainty æ;-1 given æt. This estimation typically assumes that the certainty follows Gaussian distribution:
(< A N(# ,
Here, μ and c2 represent the mean and variance that guides the U-Net's prediction, thus affecting the synthetic results. ft" and o² can be caleulated as follows:
1
3 (x:
/1 Brēx), o =2 =1
VI
Here, 3r is the variance parameter step a4 is 1 3g. The Gaussian distribution parameters guide the Diffusion model's prediction to minimize the Janus Problem at step - 1. The certainty function defined
1 exp (- (_ 1202
C (S(x1-1/Z))
c0c2
By constraining Cx(,,11111 l we can ensure that different viewpoints have different distributions. We have designed the Leen as follows:
Lcon 2 1. (x G(c)
(10)
7
where 7 is normalization constant, G(c) is the function of view-based guidance.

---

0
90
180
270
0
90"
180
270
Origin
DreamFusion
I.Ceen (Ours)
Origin
Magic3D
LCGen (Ours)
ProlificDreamer
Origin
I.CGen (Ours)
"a beagle in detective's outfit"
"a fox playing the cello"
Figure 4: Results of Qualitative Comparison. The areas enclosed in red boxes are where the Janus Problem occurs.
Back propagation. Although Eq. 10 specifies the basic form of Leens direct backpropagation involves redundancy. In this method, the diffusion model serving guidance is frozen, and it is unnecessary calculate gradients for it. We hope to perform backpropagation directly on NeRF. In SDS 255, to simplify the gradient calculation and update processes, researchers simplify the loss function Lsns, avoiding the complex gradient calculations involved with a frozen diffusion model +. The gradients are applied directly to NeRF θ and its rendered images In LCGen, we also specifically design the Leen to bypass the diffusion model and apply directly in the NeRF flow parameter updates.
For Leen we obtain:
dLett Oz;
(11)
8e
Ox,
00

---

Using the chain rule, we can expand and simplify the first term on the right side in Eq. of the process, please refer to the Appendix D
11
For details
We get:
VeLcor(O) - w(1) 1
dz,
(2(1-1112))) (G)))
7
(12)
dx.
B w(t) Leet
de
where
(tt) 21-1
vT
(t v1 B.ēc))
(13)
c2
B
By simplifying the calculations, we enable Lee to apply directly in the NeRF flow parameter θ updates, avoiding the complex gradient calculations of the Unet layer in diffusion model Φ and maintaining calculation consistency with Lsns-
Experiment
In this Section, we apply LCGen to several baseline methods of SDS-based text-to-3D, including
DreamFusion 251. Magic3D [16], and ProlificDreamer [34), and conduct corresponding experi- ments. We also compare with other methods that address the Janus Problem. Sec. presents the effects of original methods and LCGen, including qualitative and quantitative assessments. Sec. 5.3 demonstrates the ablation of hyperparameters. Furthermore, Sec. presents visualization results of I.CGen's 's impact on generation certainty.
5.1 Experiment Settings
We implement original methods and LCGen based threestudio 8 and single A100 GPU. In the experiment, we set G(c) = (o] and 7 to 10, and obtained the results after a maximum of 10,000 steps. For the sake of experimental consistency, we have chosen the Stable Diffusion 2.1 base [27] as guidance and NeRF (24) as the 3D representation in the SDS-based method. See Details in Appendix. and F
5.2 Results of LCGen
Qualitative Comparison. We selected two sets of text prompts from the library (25 and conducted experiments three SDS-based text-to-3D baseline methods, including DreamFusion-sd [25), Magic3D-sd coarse [16] and ProlificDreamer (34), both without and with LCGen, with qualitative results as shown in Fig. It can be observed that, indicated by the red boxes in Fig. 4 the original methods have high probability of exhibiting the Janus Problem. For instance, in the first set of examples, the beagle appears with faces, and the second set of cases, the fox's front and back both exhibit a cello. As previously analyzed, this is a common issue inherent to SDS-based methods, resulting from the nature of the paradigm. After applying LCGen each method, the Janus Problem was mitigated, with the generated 3D content exhibiting spatial consistency. In particular, our method has very strong suppressive effect on the Janus Problem that occurs on the backside of objects.
Quantitative Comparison. We have conducted quantitative analysis of the results from our three sets of base- lines, as shown in Fig. 5 The Janus Rate (JR) represents the rate at which the Janus Problem occurs and used to measure the 3D consistency of the generation method. The Score (CS), on the other hand, is metric for assessing the consistency between the text and the prompt and the generated image. See Appendix. for details. We selected sets of text prompts from the library and calculate the mean score. It can be observed that method significantly reduces the probability of the
Figure 5: Results of Quantitative Comparison. represents that a smaller value is better, while 1 indicates that a larger value is preferred. The gray background represents the results with LCGen.
Dreamfusion
Magic3D
Proliticdreamer Origin
Metries
Origin
LCGen
Origin
LCGen
LCGen
JR (%) CS (%)
56.67
16.67
46.67
23.33
63.33
20.00
22.73
22.95
23.77
23.61
26.23
28.94
7

---

Janus Problem occurring Meanwhile, the CS reflects that LCGen improves spatial consisteney without compromising the quality of the generated images. This demonstrates the effectiveness and practicality of ICCen.
LCGen Other Methods addressing Janus Problem. Some current work is also designed to address the Janus Problem [I5] [9-21 28 40 13 I 12]. However, these methods either require extensive multi-stage fine-tuning object-specific designs and do not address the fundamental causes of the Janus Problem, as shown in Fig. 6 (See more comparison in Appendix G Compared
these methods, our approach has the follow- ing advantages: 1) It targets the essence of the Janus Problem by tapping into the multi- perspective information within 2D priors without the need to introduce additional information. 2) It can be directly inte- grated into any SDS-based text-to-3D method.
3) There no need to alter the training paradigm, and the computational cost for cer- tainty calculations is negli- gible compared to the base- line.
Figure 6: Comparison of different methods dealing with Janus Problem.
No Additional
Single Stage and No tuning
No Object-
JR
CS
Method
Prior
specificity
(%)
(%)
MVDream
28
(3D data)
20.00
26.17
Prep-Neg 1
26.67
26.23
D-SDS 12
(LLM)
V
23.33
24.82
DreamControl
L.Geen (Prolifiicdremme)
/
20.00
28.
5.3
Ablation Study
Ablation of G(c). G(c) is an important hyper- parameter in LCGen, representing the prefer- ence for the selection tendency of generation certainty. The larger the G(c)). the higher the suppression of certainty, and the more inclined towards low certainty generation. For the text prompt "a corgi" in ProlifiDDreamer, we de- signed different G(c) as shown in Fig. 7 It can be observed that when G is piecewise Tunction of (+) Fig. 7a). there a possibility of gener- ating multiple faces from different front views; when using an absolute value function of Φ, can generate correctly but also exhibits subtle differences in the generation process, as shown in Fig. (b)c). See Appendix. for G selection.
1k: Stop
2k Step
3k Step
4k Step
10k: Step
(()
others
(b)
Figure 7: The impact of different choices of G(c).
Certainty within the training step windows. We also conducted ablation experiments on the certainty at different time step windows during the training process. As shown in Fig. g for each step, window (slep 200, slep) is se-
LCGas las langer vanimas
Origin has larger variance
Dilloaue
lected, and the variance of certainty within the
window is calculated. By subtracting the win-
dow variance values of the LCGen and origin methods, the results are obtained. It can be ob- served that during the training process, the cer- tainty variance most step windows LCGen larger. This indicates that the LCGen method effectively separates the certainty of different viewpoints.
Stop Window
12909
Figure The difference in the variance of certainty within the training step windows between w/ and w/ LCGen.
5.4 Visualization
In this Section, we conduct visualization analysis of the LCGen and the corresponding certainty for each viewpoint. Our chosen textual example corgi". We use PromificDreamer 34 as our baseline and achieve results after maximum of 10,000 steps.

---

Origin
LCGen
1ligh
Cortaiity
99.9.9 2 devatim B
dierationn
-150-100 500
szimuthh
100
-150 100-100-000
walmuch +
5o
150
120"
180"
120"
120a
180
120"
60"
-60"
60"
60
Figure 9: Visualization of Certainty after Generation w/o and w/ LCGen. The upper half of the 3D) scatter plot has the camera view parameters (0, Φ) on the x and y axes, and Certainty on the 7-axis. The lower half depicts the generation results from different views, along with their & and the Certainty colors represented in the scatter plot.
As shown in Fig. 9 the left side shows the original ProlificDreamer results, while the right side features the ProlificDreamer using the LCGen method. For each method, the lower half presents visualized outputs showing rendering results at various azimuths é with an elevation 0 of 15%. It can be observed that the original prolificdreamer generates the corgi with faces on both the front and back, which is indicative of the Janus Problem. In contrast, the 3D representation created using LCGen exhibits high spatial consistency. We have also visualized the certainty obtained from the diffusion model after inputting images of the fully trained NeRF from various viewpoints. It is shown that due to the presence of multiple faces, the certainty in the original method does not show a clear pattern of change, consistent with the analysis presented earlier. In comparison, LCGen demonstrates a more distinct pattern of certainty varying with e where the certainty is greater when [ø]] small and decreases as [6) increases. This highlights the effectiveness of LCGen's 's constraints on certainty and also corroborates the relationship between certainty and c. See details in Appendix B
6 Related Works
Methods addressing Janus Problem. Past research has explored multi-stage networks that utilize 3D priors to reduce the Janus Problem. A two-stage 2D lifting framework has been proposed in DreamControl [13,, leveraging 3D self-priors to enhance geometric consistency in 3D) generation. Other approaches, such as Perp-Neg [1]. innovate by using negative prompts in diffusion models to remove undesirable attributes or views while maintaining the core concept. However, these approaches either necessitate extensive multi-stage pre-training without tackling the root causes of the Janus Problem. Additionally, they often require specialized and resource-intensive procedures. See details in Appendix. G
tailored to specific objects,
Conclusion and Discussion
In this study, we initially model the Janus Problem and analyze its causes visually. We then introduce LCGen to guide text-to-3I generation toward spatial consistency by establishing varied certainty priors across viewpoints. Our method, validated through experiments, can integrate scamlessly with

---

various SDS-basee text-to-3D methods to mitigate the Janus Problem. It does so without adding extra data requirements, excessive computational overhead, or degrading the quality of generated outputs.
Limitations and Broader Impacts. Our method performs well in generating individual objects but has limitations with complex multi-object scenes. Due to the lack of 3D training data resulting in an insufficient understanding of the 3D world, LCGen still has some failure cases in the text-to-3D Janus Problem. Additionally, using fixed G leaves room for improvement adaptive generation. At the same time, realistic Al-generated content may have adverse social impacts. Like other generative model researchers, we must remain vigilant and take precautions against generating false content.
Acknowledgements. This work was supported by the National Natural Science Foundation of China under Grant 62406075, National Key Research and Development Program of China under Grant 2023YFC3604802; in part by the China Postdoctoral Science Foundation under Grant 2023M730647 and Grant 2023TQ0075.
References
[1] Armandpour, M., Sadeghian, A., Zheng, H., Sadeghian, A., and Zhou, M. (2023). Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.0499
[2] Avrahami, O., Fried, O., and Lischinski, D. (2023a). Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11.
[3] Avrahami, O., Hertz, A., Vinker, Y., Arar, M., Fruchter, S., Fried, O., Cohen-Or, D., and I Lischinski, D. (2023b). The chosen one: Consistent characters in text-to-image diffusion models. arXiv preprint arXiv:2311.10092
[4] Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., and Goldstein, T. (2023). Universal guidance for diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852.
[5] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., and Cohen-Or, D. (2023). Attend-and-excite: crcntion-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10.
[6] Chen, R., Chen, Y., Jiao, N., and Jia, K. (2023). Fantasia3d: Disentangling geometry and appear- ance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2224-222555
[7] Cheng, T.-Y., Gadelha, M., Groueix, T., Fisher, M., Mech, R., Markham, A., and Trigoni, N. (2024). Learning continuous 3d words for text-to-image generation. arXiv preprint arXiv:2402.08855.
[8] Guo, Y.-C., Liu, Y.-T., Shao, R., Laforte, C., Voleti, V., Luo, G., Chen, C.-.,, Zou, Z.-X., Wang, C., Cao, Y.-P., and Zhang, S.-H. (2023). threestudio: A unified framework for 3d content generation. httph:://////.. ht///gittuubb com
[9] Henzler, P., Mitra, J., and Ritschel, T. (2019). Escaping plato's cave: 3d shape from adversarial rendering. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9984-9993.
[10] Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851.
[11] Hong, S., Ahn, D., and Kim, S. (2023). Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation. arXiv preprint 2rXiv:2303.15413.
[12] Hong, S., Ahn, D., and Kim, S. (2024). Debiasing scores and prompts of 2d diffusion for view-consistent text-to-3d generation. Advances in Neural Information Processing Systems, 36.
[13] Huang, T., Zeng, Y., Zhang, Z., Xu, W., Xu, H., Xu, S., Lau, R. W., and Zuo, (2023). Dream- control: Control- -aard text-to-3d generation with 3d self-prior. arXiv preprint arXiv:2312.06449,
10

---

[14] Kumari, N., Zhang, B., Zhang. R., Shechtman, E., and Zhu, J.-Y. (2023). Multi-concept cus- tomization of -on-tto-mage diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 1931-1941.
[15] Li, W., Chen, R., Chen, X., and Tan, P. (2023), Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. arXiv preprini arXiv:2310.02596
[16] Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., and Lin, T.-Y. (2023). Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309.
[17] Lin,J. J., Tao, Z., Tong, X., Mai, X., Wang, H., Wang. B., Wang, Y., Zhao, Q- Yu, J., Lin, Y., et al. (2024a). Suppressing uncertainties in degradation estimation for blind super-resolution. arXiv preprint arXiv:2406. 16459.
[18] Lin, J., Wang. Y., Tao. Z., Wang, B., Zhao, Q. Wang, H., Tong, X., Mai. X., Lin, Y., Song, W., al. (2024b). Adaptive multi-modal fusion of spatially variant kernel refinement with diffusion model for blind image super-resolution. arXiv preprint 2503080403.05808
[19] Liu. R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., and Vondrick, C. (2023a). Zero-I-to-3: Zero-shot one image to 3d object. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
[20] Liu, Y. Lin, C., Zeng. Z. Long, X., Liu, L., Komura, T., and Wang, W. (2023b). Sync- dreamer: Generating multiview-consissentt images from a single-view view image. arXiv preprint arXiv:2309.0345.
[21] Long, X., Guo, Y.-C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.-H., Habermann, M., Theobalt, C., et al. (2023). Wonder3d: Single image to 3d using cross moos--dmmain diffusion. arXiv preprint arXiv:2310.15000
[22] Mai, X., Lin, J., Wang, H., Tao, Z., Wang, Y., Yan, S., Tong, X., Yu, J., Wang, B., Zhou, Z., et al. (2024). All rivers run into the sea: Unified modality brain-inspired emotional central mechanism. In ACM Multimedia 2024.
[23] Metzer, G., Richardson, E., Patashnik, O., Giryes, R., and Cohen-Or, D. (2023). Latent-neri for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663-12673.
[24] Mildenhall, B., Srinivasan, P., Tancik, M., Barron, T., Ramamoorthi, R., and Ng. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 6561)999-106.
[25] Poole, B., Jain, A., Barron, T., and Mildenhall, B. (2022). Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.1499
[26] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PMLR.
[27] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695.
[28] Shi, Y., Wang, P., Ye, J., Long, M., Li, K., and Yang. (2023). Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512.
[29] Song, Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32.
[30] Tao, Z., Wang, Y., Chen, Z. Wang, B., Yan, S., Jiang, K., Gao, S., and Zhang, (2023). Freq-hd: An interpretable frequency-ban based high-dynamics affective clip selection method for in-the-wild facial expression recognition in videos. Proceedings of the 31 ACM International Conference on Multimedia, pages 843-852.

---

[31] Tao, Z., Wang. Y., Lin. J., Wang, H., Mai, X., Yu, Tong, X., Zhou. Z., Yan, S., Zhao, Q. et al. (2024). lign-dfer: Pioneering comprehensive dynamic affective alignment for dynamic facial expression recognition with clip. arXiv preprint wrXiv:2403.04294,
[32] Tewari, A., Thies, J., Mildenhall, B., Srinivasan, P., Tretschk, E., Yifan, W., Lassner, C., R., Lombardi, S., et al. (2022). Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library.
Sitzmann, V.,
[33] Wang, P.. Xu, D., Fan, Z., Wang, D., Mohan, S., landola, F., Ranjan, R., Li, Y., Liu, Q. Wang, Z. et al. (2023). Taming mode collapse in score distillation for text-to-3d generation. arXiv preprint arXiv:2401.0090
[34] Wang. Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, (2024). Prolificdreamer: High-fideling fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36.
[35] Wu, J., Zhang, C., Xue, T., Freeman, B., and Tenenbaum, J. (2016). Learning a probabilis- tic latent space of object shapes via 3d generative-adveesarfi modeling. Advances in neural information processing systems, 29.
[36] Xu, X., Guo, J., Wang, Z., Huang, G., Essa, I., and Shi, H. (2023). Prompt-free diffusion: Taking" text" out of text-to-image diffusion models. arXiv preprint vrXiv:2305.16223
[37] Yu, Y., Zhu, S., Qin, H., and Li, H. (2024). Boostdream: Efficient refining for high-quality text-to-3d generation from multi-view diffusion. arXiv preprini arXiv:2401. 16764.
[38] Zhang, L.. Rao, A., and Agrawala, M. (2023). Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847.
[39] Zhang, X. Yang. L., Cai, Y., Yu, Z., Xie, J. Tian, Y., Xu, M., Tang, Y., Yang, Y., and Cui, B. (2024). Realcompo: Dynamic equilibrium between realism and compositionality improves text-to- image diffusion models. arXiv preprint arXiv:2402.1 12908.
[40] Zhao, M., Zhao, C., Liang, X., Li, L., Zhao, Z., Hu, Z.,, Fan, C., and Yu, X. (2023). Efficient- dreamer: High- fideliiilitty and robust 3d creation via orthogonal-view diffusion prior. arXiv preprint arXiv:2308. 130.2333..33.3.
[41] Zhengwentai, (2023), clip-score: CLIP Score for PyTorch. httpps://gituub..co/aaaatt clip-score Version 0.1.1.
12

---

A
Symbol Reference Table
Table 1: Symbol Reference Table
Symbol
Mcaning
Description
State variable or ren- dered image
Represents the state at time in the diffusion process rendered image from 3D models
Transition probability distribution
((2)) 11-1)
Probability of state æ:-1 given state æ; during the diffu- sion forward process
Transition probability distribution
Probability of state Tt-1 given state æ, during the diffu- sion reverse process
Variance parameter
Variance term at time / in the diffusion process
Identity matrix
Identity matrix with ones on the diagonal and zeros else- where
./
Gaussian distribution
Normal (Gaussian) distribution Mean of the Gaussian distribution
H4,
Mean
Variance & covari-
a² E, Σ
Variance & covariance matrix of the Gaussian distribution
ance matrix
Pretrained model
Pretrained 2D diffusion model
+
3D model
3D model to be optimized
CSDS
SDS loss function
Loss function used in Score Distillation Sampling
LVSD
VSD loss function
Loss function used Variational Score Distillation
52
Weighting factor
Weighting factor used in LsnS and LVsD
Noise
Noise term in the diffusion process
ét
Predicted noise
Noise predicted by the diffusion model at time
Time step
Time step in the diffusion process
Text prompt
The text input that guides the generation process in text-
to-3D text-to-image models
Camera viewpoint rameter
c
Represents camera viewpoint parameters in the 3D model
Head position in 3D representation
b, b
Represents the position of the head in the 3D model
S2
Sphere
The 2D spherical surface on which and lie Regions of the sphere S
I
Interval
Weight of the i-th Gaussian component
W/
Represents the weighting factor for the i-th Gaussian com- ponent in mixture model
p(b,c) c)
Probability
Probability of generating head at position b under view- point c
P(b)
Probability
Probability of the head appearing at position b
A
Solid angle element
Differential solid angle element in spherical coordinates
(0,
Spherical coordinates
The clevation and azimuth used to represent points on S Coefficient related to Br in the diffusion process
0t-1
Coefficient
(a(a--) (
Certainty
Defined certainty in LCGen
Certainty loss func-
Leert
Loss function used in LCGen ensure certainty consis- tency
tion
Normalization con-
Constant used for normalization in Leen
stant
G(c)
View-based guidance
Function for view-based guidance in Leet Weighting factor used in Leett
Weighting factor
B View Distribution in 2D Diffusion Model
We generated 100 2D images for each text condition o I in Stable Diffusion 2.1 base [27], with the texts y being "a pig wearing backpack, front view", "a pig wearing backpack, side view".
13

---

Distribution over elevation 0
20000
"a pig wearing backpack, front view"
00
MEB: IT RCTEIT KUDI NOTER IE RE 10 vrrm EOM Mor W CORSE
43
00
pig wearing backpack, overhead view"
Distribution over azimuth &
"a pig wearing a backpack, front view"
pig wearing backpack,
bock view"
29
MOTTON
"a pig weoring backpock.
side view"
1815
17 ERREI PM 21 8.00
10 OI MUNE BERO 1O 5
11
E19 L
135
90
-45 -90 -13-5-1-880
135
-80
Figure 10: View distributions given different text conditions in 2D diffusion model.
"a pig wearing backpack, back view", and "a pig wearing backpack, overhead view". For the generated images, we manually assessed and categorized the camera parameters in the elevation and azimuth & dimensions, obtaining the results shown Fig. 10
can be observed that the images generated by the 2D diffusion model have some variation in view distribution corresponding to different y of I. exhibiting different peaks. From the 0 dimension, in the front view, almost all (99%) of the generated 2D images are at 0 = 0. If the text is changed to an overhead view. 22% of the images are above 0 0. In the azimuth dimension, when the view is front, 75% the generated images are concentrated at ø = O; when the view back, 44% of the images 180, while 29% are at á 0; for the side view, the distribution is relatively scattered, generally symmetrically distributed around ő = 0, exhibiting diverse view characteristics.
This corresponds to the content in Sec. When different within I are given the same y, the 2D diffusion guidance follows the same distribution. In this work, we model this distribution as combination of Gaussian distributions, shown in Eq. 5
14

---

C Detailed Analysis of Janus Problem
According Sec. 3] SDS-based text-to-3D methods utilize a view-dependent prompt approach with discrete encoding. In the context of the sphere S based on elevation and azimuth, Camera parameters e are classified into intervals I representing four view categories: side, front, back, and (a).
overhead, as shown in Fig.
However, text prompts are generally limited to high-level descriptions and do not offer the precise control achievable in actual photography. This issue largely stems from the lack of detailed data in diffusion training datasets, where specific camera settings are rarely documented 77]. As result, images generated based on viewpoint text conditions in the diffusion model tend to be imprecise and show significant viewpoint fluctuations. Images of the front may be generated under guidance from textual prompts of different viewpoints []] To address this, the conditional probability density function of (æx c, y) for each interval I modeled as a mixture of multivariate Gaussian distributions, each parameterized by (MF, )
Specifically, taking a one-dimensional Gaussian distribution an example, for camera parameter E I, the probability density for synthesizing a head at position with bias b E S² is denoted as follows:
xf (b.c) = =Eet wt
1
exp
(_(c-b- (G-bb117)
(14)
2
207²2
where: wp are the mixture weights for each Gaussian component interval I, uo7 and o7² are the mean and variance of the i-th Gaussian component, Σ, wf to ensure that the total probability of the mixture distribution integrates to one over its domain.
As shown in Fig. (b), the integral of p² with respect over the interval I is given by:
(- (c b b- b- 12
p=(b) = / dc Σ w
exp
dc.
(15)
T.2
V 2
[Co, C1] represents the view interval of integration. The integral pz provides the probability density function PDF of Janus Problem occurring
Here, I
We now extend our analysis to the multivariate Gaussian distribution, shown in Fig. (c). This extension actually incorporates several inherent settings regarding camera geometry 3D space, as shown in Fig.
looker
Φ
1. The camera's "lookat" axis intersects the world coordinate origin, effectively fixing two rotational degrees of freedom of c,
2. The "up" axis is aligned with the z-axis of the world coordi- nate system, thereby constraining another rotational degree of freedom,
Figure 11: Camera geometry in 3D space.
3. The assumptions hold under a constant distance between the camera and the world coordinate origin, which fixes the translational freedom along the radial axis from the origin a parameter that does not explicitly influence the perspective modeling.
Given these conditions, the remaining camera parameters can described by two active degrees of freedom: elevation 0 and azimuth o. If we evaluate the impact of the vector b on this distribution over a specified interval Z, the resulting integral pV (b) can be interpreted as the probability density of b over that interval:
15

---

(b)
p (b. c)
(16)
Σ wf [2 exp
1 - -
E (c- b m²) dc.
Here, k = 2 is the dimension of the space, с E I and b E S are vectors defining the position in this space, and pu and 2x represent the mean vector and covariance matrix of the distribution, respectively.
Recognizing that different intervals I yield distinct probability functions, the total probability density P(b) for the Janus Problem occurring at the position defined by b E s is calculated as the sum of integrals over distinct regions I of the sphere:
P(b) Σ / (b.. I
Σ p (b, (b, I
sin(0) d0 dø
(17)
where represents different regions of the sphere, dA is the differential solid angle element in spherical coordinates, and sin(@) accounts for the element the sphere. Each integral JI calculates the contribution to P(b) from each region I.
While direct integration theoretically feasible, the integration of multivariate Gaussian distributions can present considerable com- plexity in practical execution. Consequently, numerical integra- tion techniques are employed to approximate P(b) effectively, as shown in Fig. 2 (d). It can observed that when the parameters I, NF (1² Ef and w/ are altered, P(b) exhibits a high likelihood of manifesting the Janus Problem various points E S
P(b)
This provides us with an approach to address the Janus Problem. If we can decouple the relationship between viewpoints and distribu- tions, allowing different to follow different distributions (with coinciding with c), we can achieve the modeling results shown Fig. 12 It can be observed that there is only one probability peak in S indicating the presence only one head. Based on this, we propose LCGen, utilizing the method of low certainty generation, to effectively mitigate the Janus Problem, as shown in Sec.
dolh
of
Figure 12: Modeling result without Janus Problem.
D Back Propagation Details
For Leet, we obtain:
ÖLeen 00
dLeet
Oæ: dx,
(18)
0e
Using the chain rule, we first calculate the partial derivative with respect to zr, involving derivatives of μe and o. Given u and σ since o does not depend on It its derivative zero. The derivative of μº is:
0h dx,
(19)
Br
Since a is independent 2,, inserting into the expression for ("(;=_1/x))) and using properties of the Gaussian distribution, we have:
Op
(x1-1- ('(
(20)
dx.
gc2
dx;
Since G(c) is independent of æ;, incorporating the derivative into
0C.m. de.
16

---

donn
Op G(c)
1 (oc
(---1 ve)
pp
(21)
Oa, 0
σ
dx.
Substituting the expression for ue, we get:
VeLcar(S) BE,cc w(1) 1 (-(11-1 -G))
Oz,
(22)
dx,
B w(t) Leen
where
3t-1 vI-B.
Dt vI
BrEx)
(t)
(23)
gc2
5
E
Implementation Details
E.1
Analysis of Janus Problem
In the Sec. and C we simulated different data distributions by setting different values of μ and Σ
in Eq. 16
To simplify the model, we use o² instead of E.
Intervals and Boundaries
Threshold across elevation 0:
foverhead 4
2. Thresholds across azimuth de
7 8
Frront 6
baak
Parameters of Gaussian Distributions
Side view:
Integration Range of Isido:
0 0 <
loverhend
0 < < loverhead
and
ifrnt
3r + toack
m + trrnt < <
< 3r
back
Parameters Gaussian Distribution 1:
(0, = b0)
σ1 = 0.3
Parameters of Gaussian Distribution 2:
(0,
π
+ bb π
+ bo) 0.3
2
2
Front view:
Integration Range of Tiront
0 < 0 < π loverbead
ttrout < $ + trront
Parameters Gaussian Distribution 3:
(0, क + bb). 03 0.2
3. Back view:
17

---

Integration Range of Tocck:
0 S 1 < π Loverhead
3x 2
back S ② S 2 Bback
Parameters Gaussian Distribution 4:
3π bo
(0,
bb
04 0.4
0.4
2
4. Overhead view:
Integration Range of "
2 π foverhead
Parameters Gaussian Distribution 5:
(0
1bg), b6). 03 0.3
Integral Computation Range of b by Values:
by ranges from
to with 50 intervals
by Values:
by ranges from x with 50 intervals
From this, we can obtain the 3D plot in Fig. 2 dd) By varying I and the parameters of the Gaussian distribution, can model the probability of generating a Janus Problem at different b under different distributions. The location of the probability peak represents the potential position where head
might appear.
E.2 LCGen.
We implement original methods and LCGen based on threestudio ( and single A A100 GPU, using the PyTorch framework. For Dreamfusion, Magic3D, and ProlificDreamer, the hyperparameters are all kept consistent with the default settings in config files in the repository. Specifically, according to the instructions in threestudio,
In Dreamfusion, to prevent scene being stuffed with floaters/becominng empty, set vystem.loss.lamdla Laaaaammadssssstt_O..... s..
In Dreamfusion and Magic3D, to prevent the model incorrectly treating the background as part of the object, we replace the background with random colors with a probability 0.5 by setting system. www.mmacckrood..raaaa aandom_aug-lnn
In the experiment, we set Gi(c) = (ő), and obtained the results after a maximum 10,000 steps. We also attempted to embed into G. In the experiments, never appeared in 0 dimension, and setting 0 did not significantly affect the results. Therefore, ultimately chose the above G. For the sake of experimental consistency, Stable Diffusion 2.1 base [27] as guidance and NeRF (24 as the 3D representation the SDS-based method. In Leet, provides the most stable mitigation of the Janus problem.
found that the multi-head problem almost
have chosen the
" the normalization parameter. In our experiments, we y to 10, which
E.3 Visualization.
In Sec. 5.4 to make the results clearer, we smooth the values at each position. Specifically, we take the average certainty of all points within distance of 0.1 from the given position and use this average as the certainty the scatter point. This helps to avoid disturbances the visualization results caused by factors such as random view sampling and t.
18

---

Metrics
In the text-to-3D task, it is difficult to quantitatively evaluate the generated results. This task lacks ground truth for objective scoring. In this work, our goal is to reduce the occurrence of the Janus Problem, so it is necessary to assess the 3D consistency of the generated models and the extent to which the Janus Problem is mitigated. Similar to DreamControl [13, we choose CLIP-Score (CS) and Janus Problem Rate (JR) as our quantitative metrics Sec.
CLIP-Score (CS), The CLIP Score [41 is calculated using the CLIP (Contrastive Language-Iman Pretraining) model [26). which language-ima -inggg model learned through contrastive learning. The CLIP model consists of an image encoder and a text encoder, enabling it to measure the similarity between different modalities. The CLIP Score measures the Cosine Similarity between two embedded features. This implementation utilizes the pretrained CLIP model to calculate the mean average of cosine similarities between text and generated images from different views.
Janus Problem Rate (JR). To assess the consistency of 3D geometries, we count the instances of inconsistent 3D content produced by each method and calculate their proportion in all content. This represents the occurrence rate of the Janus problem (JR).
G Comparison with other methods
Table 2: Comparison of different methods dealing with Janus Problem.
No Additional
Single Stage
No Fine-tuning
No Object- specificity
Method
Task
Prior
SweetDreamer
15
Image-to-3D
X (3D data)
X
X
SyneDreamer
20
Image-to-3D
(3D data)
X
x
Wonder3D 21
Image-to-3D
X (3D data)
X
X
EfficientDreamer
Image-to-3D
(3D data)
X
X
Zero-1-to-3
19
Image-to-3D
X (3D data)
X
X
MVDream
28
Text-to-ID
x (3D data)
X
Prep-Neg
Text-to-ID
X
D-SDS
12
Text-to-II
X (LLM)
J
DreamControl 11
Text-to-3D
X
LCGen (Ours)
Text-to-3D
Table. presents a comparison of our LCGen method with other approaches addressing the Janus Problem. It can be observed that some other methods utilize additional prior information, some employ multi-stage training strategies and fine-tuning of pre-trained models, and some require object- specific designs. In contrast, directly embedded into existing SDS-based text-to-3D methods without fine-tuning the models. alleviates the Janus Problem with minimal computational cost.
method does not require any additional information and can be
H
Some Other Results
Examples without concept of heads. In Fig. 13 A and B, we show the results of 'a street lamp' and tree' as you mentioned. It can be seen that both the original Prolifiedreamer and our method achieve spatial consistency generation. Considering that the above two examples do not have obvious differences between different views, we conducted experiments on sunflower' and "a piano' as shown in Fig.C. C and D in Fig. 13 It can be seen that the original Prolificdreamer produced spatial inconsistencies, showing multiple frontal images of the flower and multiple keyboards in single piano from different views. Our method successfully alleviated this issue, generating sunflower and piano with correct front and back images. This indicates that our method is effective not only for objects with heads but also for spatial consistency modeling of other objects.
Multiple objects with multiple heads. We will supplement the discussion details in the limitations section of the Main Paper. Multi-object generation is another important and challenging field in
19

---

A
B Origin
"A street lamp"
AA tree"
"A sunflower"
Origin
LCGen (Ows)
LCGen (Ows))
Origin
LCGen (Ours)
D
E
"A piano"
Origin
"A corgi"
Origin
LCGen (Ours)
LCGen (Our))
"A sleek, silver gray dolphin leaping grocefully out of the crystol clear ocean, its body glistening the sunlight as it arcs through the air with joyful exuberance" LCGen(Ours)
F
Origin
"A Motte painting of a Christmas coke made of cheese surrounded by moot made of ice cream" Origin
G
LCGen (Ours)
"Three corgis facing different directions" " D DreamContro!
H
Origin
LCGen (Ours)
Figure 13: Some other results.
text-to-3D tasks. For SDS-based methods, the current most important issue in multi-object generation is not the Janus problem but how to handle the relationships between different objects. In H in Fig. [3 we show the results of 'three corgis facing different directions.' can seen that the original Prolificdreamer cannot correctly generate 3 corgis, and there is some degree of sticking between the objects. Both our method and the current state-of-the-ar art multi-vier consistent generation method DreamControl cannot correctly handle this example. Once the multi-vies generation issue resolved, our work will have more exploration possibilities.
A large amount of descriptive language. In Fig. 13 F: "A sleek, silver-gras gray dolphin leaping gracefully of the crystal-clear ocean, its body glistening in the sunlight as ares through the air with joyful exuberance." It can be observed that the original Prolificdreamer generates two dolphin heads, while our LCGen method correctly generates the image. Fig. 13 G, we also carefully designed prompt: "A Matte painting of a Christmas cake made of cheese surrounded by moat made of ice cream". can be observed that the original prolificdreamer, the front and back of Santa Claus on the cake both have faces. However, using method, the correct samples generated.
20

---

"an astronaut riding kangaroo"
Figure 14: Failure example.
I Failure Example
Although experiments demonstrate that method excels in mitigating the Janus Problem, there are still some failure cases. Fig. 14 shows a failure example. By incorporating our method into PremificDreamerr (34 and setting the prompt y as "an astronaut riding kangaroo", ", the method correctly generated a single human head and a single kangaroo head. However, resulted issues such as an incorrect human arm position and multiple kangaroo legs. This may be due the presence of multiple objects in the generated text, making it difficult to separate the combinations of objects from the perspective of certainty.
When objects are in very unusual poses, our method has a certain probability of failing. For example, "an upside-down lion." In this case, using the original certainty to separate different views fails because the bottom might be the highest certainty view. For such samples, we need to redesign the view-based guidance function to accommodate the generation of these special samples.
Our method may fail multi-object generation scenarios. This due the limitations of current text- to-3D methods in multi-object generation tasks. As shown in Fig. 13 H, when the text is "three corgis facing different directions," baseline method Prolificdreamer fails to generate three corgis correctly, with overlap between different objects. On this basis, the current best view-consistent generation methods, such as DreamControl and our method, cannot directly address the Janus Problem. To solve this issue, the primary task is to solve the multi-object generation problem in text-to-3D, which is significant research area.
Due to the lack of 3D prior knowledge, like other SDS-based methods, our method can only model 3D representations that appear more realistic from various perspectives, but cannot ensure that these 3D representations adhere to the physical laws of the real world. For example, during the generation of octopus tentacles, since there no difference in tentacles from different views and the model does not know how many tentacles should be generated, it may produce 3D representations that do not conform to objective reality. To address this issue, we need to endow the model with the ability to understand the 3D world. One possible approach is to collect massive 3D data and design appropriate representation forms to establish a pre-trained 3D generation model (e.g.. 3D diffusion). Given the enormous data requirements and training difficulty, this requires the collective effort of the entire AIGC community.
Prompt Library
"a bald eagle carved out of wood", "a beagle in a detective's outfit", "a beautiful rainbow fish", "a bichon frise wearing academic regalia", "a cat with mullet", "a ceramic lion", chihuahua wearing tutu", "a chimpanzee holding peeled banana", "a chimpanzee looking through a telescope", "a confused beagle sitting at desk working on homework", "a corgi taking a selfie", "a cute steampunk elephant", "a DSLR photo of baby dragon drinking boba". DSLR photo cat wearing bee costume". "a DSLR photo of a corgi puppy", "a DSLR photo of dog made out of salad", "a DSLR photo of a frog wearing a sweater". DSLR photo humanoid robot using a laptop", "a DSLR photo of lion reading the newspaper", "a DSLR photo of mouse playing the tuba", "a DSLR
21

---

photo of pig playing a drum set", "a DSLR photo of a robot dinosaur", "a fox playing the cello", "a highland cow", "a lionfish", "a pig wearing backpack". "a red panda", "a tiger playing the violin", "a
zoomed out DSLR photo of baby dragon", "a zoomed out DSLR photo of monkey riding a bike"
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work performed by the authors in
See. 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: The paper provides the full set of assumptions and a complete and correct
proof in Sec. B and Appendix
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
The paper fully discloses all the information needed Sec. and Appendix,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be available [here] accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa///guddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssioiolicyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The paper specify all the training and test details in Sec. 5 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Statistical significance bars not reported because it would be too computa- tionally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: There are provided in Sec. [|ndd Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses potential societal impacts the work performed in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No released data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "S2mepWFmwh", "reviewer": "Reviewer_sRbF", "summary": "This paper presents a method to tackle the issue of the Janus Problem in text-to-3D content generation method. A method named LCGen has been proposed that focuses on low certainty regions to generate view-consistent generation.", "questions": "Please see the limitations and weakness sections, answer those questions.\nWhy is there a blue shade in the generated corgi in your method in fig 7 and 10 ?. Is this some kind of artefact due to low uncertainty generation ?", "limitations": "It seems the method will work on single object generations. It is also not clear how good this method is compared to the existing method quantitatively.\nExamples used for visual validation are also limited corgi and pig, It is not clear which 30 prompts are selected from the library for the experiments. No clarity on the dataset point of view. A clear selection of data samples MUST be added for reproducibility and validation of the proposed method. \nMany generated samples in fig 10 are not even of pig.\nOnly one failure case is shown, while there could be many other scenarios, a detailed analysis of failure cases is also missing. It is important to demonstrate the limitations within which method will work.\nWhile the evaluation is mostly based on the visuals or derived metrics from visuals. The main paper and appendix both doesn’t have sufficient diverse visual results to validate the proposed method", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Some causes of the Janus Problem have been analysed visually. We then introduce\nLCGen method is proposed to guide text-to-3D generation toward spatial consistency by establishing varied certainty priors across viewpoints. \nProposed method works without adding extra data requirements, and excessive computational overhead.", "weaknesses": "Proposed method encourage single head, but there could be scenarios where multiple heads needs to be generated e.g.  three corgis facing different directions.\nWhat about those examples where there is no concept of head. How this method will behave in those situations e.g a street lamp, tree etc.\nIt seems that the proposed low uncertainty approach would lead to low quality samples where fine-grain details are important.\nThere is no comparison with the existing approaches mitigating Janus problem. A table in the main paper with metrics CS and Janus Rate is a MUST. I acknowledge Table in the appendix G, where only functional level comparison is given. Quantitative comparison is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your efforts in reviewing our paper! Below is our response. For Figure and Table, please see *Rebuttal File*.\n\n## 1. Diverse Examples\n\nThe main purpose of our method is to help address Janus Problem appearing on a single object in text-to-3D baselines. In the **limitations section of the *Main Paper*** (lines 291-294), we mention, \"Our method performs well in generating individual objects but has limitations with complex multi-object scenes.\" We will provide more detailed explanations in the limitations section and include the following examples:\n\n1. **Examples without concept of heads.** In ***Rebuttal File* Fig.A and B**, we show the results of 'a street lamp' and 'a tree' as you mentioned. It can be seen that both the original Prolificdreamer and our method achieve spatial consistency generation. Considering that the above two examples do not have obvious differences between different views, we conducted experiments on 'a sunflower' and 'a piano' as shown in **Fig.C and D in *Rebuttal File***. It can be seen that the original Prolificdreamer produced spatial inconsistencies, showing multiple frontal images of the flower and multiple keyboards in single piano from different views. Our method successfully alleviated this issue, generating a sunflower and piano with correct front and back images. This indicates that our method is effective not only for objects with heads but also for spatial consistency modeling of other objects.\n\n2. **Multiple objects with multiple heads.** We will supplement the discussion details in the limitations section of the *Main Paper*. Multi-object generation is another important and challenging field in text-to-3D tasks. For SDS-based methods, the current most important issue in multi-object generation is not the Janus problem but how to handle the relationships between different objects. In **Fig.H in *Rebuttal File***, we show the results of 'three corgis facing different directions.' It can be seen that **the original Prolificdreamer cannot correctly generate 3 corgis, and there is some degree of sticking between the objects**. **Both our method and the current state-of-the-art multi-view consistent generation method DreamControl cannot correctly handle this example**. Once the multi-view generation issue is resolved, our work will have more exploration possibilities. For example, a current mainstream idea is to generate individual assets for each object and then stitch them together[1]. **Our method still has the potential to adapt to such methods**.\n\n## 2. Generation Quality\n\n1) Regarding the blue shading you mentioned, this is an normal process handled by prolificdreamer during the generation. As shown in **Fig.E in *Rebuttal File***, the original Prolificdreamer can also produce blue shading when changing seed, and our method can generate results without blue shading.\n\n2) As shown in the *Main Paper*, our method does not suffer from a loss in generation quality compared to baseline methods and can improve the CLIP-Score. Low certainty generation helps the model find an optimization direction that better aligns with specific views, while the fine-grain detail of the generated images is determined by the generation model rather than certainty. Additionally, by aiding in view consistency modeling, the final generation results exhibit spatial consistency and better overall quality.\n\n3) We also compared some fine-designed prompts, as shown in **Fig.F and G in *Rebuttal File***. Our method ensures high-quality generation even when dealing with finely designed objects.\n\n## 3. Quantitative Comparison\n\nPlease see the **Table in *Rebuttal File*.**\n\nIt presents a quantitative comparison with other methods dealing with the Janus Problem. It can be seen that our method, while having a series of advantages, surpasses existing methods in various metrics: it achieves the best overall performance when integrated with Prolificdreamer, strongly suppresses the Janus problem, and maintains the highest CLIP-score; when integrated with Dreamfusion, it has the best suppression effect on the Janus problem. \n\n## 4. Prompt Library\n\nDue to constraint of 6000 max length, please see **Prompt Library in General Rebuttal to all reviewers**.\n\nWe present 30 prompts selected from the Dreamfusion prompt library. Since our focus is on the Janus Problem, we have specifically chosen prompts that can evaluate method's effectiveness.\n\n## 5. The Pig Generated by Stable Diffusion\n\nAll the samples in Fig.10 in *Main Paper* are generated by Stable Diffusion 2.1 base (not our method) to demonstrate that Stable Diffusion 2.1 base has view biases during generation. This supports our analysis and thus introduces our method. In some samples, the pig is not generated correctly, reflecting the limitations of the Stable Diffusion pre-trained model. We generated all the images at once on Stable Diffusion 2.1 base and evaluated their views without any selecting to ensure the accuracy of the presented results.\n\n\n## 6. Failure Cases\n\nDue to constraint of 6000 max length, please see **Limitation Discussion in respone 2 for reviewer wiP5**. Thanks very much.\n\n*Thank you again for your efforts and suggestions! We will include detailed discussions on the above points in the Main Paper and Appendix.*\n\n## Reference\n[1] Zhou, Xiaoyu, et al. \"Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting.\" arXiv preprint arXiv:2402.07207 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecimtwQNzE", "reviewer": "Reviewer_FWvW", "summary": "This paper presents a simple and effective method to address the Janus Problem for Score Distillation Sampling (SDS)-based text-to-3D methods. This paper argues view consistency is related to that the 3D model tends to learn content with higher certainty from each perspective, and using different priors with different certainty will help with more consistent generation. Specifically, they assume that the certainty of one denoising step follows a Gaussian distribution and design additional loss to constrain the distribution so that different viewpoints have different distributions. Experiments illustrate that their proposed method alleviates the Janus Problem.", "questions": "1. Can you provide a quantitative performance comparison with the previous work on the Janus Problem?\n\n2. Is there a quantitative analysis of the advantages of training time cost compared to DreamControl?", "limitations": "The author mentioned the limitation with regard to the modeling of multiple objects or complex objects. Appendix H also includes a failure example of the multiple object issue.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple and effective. Compared to previous work, this method does not require additional data or models, and can be well integrated into the computation of the original SDS loss.\n\n2. This paper is well-motivated by a detailed analysis of the Janus Problem and the underlying reason for decoupling the relationship between viewpoints and distributions.\n\n3. The empirical experimental results show that it can effectively alleviate the Janus Problem.", "weaknesses": "1. One major concern is that this paper lacks enough comparison with previous work. Although in Figure 6, the authors mention several advantages (on the resources, universality, and training efficacy), comparing against the empirical performance quantitatively is still useful. The model may not outperform other models that have more requisites, but it is still important to know where we stand.\n\n2. In addition, for the comparison with DreamControl, the main difference is that this model does not need an additional fine-tuning stage. Because it does not affect the inference time consumption (which is in general more important when we consider the efficacy issue), the contribution will be a bit limited if the proposed method does not outperform DreamControl.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your effort in reviewing our work! Our responses are as follows:\n\n## 1. Quantitative Comparison with Other Methods Dealing with the Janus Problem\n\nWe have provided the quantitative comparison results with other methods in the table in *Rebuttal File* (as well as in the table below). It can be observed that our method can comprehensively surpass or partially match the performance of other methods. Specifically, our method shows the best suppression of the Janus Problem when embedded in dreamfusion; when embedded in prolificdreamer, it achieves the best overall performance, strongly suppressing the Janus Problem while maintaining the highest CLIP score.\n\nNotably, our method can achieve good performance without requiring additional 3D priors or multi-stage fine-tuning, consuming minimal extra computational power (see quantitative comparison in the next response) and can be directly applied to SDS-based text-to-3D methods.\n\n\n| Method           | No Additional Prior | Single Stage | No Fine-tuning | No Object-specificity | JR(%)↓ | CS(%)↑ |\n|------------------|---------------------|--------------|----------------|-----------------------|--------|--------|\n| Zero-1-to-3      | ✘         | ✘            | ✘              | ✔                     | 23.33       | 22.94       |\n| MVDream          | ✘        | ✘            | ✘              | ✔                     |   20.00     | 26.31       |\n| Prep-Neg         | ✔                   | ✔            | ✔              | ✘                     | 26.67       |  26.23      |\n| D-SDS            | ✘             | ✔            | ✔              | ✔                     | 23.33       | 24.82       |\n| DreamControl     | ✔                   | ✘            | ✘              | ✔                     |20.00        |  28.03      |\n| Dreamfusion      |                     |              |                |                       |        |        |\n| Origin           | ✔                   | ✔            | ✔              | ✔                     | 56.67  | 22.73  |\n| LCGen (Ours)     | ✔                   | ✔            | ✔              | ✔                     | **16.67**  | 22.95  |\n| Magic3D          |                     |              |                |                       |        |        |\n| Origin           | ✔                   | ✔            | ✔              | ✔                     | 46.67  | 23.77  |\n| LCGen (Ours)     | ✔                   | ✔            | ✔              | ✔                     | 23.33  | 23.61  |\n| ProlificDreamer  |                     |              |                |                       |        |        |\n| Origin           | ✔                   | ✔            | ✔              | ✔                     | 63.33  | 26.23  |\n| LCGen (Ours)     | ✔                   | ✔            | ✔              | ✔                     | 20.00  | **28.94**  |\n\n\n\n## 2. Quantitative Comparison of Computation Cost with DreamControl\n\nWe conducted a quantitative comparison of computation costs with Dreamcontrol. Notably, in the text-to-3D process, Dreamcontrol consists of two stages: Stage 1 - 3D Self-Prior Generation and Stage 2 - Control-Based Score Distribution. However, only the code for the latter has been released in their official codebase, while the former remains on their to-do list. Using Stage 2 alone cannot directly perform text-to-3D generation when given a text prompt. According to the official instructions, we need to provide an obj file or a threestudio checkpoint as a condition during the inference stage. Therefore, in the quantitative computation process for text-to-3D, we need to consider the costs of both stages (baseline+stage 2 or stage 1+stage 2).\n\nWe used prolificdreamer as the baseline model for both methods, since it is one of the best text-to-3D methods currently. We conducted our experiments on a single NVIDIA RTX A6000 GPU with the max steps set to 10,000. **It is worth noting that the following data were all calculated during the text-to-3D generation (i.e., generating the corresponding 3D representation from the given text, not include pre-training time), which can be considered as the inference time in other methods, since the trainable params are the 3d representation we need.** The quantitative comparison of computational overhead in text-to-3D task for one sample is as follows:\n\n| Method           | Trainable params in text-to-3D ↓ | Total estimated model params size ↓ | GPU memory usage ↓ | Text-to-3D Generation Runtime ↓ |\n|------------------|---------------------|--------------|----------------|-----------------------|\nProlificdreamer (Baseline) | 15.1 M | 60.384 M | 28366 M |  1h30min38s | \n**DreamControl (Stage2)** | *17.6 M* | *70.422 M* | *35554 M* | *1h55min21s*|\n**DreamControl (Baseline + Stage2)** | *32.7 M* | *130.806 M* | *35554 M* | *> 3h25min* |\n**LCGen (Ours)** | *15.1 M* | *60.384 M* | *31458 M* | *1h35min54s* |\n\n\nIt can be observed that compared to DreamControl, our method: \n\n1) Our method does not require two-stage processing, and the model parameter count, and runtime are reduced by more than 50% compared to DreamControl, GPU memory usage is also lower than DreamControl. \n\n2) When comparing DreamControl (Stage 2) and LCGen (Ours), our method also performs better than DreamControl in all computational overhead metrics. \n\n3) According to the Table in response 1, the performance of our method is not inferior to DreamControl across all metrics.\n\n\n*Thank you again for your efforts and suggestions! We will include detailed discussions on the above points in the Main Paper and Appendix.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hS6qwp4i9F", "reviewer": "Reviewer_wiP5", "summary": "The paper attempts to address the Janus Problem in SDS-based text-to-3D methods. It first analyzes the cause of the Janus Problem in SDS-based approaches, identifying that discrete view encoding and shared priors in 2D lifting are the primary causes. To address this, it proposes the LCGen method, which guides text-to-3D generation to obtain different priors with varying certainty from different perspectives, thereby ensuring view consistency. Experiments show that the LCGen method can be seamlessly integrated into various SDS-based text-to-3D methods, effectively mitigating the Janus Problem without significant side effects.", "questions": "See weaknesses", "limitations": "n/a", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work attempts to address the challenge of the Janus Problem by tackling its root causes and identifying the key factors that contribute to this issue, making a significant contribution to solving this critical problem.\n\n2. The proposed method addresses the issue from a novel perspective, and experimental results have confirmed its effectiveness.", "weaknesses": "1. The examples presented in this work are relatively homogeneous. Have you tried other types of examples? (For instance, a finely designed object)\n2. The limitations section and the appendix in the paper show some limitations and failure cases. If this method is supposed to fundamentally address the Janus problem, what are the reasons for these failure cases and limitations? I did not see much detailed discussion about this in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your effort in reviewing our work! Our responses are as follows:\n\n## 1. Diversified Examples\n\n\nIn *Rebuttal File*, we demonstrate how our method alleviates the Janus Problem in various other types of examples.\n\nIn **Fig. C and D**, we tested on \"a sunflower\" and \"a piano\" to show the effectiveness of our method when the object does not contain a \"head\". It can be observed that the original Prolificdreamer generates sunflowers and keyboards of piano on both the front and back, resulting in spatial inconsistency. After applying our method, it correctly renders the front and back images of the sunflower and piano, resolving the spatial inconsistency issue.\n\nIn **Fig. F**, we carefully designed a prompt to show the effect of our method when a large amount of descriptive language is added: \"A sleek, silver-gray dolphin leaping gracefully out of the crystal-clear ocean, its body glistening in the sunlight as it arcs through the air with joyful exuberance.\" It can be observed that the original Prolificdreamer generates two dolphin heads, while our LCGen method correctly generates the image. In Fig. G, we also carefully designed a prompt: \"A Matte painting of a Christmas cake made of cheese surrounded by a moat made of ice cream\". It can be observed that in the original prolificdreamer, the front and back of Santa Claus on the cake both have faces. However, using our method, the correct samples are generated.\n\n\n\n\n## 2. Limitation Discussion\n\nFor the failure cases, our analysis is as follows:\n\n1. When objects are in very unusual poses, our method has a certain probability of failing. For example, \"an upside-down lion.\" In this case, using the original certainty to separate different views fails because the bottom might be the highest certainty view. For such samples, we need to redesign the view-based guidance function to accommodate the generation of these special samples.\n\n2. Our method may fail in multi-object generation scenarios. This is due to the limitations of current text-to-3D methods in multi-object generation tasks. As shown in Fig. H in *Rebuttal File*, when the text is \"three corgis facing different directions,\" baseline method Prolificdreamer fails to generate three corgis correctly, with an overlap between different objects. On this basis, the current best view-consistent generation methods, such as DreamControl and our method, cannot directly address the Janus Problem. To solve this issue, the primary task is to solve the multi-object generation problem in text-to-3D, which is a significant research area. For example, a current mainstream idea is to generate individual assets for each object and then stitch them together[1]. Our method still has the potential to adapt to such methods.\n\n3. Due to the lack of 3D prior knowledge, like other SDS-based methods, our method can only model 3D representations that appear more realistic from various perspectives, but cannot ensure that these 3D representations adhere to the physical laws of the real world. For example, during the generation of octopus tentacles, since there is no difference in tentacles from different views and the model does not know how many tentacles should be generated, it may produce 3D representations that do not conform to objective reality. To address this issue, we need to endow the model with the ability to understand the 3D world. One possible approach is to collect massive 3D data and design appropriate representation forms to establish a pre-trained 3D generation model (e.g., 3D diffusion). Given the enormous data requirements and training difficulty, this requires the collective effort of the entire AIGC community.\n\n\n*Thank you again for your efforts and suggestions! We will include detailed discussions on the above points in the Main Paper and Appendix.*\n\n## Reference\n[1] Zhou, Xiaoyu, et al. \"Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting.\" arXiv preprint arXiv:2402.07207 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for their hard work! We are very honored that our work has been recognized for: 1) **significant contribution to addressing key challenges**, 2) **being well-motivated by a detailed analysis of root causes**, 3) **simplicity and effectiveness**, and 4) **good experimental results**.

## Contribution Restatement

Here, we would like to restate our core contributions:

1)	We modeled and analyzed the root causes of the Janus Problem in SDS-based text-to-3D, and designed the LCGen (Low Certainty Generation) method to alleviate the Janus Problem in single-object generation.

2)	Our method can be directly embedded into existing SDS-based text-to-3D methods, effectively alleviating the Janus Problem without compromising generation quality and with minimal computational cost.

## Main Questions

The reviewers' questions mainly focused on two aspects:

1) **Quantitative comparison with other methods dealing with the Janus Problem.** We conducted a quantitative comparison with the methods mentioned in the paper, and the specific results can be found in the table in the rebuttal file. It can be observed that our method outperforms others: it achieves the best overall performance when embedded in prolificdreamer, significantly suppressing the Janus Problem while maintaining the highest CLIP score; our method with dreamfusion achieves the best suppression of the Janus Problem. Additionally, our method does not require the introduction of prior knowledge, multi-stage training, or fine-tuning, and achieves good results with minimal additional computational cost (see response to Reviewer FWvW).

2) **Performance in diversified examples.** Our method aims to alleviate the single-object Janus Problem in SDS-based text-to-3D. We also conducted experiments on various other examples, including objects without heads, fine-designed text, and multiple objects, as detailed in the figures in the rebuttal file. The baseline method is ProlificDreamer, since it is one of the best text-to-3D methods currently. Our method can achieve spatially consistent results when dealing with objects without heads and fine-designed text. For the Janus Problem of multiple objects, as mentioned in the limitations section of the *Main Paper*, this is constrained by the text-to-3D multi-object generation capability, which will be a direction for our future research.

## Prompt Library

"a bald eagle carved out of wood", "a beagle in a detective's outfit", "a beautiful rainbow fish", "a bichon frise wearing academic regalia", "a cat with a mullet", "a ceramic lion", "a chihuahua wearing a tutu", "a chimpanzee holding a peeled banana", "a chimpanzee looking through a telescope", "a confused beagle sitting at a desk working on homework", "a corgi taking a selfie", "a cute steampunk elephant", "a DSLR photo of a baby dragon drinking boba", "a DSLR photo of a cat wearing a bee costume", "a DSLR photo of a corgi puppy", "a DSLR photo of a dog made out of salad", "a DSLR photo of a frog wearing a sweater", "a DSLR photo of a humanoid robot using a laptop", "a DSLR photo of a lion reading the newspaper", "a DSLR photo of a mouse playing the tuba", "a DSLR photo of a pig playing a drum set", "a DSLR photo of a robot dinosaur", "a fox playing the cello", "a highland cow", "a lionfish", "a pig wearing a backpack", "a red panda", "a tiger playing the violin", "a zoomed out DSLR photo of a baby dragon", "a zoomed out DSLR photo of a monkey riding a bike"

*We sincerely thank all the reviewers once again. For the detailed responses to each reviewer's questions, please refer to the individual replies.*

Meta-review: This paper addresses the Janus Problem in SDS-based text-to-3D methods by introducing the LCGen, which utilizes varied certainty priors across viewpoints to ensure view-consistent generation. The approach is designed to mitigate view inconsistencies without additional data or computational overhead, effectively integrating into existing text-to-3D frameworks.

The method stands out for its simplicity and effectiveness, as highlighted by reviewers. It innovatively tackles the root causes of the Janus Problem without the need for extra data or significant computational resources, aligning well with the current SDS framework. The empirical results, as noted, validate the approach’s ability to improve view consistency in generated 3D models.

Reviewers consistently pointed out the lack of comprehensive comparisons with existing methods, which could strengthen the paper’s impact. There is also concern about the method's performance in complex scenarios not covered in the study, such as objects without a clear 'head' orientation or those requiring fine-grained details. Enhancing the dataset diversity used in experiments and providing more detailed quantitative analyses could address these concerns. Moreover, expanding the discussion on the method's limitations and potential failure cases would provide a clearer understanding of its applicability and scope.
Decision: Accept (poster)