LCGen: Mining in Low-Certainty Generation for View-consistent Text-to-3D

Authors: zeng  tao
tong  yang
junxiong  lin
xinji  mai
haoran  wang
beining  wang
enyu  zhou
yan  wang
wenqiang  zhang
janus  problem
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 07:14:34

Abstract: The Janus Problem is a common issue in SDS-based text-to-3D methods. Due to view encoding approach and 2D diffusion prior guidance, the 3D representation model tends to learn content with higher certainty from each perspective, leading to view inconsistency. In this work, we first model and analyze the problem, visualizing the specific causes of the Janus Problem, which are associated with discrete view encoding and shared priors in 2D lifting. Based on this, we further propose the LCGen method, which guides text-to-3D to obtain different priors with different certainty from various viewpoints, aiding in view-consistent generation. Experiments have proven that our LCGen method can be directly applied to different SDS-based text-to-3D methods, alleviating the Janus Problem without introducing additional information, increasing excessive training burden, or compromising the generation effect.

Main paper: 
 introduction: 
 At the forefront of the digital domain, considerable advancements have been achieved in converting textual prompts into realistic 2D images, signaling a new epoch of computational creativity [14, 5, 4, 3, 2, 36, 38, 39, 30, 31, 22, 17, 18] . However, transferring such achievements to the 3D domain introduces an additional layer of intricacy. While 3D generation technology [35, 29, 9, 24, 32] is becoming increasingly indispensable across various fields, from virtual reality to architectural design, traditional 3D content generation demands a substantial investment of time and expertise, and the difficulty in acquiring 3D data makes explicit text-to-3D modeling exceedingly challenging. In response, Score Distillation Sampling (SDS) [25] , based on 2D lifting, has emerged to simplify and advance the 3D creative process [6, 16, 23, 34] . In text-to-3D tasks, 3D representation (such as NeRF [24] ) renders and outputs the corresponding visual image of given camera viewpoints.After that, SDS employs the guidance from priors embedded within pre-trained text-conditioned 2D diffusion models to compute losses for images or latents, iteratively guiding the 3D representation to its optimality.However, this paradigm carries certain risks, with the Janus Problem [25] standing out as a significant and common issue [33, 37, 40] . This problem occurs when 3D models exhibit multiple, often conflicting viewpoints, resulting in inconsistencies with the original text conditions. As illustrated in Fig. 1 , faces appear at various positions within the same 3D object. This is an inevitable result stemming from the inherent nature of SDS-based text-to-3D approaches. Firstly, current SDS-based text-to-3D methods employ discrete viewpoint encoding. Camera perspectives are classified into regions, with each region sharing a uniform view and text guidance. Consequently, the images within each region share the same prior distribution, biasing the 3D representation towards locally optimal synthesis with the highest certainty, as shown in Fig. 2 and Sec. 3. From a global perspective, there is a high probability of bias towards synthesizing heads at multiple biased positions. Secondly, diffusion models lack diverse 3D training data and thus a nuanced understanding of 3D space [19] .For different camera views, there is a tendency to generate images with high certainty that emphasize the most characteristic features of the object [11] , such as the head [1] . These intrinsic shortcomings render the Janus Problem a dominant challenge in the text-to-3D process.Addressing Janus Problem remains a critical challenge, with numerous studies dedicated to mitigating its effects [15, 19-21, 28, 40, 12] , like DreamControl [13] and Perp-Neg [1] . However, these methods either require extensive multi-stage fine-tuning or object-specific designs and do not address the fundamental causes of the Janus Problem. In response, we have innovatively modeled and analyzed the underlying causes of the text-to-3D Janus Problem and proposed a novel approach named Low Certainty Generation (LCGen) method.Specifically, we first analyzed the causes of the Janus Problem through probability modeling. Under the paradigm of discrete viewpoint encoding, viewpoint inputs from the same region possess the same prior distribution. We modeled this distribution and analyzed the relationship between the occurrence of the Janus Problem at various biased positions and the distribution itself. By calculating probabilities, we derived the likelihood of biases occurring at specific positions, as illustrated in Fig. 2 showing probability density peaks at different biases, indicating the Janus Problem.Thus, addressing the text-to-3D Janus Problem necessitates reevaluating the relationship between views and distributions of guidance, advocating for their decoupling rather than adherence to a shared distribution. Direct explicit modeling of the distribution is quite challenging, so we sought implicit distribution constraints based on certainty learning. Here, we define the probability C(x t-1 |x t ) in Eq. 7, estimated at the current timestep, as generation certainty in diffusion. We discovered that different viewpoint images exhibit varying levels of certainty during the diffusion denoising process. For areas rich in object features (such as the front), diffusion model tends to exhibit higher certainty during the denoising process, and conversely lower elsewhere. This is linked to the data bias in pretrained diffusion models [33] (see Appendix. B). Leveraging this characteristic, we constrain the generation of 3D representations so that their input x t into diffusion model achieves certainty consistent with the viewpoint. Consequently, we obtain a decoupled data distribution for precise distribution localization of separated viewpoints, thus releasing the Janus Problem. Extensive data analysis and visualization substantiate the scientific validity and effectiveness of LCGen. Our method can be integrated into various SDS-based text-to-3D methods, consistently mitigating the Janus Problem without compromising generative performance.Our contributions are as follows:• We model and analyze the Janus Problem in text-to-3D, identifying the fundamental reasons for its occurrence. Our findings indicate that the inevitability of the Janus Problem is associated with the SDS-based text-to-3D framework that employs discrete view encoding and 2D diffusion lifting. • We specifically develop LCGen, a method that decouples the distribution of viewpoint data from the perspective of generation certainty, thereby guiding precise view localization and effectively mitigating the Janus Problem. • We conduct extensive data analysis and experiments to demonstrate the scientific validity and effectiveness of LCGen. Our method is transferable and consistently mitigates the Janus Problem across various baselines without compromising the generative quality.2 Background 
diffusion model: 
 Diffusion models [10, 27] have proven a powerful class of generative models, particularly excelling in text-to-image synthesis. Building on the progress made with text-to-image diffusion models, new approaches have been developed to extend these capabilities to 3D content generation.A diffusion model typically involves a forward process that gradually adds noise to a data sample and a reverse process that aims to reconstruct the original sample by progressively denoising it. Mathematically, the forward process is described as:EQUATIONwhere N is Gaussian distribution, β t are variance terms increasing over time, and I is the identity matrix. The reverse process, which is more pertinent to generative tasks, is modeled as:EQUATIONwhere µ Φ (x t , t) represents the mean learned by the diffusion model Φ, and σ 2 t are learned variances.Score Distillation Sampling (SDS). Score Distillation Sampling (SDS) [25] is a novel approach tailored for bridging the gap between 2D image generation and 3D model synthesis. The SDS process utilizes the gradients from a pretrained 2D diffusion model Φ to guide the generation of 3D model Θ. The key idea is to render 2D projections of a 3D model from various views and adjust the model parameters to maximize the agreement between these projections and the images generated by a text-conditioned diffusion model. Given text prompt y, SDS can be described as:EQUATION)where εΦ is the noise predicted by Φ, x is the rendered image of view c by Θ, Ω is weighting factor.Variational Score Distillation (VSD). Building upon SDS, Variational Score Distillation (VSD) [34] introduces a probabilistic framework that treats the problem of text-to-3D synthesis as a distribution optimization task. VSD seeks to create a distribution over possible 3D shapes that is likely under the given text condition, rather than finding a single deterministic shape. VSD can be expressed as:EQUATIONwhere εΘ (x t , t, c, y) is the noise predicted by rendered images. This approach allows for exploring a richer space of 3D geometries, potentially capturing more complex and diverse features consistent with the textual description.• Integration Range of I front :0 ≤ θ < π -t overhead π 2 -t front ≤ ϕ ≤ π 2 + t front • Parameters of Gaussian Distribution 3: (θ, ϕ) = π 2 , π + b ϕ , σ 3 = 0.2Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper discusses the limitations of the work performed by the authors in Sec. 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
analysis of janus problem in text-to-3d: 
 In this Section, we analyze the reasons for the Janus Problem produced by the SDS-based text-to-3D method. Through modeling, we have identified how the discrete view encoding method leads to shared distributions that cause the Janus Problem. For detailed derivation, please refer to Appendix C. In response to this finding, we have developed the LCGen method, as described in Section 4. Our modeling focuses on two variables: the camera viewpoint parameter c and the position of the head b in the 3D representation. Both variables can be considered as points on a sphere S 2 , represented by (θ, ϕ). Taking the head position as an example, our modeling yields the probability P (b) of the head appearing at each position b on the sphere. The modeling steps are as follows:As shown in Fig. 2 (a), in the SDS, the sphere S 2 is divided into different region intervals I. For each c within I, the same viewpoint text is used, resulting in the same text condition and diffusion guidance (See Appendix. B). We represent the guidance as a superposition of Gaussian distributions with the spherical position as the variable. In one-dimensional Gaussian distribution across θ or ϕ dimension, the probability p of generating a head at position b under the viewpoint c is given by:p I (b, c) = i w I i 1 2πσ I i 2 exp - (c -b -µ I i ) 2 2σ I i 2 (5)where w I i are the mixture weights for ith Gaussian component in I, µ I i and σ I i 2 are the mean and variance of the i-th Gaussian component in I, i w I i = 1 to ensure that the total probability of the mixture distribution in I integrates to one over its domain.As shown in Fig. 2 2(c ) and consider spherical integration. the probability of generating a head at position b is given by:= i w I i 1 √ (2π) k |Σ I i | exp -1 2 (c -b -µ I i ) ⊤ Σ I i -1 (c -b -µ I i ) in Fig.EQUATIONwhere I represents different regions of the sphere, dA is the differential solid angle element in spherical coordinates, and sin(θ) accounts for the area element on the sphere. Each integral I calculates the contribution to P (b) from each region I.Thus, we can use numerical integration techniques to obtain the probability of generating a head at different positions b. As shown in Fig. 2 From the analysis in Sec. 3, it is evident that the discrete view-dependent text condition can lead the 3D representation to manifest the Janus Problem at different positions b during the synthesis process. Therefore, addressing the relationship between c and guidance presents a viable method for mitigating the Janus Problem. We propose LCGen, which leverages the certainty characteristics to decouple distributions across different viewpoints, thereby mitigating the Janus Problem in the text-to-3D task. Specifically, for c ∈ I, we constrain the guidance towards p c t (x t | c, y) rather than p I t (x t | c, y).LCGen. During the diffusion denoising process, different c with different x t of the same object possess distinct certainty C(x t-1 |x t ). We decouple these to guide the synthesis process to generate images that correspond more closely to the desired viewpoint.In the diffusion model in SDS, each denoising step is considered a probabilistic inference process from the current state x t to the previous state x t-1 . This process typically relies on the following assumptions: 1) The noise ϵ follows a Gaussian distribution, which is estimated by the model at each step.2) The mapping from x t to x t-1 can be represented using a parameterized Gaussian process.Assuming we have obtained the prediction of εt through the diffusion model, we can predict the estimation of the certainty of x t-1 given x t . This estimation typically assumes that the certainty follows a Gaussian distribution:EQUATIONHere, µ c and σ c2 represent the mean and variance that guides the U-Net's prediction, thus affecting the synthetic results. µ c and σ c2 can be calculated as follows:EQUATIONHere, β t is the variance parameter at step t, α t is 1 -β t . The Gaussian distribution parameters guide the Diffusion model's prediction to minimize the Janus Problem at step t -1. The certainty function is defined as:EQUATIONBy constraining C c (x t-1 |x t ), we can ensure that different viewpoints have different distributions.We have designed the L cert as follows:EQUATION)where γ is normalization constant, G(c) is the function of view-based guidance. Back propagation. Although Eq. 10 specifies the basic form of L cert , direct backpropagation involves redundancy. In this method, the diffusion model serving as guidance is frozen, and it is unnecessary to calculate gradients for it. We hope to perform backpropagation directly on NeRF. In SDS [25] , to simplify the gradient calculation and update processes, researchers simplify the loss function L SDS , avoiding the complex gradient calculations involved with a frozen diffusion model Φ. The gradients are applied directly to NeRF Θ and its rendered images x t . In LCGen, we also specifically design the L cert to bypass the diffusion model and apply directly in the NeRF flow parameter updates.For L cert , we obtain:EQUATIONUsing the chain rule, we can expand and simplify the first term on the right side in Eq. 11. For details of the process, please refer to the Appendix D. We get:EQUATIONwhereEQUATIONBy simplifying the calculations, we enable L cert to apply directly in the NeRF flow parameter Θ updates, avoiding the complex gradient calculations of the Unet layer in diffusion model Φ, and maintaining calculation consistency with L SDS .• Integration Range of I back :0 ≤ θ < π -t overhead 3π 2 -t back ≤ ϕ ≤ 3π 2 + t back• Parameters of Gaussian Distribution 4: We implement original methods and LCGen based on threestudio [8] and a single A100 GPU, using the PyTorch framework. For Dreamfusion, Magic3D, and ProlificDreamer, the hyperparameters are all kept consistent with the default settings in config files in the repository. Specifically, according to the instructions in threestudio,EQUATION• In Dreamfusion, to prevent scene being stuffed with floaters/becoming empty, we set system.loss.lambda_sparsity=0.1.• In Dreamfusion and Magic3D, to prevent the model incorrectly treating the background as part of the object, we replace the background with random colors with a probability 0.5 by setting system.background.random_aug=true.In the experiment, we set G(c) = |ϕ|, and obtained the results after a maximum of 10,000 steps. We also attempted to embed ϕ into G. In the experiments, we found that the multi-head problem almost never appeared in the θ dimension, and setting θ did not significantly affect the results. Therefore, we ultimately chose the above G. For the sake of experimental consistency, we have chosen the Stable Diffusion 2.1 base [27] as guidance and NeRF [24] as the 3D representation in the SDS-based method. In L cert , we use γ as the normalization parameter. In our experiments, we set γ to 10, which provides the most stable mitigation of the Janus problem.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Code will be available [here] if accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 In this Section, we apply LCGen to several baseline methods of SDS-based text-to-3D, including DreamFusion [25] , Magic3D [16] , and ProlificDreamer [34] , and conduct corresponding experiments. We also compare with other methods that address the Janus Problem. Sec. 5.2 presents the effects of original methods and LCGen, including qualitative and quantitative assessments. Sec. 5.3 demonstrates the ablation of hyperparameters. Furthermore, Sec. 5.4 presents visualization results of LCGen's impact on generation certainty.We implement original methods and LCGen based on threestudio [8] and a single A100 GPU. In the experiment, we set G(c) = |ϕ| and γ to 10, and obtained the results after a maximum of 10,000 steps. For the sake of experimental consistency, we have chosen the Stable Diffusion 2.1 base [27] as guidance and NeRF [24] as the 3D representation in the SDS-based method. See Details in Appendix. E and F.Qualitative Comparison. We selected two sets of text prompts from the library [25] and conducted experiments on three SDS-based text-to-3D baseline methods, including DreamFusion-sd [25] , Magic3D-sd coarse [16] , and ProlificDreamer [34] , both without and with LCGen, with qualitative results as shown in Fig. 4 . It can be observed that, as indicated by the red boxes in Fig. 4 , the original methods have a high probability of exhibiting the Janus Problem. For instance, in the first set of examples, the beagle appears with two faces, and in the second set of cases, the fox's front and back both exhibit a cello. As previously analyzed, this is a common issue inherent to SDS-based methods, resulting from the nature of the paradigm. After applying LCGen to each method, the Janus Problem was mitigated, with the generated 3D content exhibiting spatial consistency. In particular, our method has a very strong suppressive effect on the Janus Problem that occurs on the backside of objects. Quantitative Comparison. We have conducted a quantitative analysis of the results from our three sets of baselines, as shown in Fig. 5 . The Janus Rate (JR) represents the rate at which the Janus Problem occurs and is used to measure the 3D consistency of the generation method. The CLIP-Score (CS), on the other hand, is a metric for assessing the consistency between the text and the prompt and the generated image. See Appendix. F for details. We selected 30 sets of text prompts from the library and calculate the mean score. It can be observed that our method significantly reduces the probability of the Janus Problem occurring. Meanwhile, the CS reflects that LCGen improves spatial consistency without compromising the quality of the generated images. This demonstrates the effectiveness and practicality of LCGen.LCGen vs. Other Methods addressing Janus Problem. Some current work is also designed to address the Janus Problem [15, 19-21, 28, 40, 13, 1, 12] . However, these methods either require extensive multi-stage fine-tuning or object-specific designs and do not address the fundamental causes of the Janus Problem, as shown in Fig. 6 (See more comparison in Appendix G). Compared to these methods, our approach has the following advantages: 1) It targets the essence of the Janus Problem by tapping into the multiperspective information within 2D priors without the need to introduce additional information. 2) It can be directly integrated into any SDS-based text-to-3D method.3) There is no need to alter the training paradigm, and the computational cost for certainty calculations is negligible compared to the baseline. In this Section, we conduct a visualization analysis of the LCGen and the corresponding certainty for each viewpoint. Our chosen textual example is "A corgi". We use ProlificDreamer [34] as our baseline and achieve results after a maximum of 10,000 steps. As shown in Fig. 9 , the left side shows the original ProlificDreamer results, while the right side features the ProlificDreamer using the LCGen method. For each method, the lower half presents visualized outputs showing rendering results at various azimuths ϕ with an elevation θ of -15 • . It can be observed that the original prolificdreamer generates the corgi with faces on both the front and back, which is indicative of the Janus Problem. In contrast, the 3D representation created using LCGen exhibits high spatial consistency. We have also visualized the certainty obtained from the diffusion model after inputting images of the fully trained NeRF from various viewpoints. It is shown that due to the presence of multiple faces, the certainty in the original method does not show a clear pattern of change, consistent with the analysis presented earlier. In comparison, LCGen demonstrates a more distinct pattern of certainty varying with c where the certainty is greater when |ϕ| is small and decreases as |ϕ| increases. This highlights the effectiveness of LCGen's constraints on certainty and also corroborates the relationship between certainty and c. See details in Appendix E. 
related works: 
 Methods addressing Janus Problem. Past research has explored multi-stage networks that utilize 3D priors to reduce the Janus Problem. A two-stage 2D lifting framework has been proposed in DreamControl [13] , leveraging 3D self-priors to enhance geometric consistency in 3D generation. Other approaches, such as Perp-Neg [1] , innovate by using negative prompts in diffusion models to remove undesirable attributes or views while maintaining the core concept. However, these approaches either necessitate extensive multi-stage pre-training or are tailored to specific objects, without tackling the root causes of the Janus Problem. Additionally, they often require specialized and resource-intensive procedures. See details in Appendix. GQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specify all the training and test details in Sec. 5 and Appendix. E.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and discussion: 
 In this study, we initially model the Janus Problem and analyze its causes visually. We then introduce LCGen to guide text-to-3D generation toward spatial consistency by establishing varied certainty priors across viewpoints. Our method, validated through experiments, can integrate seamlessly with various SDS-based text-to-3D methods to mitigate the Janus Problem. It does so without adding extra data requirements, excessive computational overhead, or degrading the quality of generated outputs.Limitations and Broader Impacts. Our method performs well in generating individual objects but has limitations with complex multi-object scenes. Due to the lack of 3D training data resulting in an insufficient understanding of the 3D world, LCGen still has some failure cases in the text-to-3D Janus Problem. Additionally, using a fixed G leaves room for improvement in adaptive generation. At the same time, realistic AI-generated content may have adverse social impacts. Like other generative model researchers, we must remain vigilant and take precautions against generating false content. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Statistical significance bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper discusses potential societal impacts the work performed in Sec. 7. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No released data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
no janus problem: 
 The main cause of the Janus Problem in SDS-based text-to-3D methods is their reliance on high-certainty 2D priors for 3D representation, which can result in heads appearing from multiple viewpoints. To address this, we introduced the LCGen method, using low-certainty generation to align viewpoints with the optimization direction.The Janus Problem is a common issue in SDS-based text-to-3D methods. Due to view encoding approach and 2D diffusion prior guidance, the 3D representation model tends to learn content with higher certainty from each perspective, leading to view inconsistency. In this work, we first model and analyze the problem, visualizing the specific causes of the Janus Problem, which are associated with discrete view encoding and shared priors in 2D lifting. Based on this, we further propose the LCGen method, which guides text-to-3D to obtain different priors with different certainty from various viewpoints, aiding in view-consistent generation. Experiments have proven that our LCGen method can be directly applied to different SDS-based text-to-3D methods, alleviating the Janus Problem without introducing 
 
 Appendix: We generated 100 2D images for each text condition y of I in Stable Diffusion 2.1 base [27] , with the texts y being "a pig wearing a backpack, front view", "a pig wearing a backpack, side view", "a pig wearing a backpack, back view", and "a pig wearing a backpack, overhead view". For the generated images, we manually assessed and categorized the camera parameters in the elevation θ and azimuth ϕ dimensions, obtaining the results shown in Fig. 10 .It can be observed that the images generated by the 2D diffusion model have some variation in view distribution corresponding to different y of I, exhibiting different peaks. From the θ dimension, in the front view, almost all (99%) of the generated 2D images are at θ = 0. If the text is changed to an overhead view, 22% of the images are above θ = 0. In the azimuth dimension, when the view is front, 75% of the generated images are concentrated at ϕ = 0; when the view is back, 44% of the images are at ϕ = 180, while 29% are at ϕ = 0; for the side view, the distribution is relatively scattered, generally symmetrically distributed around ϕ = 0, exhibiting diverse view characteristics.This corresponds to the content in Sec. 3. When different c within I are given the same y, the 2D diffusion guidance follows the same distribution. In this work, we model this distribution as a combination of Gaussian distributions, as shown in Eq. 5.According to Sec. 3, SDS-based text-to-3D methods utilize a view-dependent prompt approach with discrete encoding. In the context of the sphere S 2 , based on elevation and azimuth, Camera parameters c are classified into intervals I representing four view categories: side, front, back, and overhead, as shown in Fig. 2 (a).However, text prompts are generally limited to high-level descriptions and do not offer the precise control achievable in actual photography. This issue largely stems from the lack of detailed data in diffusion training datasets, where specific camera settings are rarely documented [7] . As a result, images generated based on viewpoint text conditions in the diffusion model tend to be imprecise and show significant viewpoint fluctuations. Images of the front may be generated under guidance from textual prompts of different viewpoints [1] . To address this, the conditional probability density function p I t (x t | c, y) for each interval I is modeled as a mixture of multivariate Gaussian distributions, each parameterized byN I i (µ I i , Σ I i ).Specifically, taking a one-dimensional Gaussian distribution as an example, for camera parameter c ∈ I, the probability density for synthesizing a head at a position with bias b ∈ S 2 is denoted as follows:EQUATIONwhere: -w I i are the mixture weights for each Gaussian component in interval I, -µ I i and σ I i 2 are the mean and variance of the i-th Gaussian component,i w I i = 1 to ensure that the total probability of the mixture distribution integrates to one over its domain.As shown in Fig. 2 (b), the integral of p I with respect to c over the interval I is given by: We now extend our analysis to the multivariate Gaussian distribution, as shown in Fig. 2 (c ). This extension actually incorporates several inherent settings regarding camera geometry in 3D space, as shown in Fig. 11 .EQUATION1. The camera's "lookat" axis intersects the world coordinate origin, effectively fixing two rotational degrees of freedom of c, 2. The "up" axis is aligned with the z-axis of the world coordinate system, thereby constraining another rotational degree of freedom, 3. The assumptions hold under a constant distance between the camera and the world coordinate origin, which fixes the translational freedom along the radial axis from the origin-a parameter that does not explicitly influence the perspective modeling.Given these conditions, the remaining camera parameters c can be described by two active degrees of freedom: elevation θ and azimuth ϕ. If we evaluate the impact of the vector b on this distribution over a specified interval I, the resulting integral P I (b) can be interpreted as the probability density of b over that interval:EQUATIONHere, k = 2 is the dimension of the space, c ∈ I and b ∈ S 2 are vectors defining the position in this space, and µ I i and Σ I i represent the mean vector and covariance matrix of the distribution, respectively.Recognizing that different intervals I yield distinct probability functions, the total probability density P (b) for the Janus Problem occurring at the position defined by b ∈ S 2 is calculated as the sum of integrals over distinct regions I of the sphere: For L cert , we obtain:EQUATION)Using the chain rule, we first calculate the partial derivative with respect to x t , involving derivatives of µ c and σ c . Given µ c and σ c , since σ c does not depend on x t , its derivative is zero. The derivative of µ c is:EQUATION)Since σ c is independent of x t , inserting into the expression for C c (x t-1 |x t ) and using properties of the Gaussian distribution, we have:EQUATIONSince G(c) is independent of x t , incorporating the derivative into ∂Lcert ∂xt :EQUATIONSubstituting the expression for µ c , we get:EQUATIONwhereEQUATIONE Implementation Details • Integration Range of I side :0 ≤ θ < π -t overhead ϕ < π 2 -t front or ϕ > 3π 2 + t back and 0 ≤ θ < π -t overhead π 2 + t front < ϕ < 3π 2 -t back • Parameters of Gaussian Distribution 1: (θ, ϕ) = π 2 , b ϕ , σ 1 = 0.3• Parameters of Gaussian Distribution 2:(θ, ϕ) = π 2 + b θ , π 2 + b ϕ , σ 2 = 0.3In Sec. 5.4, to make the results clearer, we smooth the values at each position. Specifically, we take the average certainty of all points within a distance of 0.1 from the given position and use this average as the certainty of the scatter point. This helps to avoid disturbances in the visualization results caused by factors such as random view sampling and t.In the text-to-3D task, it is difficult to quantitatively evaluate the generated results. This task lacks ground truth for objective scoring. In this work, our goal is to reduce the occurrence of the Janus Problem, so it is necessary to assess the 3D consistency of the generated models and the extent to which the Janus Problem is mitigated. Similar to DreamControl [13] , we choose CLIP-Score (CS) and Janus Problem Rate (JR) as our quantitative metrics in Sec. 5.CLIP-Score (CS). The CLIP Score [41] is calculated using the CLIP (Contrastive Language-Image Pretraining) model [26] , which is a language-image model learned through contrastive learning. The CLIP model consists of an image encoder and a text encoder, enabling it to measure the similarity between different modalities. The CLIP Score measures the Cosine Similarity between two embedded features. This implementation utilizes the pretrained CLIP model to calculate the mean average of cosine similarities between text and generated images from different views.Janus Problem Rate (JR). To assess the consistency of 3D geometries, we count the instances of inconsistent 3D content produced by each method and calculate their proportion in all content. This represents the occurrence rate of the Janus problem (JR). [15] Image-to-3D✗ (3D data) ✗ ✗ ✓ SyncDreamer [20] Image-to-3D ✗ (3D data) ✗ ✗ ✓ Wonder3D [21] Image-to-3D ✗ (3D data) ✗ ✗ ✓ EfficientDreamer [40] Image-to-3D ✗ (3D data) ✗ ✗ ✓ Zero-1-to-3 [19] Image-to-3D ✗ (3D data) ✗ ✗ ✓ MVDream [28] Text-to-3D ✗ (3D data) ✗ ✗ ✓ Prep-Neg [1]Text-to-3D ✓ ✓ ✓ ✗ D-SDS [12] Text-to-3D ✗ (LLM) ✓ ✓ ✓ DreamControl [13] Text-to-3D✓ ✗ ✗ ✓ LCGen (Ours) Text-to-3D ✓ ✓ ✓ ✓ Table.2 presents a comparison of our LCGen method with other approaches addressing the Janus Problem. It can be observed that some other methods utilize additional prior information, some employ multi-stage training strategies and fine-tuning of pre-trained models, and some require objectspecific designs. In contrast, our method does not require any additional information and can be directly embedded into existing SDS-based text-to-3D methods without fine-tuning the models. It alleviates the Janus Problem with minimal computational cost.Examples without concept of heads. In Fig. 13 A and B, we show the results of 'a street lamp' and 'a tree' as you mentioned. It can be seen that both the original Prolificdreamer and our method achieve spatial consistency generation. Considering that the above two examples do not have obvious differences between different views, we conducted experiments on 'a sunflower' and 'a piano' as shown in Fig. C and D in Fig. 13 . It can be seen that the original Prolificdreamer produced spatial inconsistencies, showing multiple frontal images of the flower and multiple keyboards in single piano from different views. Our method successfully alleviated this issue, generating a sunflower and piano with correct front and back images. This indicates that our method is effective not only for objects with heads but also for spatial consistency modeling of other objects.Multiple objects with multiple heads. We will supplement the discussion details in the limitations section of the Main Paper. Multi-object generation is another important and challenging field in text-to-3D tasks. For SDS-based methods, the current most important issue in multi-object generation is not the Janus problem but how to handle the relationships between different objects. In H in Fig. 13 , we show the results of 'three corgis facing different directions.' It can be seen that the original Prolificdreamer cannot correctly generate 3 corgis, and there is some degree of sticking between the objects. Both our method and the current state-of-the-art multi-view consistent generation method DreamControl cannot correctly handle this example. Once the multi-view generation issue is resolved, our work will have more exploration possibilities.A large amount of descriptive language. In Fig. 13 F: "A sleek, silver-gray dolphin leaping gracefully out of the crystal-clear ocean, its body glistening in the sunlight as it arcs through the air with joyful exuberance." It can be observed that the original Prolificdreamer generates two dolphin heads, while our LCGen method correctly generates the image. In Fig. 13 G, we also carefully designed a prompt: "A Matte painting of a Christmas cake made of cheese surrounded by a moat made of ice cream". It can be observed that in the original prolificdreamer, the front and back of Santa Claus on the cake both have faces. However, using our method, the correct samples are generated. Although experiments demonstrate that our method excels in mitigating the Janus Problem, there are still some failure cases. Fig. 14 shows a failure example. By incorporating our method into ProlificDreamer [34] and setting the text prompt y as "an astronaut riding a kangaroo", the method correctly generated a single human head and a single kangaroo head. However, it resulted in issues such as an incorrect human arm position and multiple kangaroo legs. This may be due to the presence of multiple objects in the generated text, making it difficult to separate the combinations of objects from the perspective of certainty.When objects are in very unusual poses, our method has a certain probability of failing. For example, "an upside-down lion." In this case, using the original certainty to separate different views fails because the bottom might be the highest certainty view. For such samples, we need to redesign the view-based guidance function to accommodate the generation of these special samples.Our method may fail in multi-object generation scenarios. This is due to the limitations of current textto-3D methods in multi-object generation tasks. As shown in Fig. 13 H, when the text is "three corgis facing different directions," baseline method Prolificdreamer fails to generate three corgis correctly, with an overlap between different objects. On this basis, the current best view-consistent generation methods, such as DreamControl and our method, cannot directly address the Janus Problem. To solve this issue, the primary task is to solve the multi-object generation problem in text-to-3D, which is a significant research area.Due to the lack of 3D prior knowledge, like other SDS-based methods, our method can only model 3D representations that appear more realistic from various perspectives, but cannot ensure that these 3D representations adhere to the physical laws of the real world. For example, during the generation of octopus tentacles, since there is no difference in tentacles from different views and the model does not know how many tentacles should be generated, it may produce 3D representations that do not conform to objective reality. To address this issue, we need to endow the model with the ability to understand the 3D world. One possible approach is to collect massive 3D data and design appropriate representation forms to establish a pre-trained 3D generation model (e.g., 3D diffusion). Given the enormous data requirements and training difficulty, this requires the collective effort of the entire AIGC community."a bald eagle carved out of wood", "a beagle in a detective's outfit", "a beautiful rainbow fish", "a bichon frise wearing academic regalia", "a cat with a mullet", "a ceramic lion", "a chihuahua wearing a tutu", "a chimpanzee holding a peeled banana", "a chimpanzee looking through a telescope", "a confused beagle sitting at a desk working on homework", "a corgi taking a selfie", "a cute steampunk elephant", "a DSLR photo of a baby dragon drinking boba", "a DSLR photo of a cat wearing a bee costume", "a DSLR photo of a corgi puppy", "a DSLR photo of a dog made out of salad", "a DSLR photo of a frog wearing a sweater", "a DSLR photo of a humanoid robot using a laptop", "a DSLR photo of a lion reading the newspaper", "a DSLR photo of a mouse playing the tuba", "a DSLR photo of a pig playing a drum set", "a DSLR photo of a robot dinosaur", "a fox playing the cello", "a highland cow", "a lionfish", "a pig wearing a backpack", "a red panda", "a tiger playing the violin", "a zoomed out DSLR photo of a baby dragon", "a zoomed out DSLR photo of a monkey riding a bike" NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper addresses the Janus Problem in SDS-based text-to-3D methods by introducing the LCGen, which utilizes varied certainty priors across viewpoints to ensure view-consistent generation. The approach is designed to mitigate view inconsistencies without additional data or computational overhead, effectively integrating into existing text-to-3D frameworks.

The method stands out for its simplicity and effectiveness, as highlighted by reviewers. It innovatively tackles the root causes of the Janus Problem without the need for extra data or significant computational resources, aligning well with the current SDS framework. The empirical results, as noted, validate the approach’s ability to improve view consistency in generated 3D models.

Reviewers consistently pointed out the lack of comprehensive comparisons with existing methods, which could strengthen the paper’s impact. There is also concern about the method's performance in complex scenarios not covered in the study, such as objects without a clear 'head' orientation or those requiring fine-grained details. Enhancing the dataset diversity used in experiments and providing more detailed quantitative analyses could address these concerns. Moreover, expanding the discussion on the method's limitations and potential failure cases would provide a clearer understanding of its applicability and scope.

Reviews: review: paper_summary: paper_summary: This paper presents a method to tackle the issue of the Janus Problem in text-to-3D content generation method. A method named LCGen has been proposed that focuses on low certainty regions to generate view-consistent generation.
strength_weakness: strength_weakness: Proposed method encourage single head, but there could be scenarios where multiple heads needs to be generated e.g.  three corgis facing different directions.
What about those examples where there is no concept of head. How this method will behave in those situations e.g a street lamp, tree etc.
It seems that the proposed low uncertainty approach would lead to low quality samples where fine-grain details are important.
There is no comparison with the existing approaches mitigating Janus problem. A table in the main paper with metrics CS and Janus Rate is a MUST. I acknowledge Table in the appendix G, where only functional level comparison is given. Quantitative comparison is missing.
questions: questions: Please see the limitations and weakness sections, answer those questions.
Why is there a blue shade in the generated corgi in your method in fig 7 and 10 ?. Is this some kind of artefact due to low uncertainty generation ?
limitations: limitations: It seems the method will work on single object generations. It is also not clear how good this method is compared to the existing method quantitatively.
Examples used for visual validation are also limited corgi and pig, It is not clear which 30 prompts are selected from the library for the experiments. No clarity on the dataset point of view. A clear selection of data samples MUST be added for reproducibility and validation of the proposed method. 
Many generated samples in fig 10 are not even of pig.
Only one failure case is shown, while there could be many other scenarios, a detailed analysis of failure cases is also missing. It is important to demonstrate the limitations within which method will work.
While the evaluation is mostly based on the visuals or derived metrics from visuals. The main paper and appendix both doesn’t have sufficient diverse visual results to validate the proposed method
paper_summary: paper_summary: This paper presents a simple and effective method to address the Janus Problem for Score Distillation Sampling (SDS)-based text-to-3D methods. This paper argues view consistency is related to that the 3D model tends to learn content with higher certainty from each perspective, and using different priors with different certainty will help with more consistent generation. Specifically, they assume that the certainty of one denoising step follows a Gaussian distribution and design additional loss to constrain the distribution so that different viewpoints have different distributions. Experiments illustrate that their proposed method alleviates the Janus Problem.
strength_weakness: strength_weakness: 1. One major concern is that this paper lacks enough comparison with previous work. Although in Figure 6, the authors mention several advantages (on the resources, universality, and training efficacy), comparing against the empirical performance quantitatively is still useful. The model may not outperform other models that have more requisites, but it is still important to know where we stand.

2. In addition, for the comparison with DreamControl, the main difference is that this model does not need an additional fine-tuning stage. Because it does not affect the inference time consumption (which is in general more important when we consider the efficacy issue), the contribution will be a bit limited if the proposed method does not outperform DreamControl.
questions: questions: 1. Can you provide a quantitative performance comparison with the previous work on the Janus Problem?

2. Is there a quantitative analysis of the advantages of training time cost compared to DreamControl?
limitations: limitations: The author mentioned the limitation with regard to the modeling of multiple objects or complex objects. Appendix H also includes a failure example of the multiple object issue.
paper_summary: paper_summary: The paper attempts to address the Janus Problem in SDS-based text-to-3D methods. It first analyzes the cause of the Janus Problem in SDS-based approaches, identifying that discrete view encoding and shared priors in 2D lifting are the primary causes. To address this, it proposes the LCGen method, which guides text-to-3D generation to obtain different priors with varying certainty from different perspectives, thereby ensuring view consistency. Experiments show that the LCGen method can be seamlessly integrated into various SDS-based text-to-3D methods, effectively mitigating the Janus Problem without significant side effects.
strength_weakness: strength_weakness: 1. The examples presented in this work are relatively homogeneous. Have you tried other types of examples? (For instance, a finely designed object)
2. The limitations section and the appendix in the paper show some limitations and failure cases. If this method is supposed to fundamentally address the Janus problem, what are the reasons for these failure cases and limitations? I did not see much detailed discussion about this in the paper.
questions: questions: See weaknesses
limitations: limitations: n/a

Comments: title: Response to Author's response
Still happy to have further discussion
Happy to have further discussion
comment: Dear Reviewers,

Thank you for your efforts and valuable suggestions! We will continue revising the paper based on your feedback. Thank you once again to everyone!

Best regards,

Authors of submission 898
I read other reviews and the authors' responses. It seems the authors addressed my issues. Other reviews now also have positive scores.
Dear Reviewers sRbF, FWvW, and wiP5,

As today marks the final day of the Author-Reviewer Discussion phase, we are still here and welcome any further discussion or feedback you may have on our paper. We sincerely look forward to your feedback. Your insights are incredibly valuable to us.

Thank you sincerely for all the time and effort during the review process.

Best,

Authors of submission 898
Dear Reviewer sRbF, FWvW, and wiP5,

Thank you for your efforts in reviewing our paper! We greatly appreciate the valuable feedback provided and carefully posted our response. We understand that the reviewers may have very busy schedules. However, we are happy to have further discussion and answer any further questions in the Author-reviewer discussion phase. Looking forward to your feedback!

Best,

Authors of submission 898