PlacidDreamer: Advancing Harmony in Text-to-3D Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:08:37

Abstract: Recently, text-to-3D generation has attracted significant attention, resulting in notable performance enhancements. 
Previous methods utilize end-to-end 3D generation models for initializing 3D Gaussians, and multi-view diffusion models to enforce multi-view consistency. Moreover, they employ text-to-image diffusion models to refine details with score distillation algorithms.
However, these methods exhibit two limitations. 
Firstly, they encounter conflicts in generation directions since different models aim to produce diverse 3D assets. 
Secondly, the issue of over-saturation in score distillation has not been thoroughly investigated and solved.
To address these limitations, we propose PlacidDreamer, a text-to-3D framework that harmonizes initialization, multi-view generation, and text-conditioned generation with a single multi-view diffusion model, while simultaneously employing a novel score distillation algorithm to achieve balanced saturation.
To unify the generation direction, we introduce the Latent-Plane module, a training-friendly plug-in extension that enables multi-view diffusion models to provide fast geometry reconstruction for initialization and enhanced multi-view images to personalize the text-to-image diffusion model.
To address the over-saturation problem, we propose to view score distillation as a multi-objective optimization problem and introduce the Balanced Score Distillation algorithm, which offers a Pareto Optimal solution that achieves both rich details and balanced saturation.
Extensive experiments validate the outstanding capabilities of our PlacidDreamer. 
The code will be available on GitHub.
Code will be available on Github.

Main paper: 
 introduction: 
 The task of generating 3D assets from text, known as text-to-3D, has garnered significant attention for its potential to simplify 3D creation, a process once requiring specialized knowledge. Due to the relative scarcity and lower quality of 3D data compared with 2D data, one promising approach is to adapt pre-trained 2D models for 3D generation. An optimization-based approach leveraging score distillation algorithms, which distill generative capability from pretrained 2D diffusion models to guide subsequent 3D generations, has emerged as a dominant approach in this field.Since the introduction of the first score distillation algorithm, Score Distillation Sampling (SDS) [37] , subsequent works have significantly advanced this optimization-based approach, enhancing both generation quality and speed. One significant factor influencing generation quality is the multi-face problem. To address this problem, Magic123 [38] , DreamCraft3D [47] , Consistent123 [27] , and EfficientDreamer [61] incorporate multi-view diffusion models to enhance multi-view consistency. More recently, the introduction of 3D Gaussian Splatting [22] has further optimized the pipeline with convenient initialization, faster rendering, and training speed. LucidDreamer [25] , GaussianDreamer [58] , and GSGEN [6] propose to leverage end-to-end 3D generation models [20, 35] to provide a robust 3D Gaussian initialization, thereby enhancing overall quality.Despite significant progress in the aforementioned methods, there are still two main limitations that require attention:-Conflicting Optimization Directions. The integration of multiple generative models within a single pipeline can lead to contradictory optimization directions. For instance, the score distillation guidance derived from multi-view diffusion models may be at odds with that from text-to-image diffusion models [38] , necessitating the development of specialized balancing strategies [27] or additional refinement [47] for optimal performance. Moreover, employing distinct generative models for supervision across different stages [6, 58] can cause the generative model in later stages to ignore the outputs of its predecessors and independently generate new results based on its intrinsic data interpretations. -Over-Saturation in Score Distillation. The problem of oversaturation within score distillation algorithms remains insufficiently explored and unresolved. This issue manifests as a discrepancy in color distribution between the 3D content created through score distillation techniques and the 2D images generated by diffusion inference processes. Although certain methodologies yield results with more appropriate levels of saturation, they impose a significant computational load. Examples include LoRA finetuning [53] , adversarial training [5] , or supervision in multiple spaces with iterative sampling [62] . Thus, a comprehensive understanding of the over-saturation problem and a fast yet effective score distillation algorithm is needed.To overcome these challenges, we propose PlacidDreamer, a framework that harmonizes initialization, multi-view generation, and text-conditioned generation with a single multi-view diffusion model, while simultaneously employing a novel score distillation algorithm to achieve balanced saturation. More specifically, to address the first limitation, we newly devised a Latent-Plane module. This module enhances the multi-view diffusion model by enabling fast geometry reconstruction and improving its capabilities in generating multi-view images. The reconstructed geometry is then utilized to initialize 3D Gaussian points, and the improved multiview images are used to personalize the text-to-image diffusion model with directional prompts. This coordinated approach aligns the generation directions within the pipeline with the outcomes from the multi-view diffusion model, promoting convergence and A golden retriever with a blue bowtie.A bright red fire hydrant.A leather-bound book with gold details.A pirate flag with skull and crossbones.A shiny emerald green beetle. An ivory candlestick holder.A worn-out leather briefcase. An intricate ceramic vase with peonies painted on it.A wet, vibrant beach ball.A blooming potted orchid with purple flowers. significantly enhancing the quality of the generated content. Additionally, the Latent-Plane module is designed to be training-efficient and is adaptable to a variety of multi-view diffusion models, accommodating different viewpoint configurations. To address the second limitation, we delve into the causes of over-saturation. We decompose the score distillation equation into two primary components: classifier guidance and smoothing guidance. Prior algorithms have suffered from an imbalance where classifier guidance overwhelmingly dominates smoothing guidance, leading to prevalent over-saturation. Our analysis reveals that in over 30% of instances, the optimization directions of these two guidances form obtuse angles, resulting in negative optimization along certain directions with a fixed Classifier-Free Guidance (CFG) parameter, making it challenging to maintain control over balance after multiple optimization steps. To rectify this imbalance, we propose to treat score distillation as a multi-objective optimization problem and introduce a Balanced Score Distillation (BSD) algorithm. This algorithm incorporates a multi-objective optimization solution, Multiple-Gradient Descent Algorithm (MGDA), which dynamically adjusts the optimization directions to converge at Pareto Optimal points, where the generated results exhibit both rich details and balanced saturation. We evaluate PlacidDreamer both qualitatively and quantitatively to demonstrate its effectiveness. Extensive experiments indicate the superiority of our method over previous methods. Quantitative evaluations conducted on the T3Bench [10] benchmark reveal that PlacidDreamer consistently outperforms baseline methods by a margin of at least ten points in both generation quality and alignment metrics. Furthermore, we conduct ablation studies to respectively evaluate the contribution of each proposed module to the overall quality. To further highlight the BSD algorithm's effectiveness, we replaced the score distillation algorithms in various open-source text-to-3D frameworks with BSD, while keeping other components constant. Results show that BSD consistently enhances these frameworks' performance.Our contributions can be summarized as follows:• We introduce PlacidDreamer, a novel framework designed for high-fidelity text-to-3D generation. PlacidDreamer advances a more harmonious generation process through two novel approaches: the Latent-Plane module, which enhances multi-view diffusion models, and the Balanced Score Distillation algorithm, which optimizes detail richness and saturation control.  
related works: 
 Text-to-3D Generation. DreamFields [18] initially employs a pretrained model, CLIP [40] , to guide the optimization of NeRF [34] .Trying to leverage the generative nature of the diffusion model, Dreamfusion [37] introduces SDS, a loss associated with a score function derived from distilling 2D diffusion models. Subsequent works significantly improve SDS-based methods, including those using multi-view consistent models [24, 30, 44, 61] , enhancing pipeline structures [4, 26, 32, 33, 43, 54, 58] , introducing extra generation priors [2, 15, 17, 19, 48, 56, 60] , or exploring timestep scheduling [16] . Recently, some fast-forward reconstruction models [13, 28, 30, 31, 50] have emerged for faster 3D generation. Score Distillation. Given the heavy reliance on score distillation methods in these studies, addressing associated issues with SDS is crucial. While many works [11, 21, 48, 51, 52, 59] focus on addressing over-smoothing, fewer tackle over-saturation problems. Pro-lificDreamer [53] introduces VSD, leveraging LoRA finetuning for 3D distribution modeling, which can also alleviate over-saturation. IT3D [5] handles over-saturation by training a discriminator distinguishing 3D assets from text-to-2D images. Additionally, HiFA [62] presents an iterative score distillation process for a more accurate sampling direction. It's worth noting that these methods introduce a significant computational burden, slowing down overall speed. 
preliminaries 3.1 diffusion models: 
 For discrete-time diffusion models [9, 12, 36, 45] , given a data distribution 𝑞(x) = 𝑞 0 (x 0 ), we construct a forward process with a series of distributions 𝑞 𝑡 (x 𝑡 ) = N (𝛼 𝑡 x 0 , (1 -𝛼 2 𝑡 )I) with decreasing {𝛼 𝑡 |𝑡 ∈ [0,𝑇 ], 𝑡 ∈ Z}, where 𝛼 0 = 1 and 𝛼 𝑇 ≈ 0. In the generation process, we start from x 𝑇 ∼ N (0, I) and generate a sample of the previous timestep iteratively with a denoising network 𝜖 𝜙 (x 𝑡 , 𝑡) trained by minimizing the prediction of added noise, which is given byEQUATIONwhere 𝑤 (𝑡) is to balance losses between different timesteps, 𝑡 is uniformly selected from 0 to 𝑇 , and𝜎 𝑡 = √︃ 1 -𝛼 2 𝑡 .Score distillation is an optimization-based generation method that distills knowledge from pre-trained 2D diffusion models to guide other generations, The first score distillation method, SDS [37] is denoted asEQUATIONwhere x 𝑡 = 𝛼 𝑡 x + 𝜎 𝑡 𝜖 and other symbols are defined the same as in Equation ( 1). Intuitively, this loss perturbs x with a random amount of noise corresponding to the timestep t, and estimates an update direction that follows the score function of the diffusion model to move to a higher density region.MGDA is a gradient-based algorithm used to solve multi-objective optimization problems. Multi-objective optimization refers to the task of finding a Pareto Optimal solution under multiple optimization criteria, where optimizing one objective does not deteriorate the solution of another objective during the optimization process.In the context of solving multi-objective optimization tasks, MGDA determines a descent direction that is common to all criteria. For instance, when applied to a binary-objective optimization problem optimizing L 1 (𝑥) and L 2 (𝑥), MGDA will identify the direction orthogonal to∇ 𝑥 L 1 (𝑥) -∇ 𝑥 L 2 (𝑥).The algorithm has been demonstrated to converge to a Pareto Optimal point, where there is no available optimization that can improve one objective without worsening another. Further elaboration on MGDA can be found in the work of Désidéri [8] . 
methods 4.1 pipeline: 
 Given a text prompt, we initially generate a reference image using Stable Diffusion [41] or MVDream [44] . Unlike previous methods that transform text-to-3D generation into single-view reconstruction, we do not aim to precisely reconstruct the reference object. After background removal, the reference image is fed into the multiview diffusion model to generate multi-view images. To optimize the initialization of 3D Gaussian points [22] and ensure its compatibility with the multi-view images, we introduce Latent-Plane. This module, as detailed in Section 4.2, seamlessly integrates into the latent layers of any multi-view diffusion model, enabling fast reconstruction of a volume density field and enhancement of the generated images within 40 seconds. Moreover, it is training-friendly (only requiring 4 × A6000 GPUs for 18 hours), as it directly leverages 3D spatial knowledge from pre-trained multi-view diffusion model. Subsequently, the volume density field is used to initialize the 3D Gaussian points on the object surface, while the generated multi-view images are used to fine-tune Stable Diffusion with LoRA [14] technique. During this fine-tuning process, directional prompts such as "left view" are incorporated into the original text, enhancing the diffusion model's awareness of 3D space. Finally, we supervise the splatted images using the BSD guidance from the fine-tuned In more than 30% of cases, the angle between these two guidance vectors is obtuse. In such scenarios, using a fixed CFG parameter in SDS may result in negative optimization in the 𝛿 SG direction, leading to over-saturation. However, BSD algorithm ensures that each optimization step is non-negative in both directions. (c) The integration of the Latent-Plane module with multi-view diffusion models.diffusion model. As elaborated in Section 4.3, we decompose score distillation and reveal a conflict between optimization directs, leading to color discrepancies. We suggest treating score distillation as a multi-objective optimization problem and we introduce the BSD algorithm, achieving rich details and reasonable colors.The proposed method, Latent-Plane, serves as a plug-in module compatible with various multi-view diffusion models. These diffusion models typically take an input image 𝐼 𝜋 0 of an object from a specific viewpoint 𝜋 0 and generate corresponding images 𝐼 𝜋 𝑖 from other viewpoints 𝜋 𝑖 , 𝑖 = 1, 2, . . . , 𝑁 . Inspired by ConsisNet [57] , which reinforces feature patches by utilizing coordinate relationships derived from back-projection, we hypothesize that pretrained features within multi-view diffusion models possess sufficient knowledge of the 3D space to conduct direct reconstruction.Leveraging this idea, we treat latent feature maps akin to the feature plane in the Tri-Plane [3] model, allowing us to reconstruct a volume density field. Furthermore, the reconstruction results can be utilized to enhance the latent feature map through volume rendering, thereby improving the accuracy of multi-view predictions.Multi-View Feature Gathering. Most multi-view diffusion models currently utilize the Unet [42] architecture for their denoisers. To illustrate our strategy of selecting the appropriate latent layer for inserting the Latent-Plane module, we consider the Unet architecture as an exemplar. Within the Unet architecture, assuming there are 𝐿 decoder blocks generating 𝐿 feature maps denoted as F 𝑗 ∈ R 𝐻 ×𝑊 ×𝐷 with 𝑗 indexing these maps and 𝐷 representing the feature dimension. We choose the feature map with the highest resolution and the smallest index, denoted as F 𝑘 , as it encompasses the deepest and most intricate features extracted by the model, containing comprehensive geometric information.At each timestep 𝑡, during the reverse diffusion process, we obtain 𝑁 feature maps F (𝑖 ) 𝑘 , 𝑖 = 1, 2, . . . , 𝑁 from 𝑁 viewpoints. For every point x in the 3D space, we project it onto each feature map F (𝑖 ) 𝑘 and derive its corresponding feature f (𝑖 ) 𝑘 (x) through tri-linear interpolation, denote as,EQUATIONwhere Interp_2D represents the 2D interpolation function, and Proj denotes the function for projecting 3D spatial points onto 2D planes.We then apply an additional linear layer to extract low-dimensional features from the extracted features. To better represent the features of spatial point x, we concatenate each feature with its respective camera embeddings e(𝑖 )cam and coordinate embeddings e pos (x).e (𝑖 ) (x) = Concat[Linear(f (𝑖 ) (x)), e (𝑖 )cam , e pos (x)].4.2.2 Plane-Based NeRF. So far, x has 𝑁 features gathered from 𝑁 viewpoints, serving as 𝑁 tokens, which will be mutually enhanced through the Attention layers. To obtain the volume density 𝜎 (x) for the spatial point, before feeding into the attention layer, we additionally add a token 𝜏 𝜎 corresponding to obtaining sigma. 𝜏 𝜎 is obtained by concatenating a trainable embedding e 𝜎 with embeddings of timestep 𝑡, followed by a linear layer.EQUATIONEQUATION(x), e (2) (x), . . . , e (𝑁 ) (x)],EQUATIONwhere MultiHeadSelfAttn denotes the Multi-Head Self Attention layers. After the multi-view augmentation, there are a total of N+1 tokens in 𝜏 ′ (x). For the sigma token 𝜏 ′ 𝜎 (x), we use an MLP to obtain its volume density value 𝜎 (x). For the multi-view enhanced feature tokens 𝜏 ′ (x), we conduct volume rendering to gather features enhanced on the object surface.EQUATIONEQUATIONWe directly add the feature map to the original feature map. The enhanced feature map continues to function in the decoder of the diffusion model, ultimately producing the model's predicted noise 𝜖, which is then integrated into diffusion training and inference.The training comprises two stages. In the first stage, we independently train the volume density generation module, enabling the selection of arbitrary camera viewpoints, rather than being constrained to predefined fixed camera viewpoints. Subsequently, we can calculate the depth of each spatial point relative to the selected camera. Following volume rendering, we supervise the generated occupancy map M (𝑖 ) and depth maps D (𝑖 ) , which are denoted asEQUATIONwhere, L BCE denotes the Binary Cross Entropy loss, 𝜌 represents the Pearson correlation coefficient, and 𝜆 controls the balance between the two losses. For training the feature map augmentation, we use the standard diffusion loss described in Equation (1).In this chapter, our exploration of score distillation is conducted through 2D generation experiments, where we directly optimize the pixel values of a 2D image. The use of 2D experiments allows the exclusion of various factors unrelated to score distillation algorithms, such as camera viewpoints and 3D representation selection, thereby highlighting the effectiveness of score distillation methods. Furthermore, the outcomes of 2D experiments can effectively represent the effects observed in 3D experiments [11, 53] . Our focus on the experimental results is specifically directed toward the richness of details and color saturation at two distinct time points: the optimal state of image quality (referred to as the convergence state) and the period of sustained training after reaching the convergence state (referred to as the over-trained state). In 3D generation, the lack of an early-stop mechanism frequently results in the widespread occurrence of over-training.3.1 Decomposing Score Distillation. We adopt a modeling approach different from SDS [37] to obtain our decomposition. Score distillation is an optimization-based generation method aimed at distilling knowledge from pre-trained 2D diffusion models to guide other generations. In order to utilize 2D diffusion models for guidance, it is necessary to establish a connection between the 2D image distribution 𝑝 0 modeled by diffusion models and the 3D representation distribution 𝑞. We make the assumption that the probability density distribution 𝑞(𝜃 |𝑦) for the parameters 𝜃 of the 3D representation, conditioned on the text prompt 𝑦, is proportional to the product of the conditional probability densities of its rendered images x 𝜋 0 under various viewpoint 𝜋s, denoted as,EQUATIONWe adopt the average negative logarithm of the probability,EQUATIONas the loss for neural network training, where 𝑁 is the number of 𝜋s. Consequently, as the loss decreases, we obtain a sample as described in the text prompt. We can get the gradient on 𝜃 as,EQUATIONHowever, 𝑝 0 represents the unknown distribution of general 2D images, and high-density regions of 𝑝 0 are sparsely populated [46] . Given that current diffusion models are designed to model the score function of the distribution 𝑝 𝑡 of noisy images with noise of level 𝑡, we instead leverage it to generate an optimized gradient for the noisy images x 𝑡 that is obtained by adding noise of level 𝑡 to x 𝜋 0 . Subsequently, this gradient can be back-propagated to the image x 𝜋 0 and further transmitted to the 3D parameters 𝜃 , denoted as,EQUATIONPlease note that we observe a distinction between our formula and Equation (2) from SDS [37] , particularly in the absence of the final term -𝜖 in our formulation. The first term ∇ x 𝑡 log 𝑝 0 (x 𝑡 (𝜃 )|𝑦) is a 2D score function modeled by diffusion models. In 2D diffusion generation, the score function of conditional distribution can be decomposed into the classifier guidance and the score function of an unconditional distribution:EQUATIONSo, we naturally decompose the score distillation process into two functional terms.EQUATION)EQUATIONEQUATIONwhere 𝑢 and 𝑣 control the ratio between the two terms. The first term is the classifier guidance term (𝛿 CG ). When only utilizing classifier guidance, referred to as CSD [59] , images exhibit fine details but suffer from artifacts and over-saturation, as shown in the upper right corner of Figure 2 . Therefore, the introduction of the second term aims to alleviate the artifacts caused by 𝛿 CG and guide x 0 towards the higher density region in the distribution of 2D images. As depicted in Figure 2 , starting from L CSD , as 𝑢/𝑣 decreases, the optimization directions move towards the "balance" area, the generation results become smoother. Hence, we refer to it as smoothing guidance (𝛿 SG ). However, when 𝛿 CG starts to dominate the proportion, the image tends to become overly smoothed. 4.3.2 Balanced Score Distillation. Based on our analysis, the parameters of CFG (𝑢/𝑣) in score distillation algorithms play a critical role in regulating the balance between the two guidance terms during each optimization step. However, we have observed that their effectiveness becomes highly unstable after numerous optimization steps, particularly highlighted by the failure of most score distillation methods to preserve image properties during the over-training phase. We conduct experiments of previous score distillation algorithms including SDS [37] , VSD [53] , ISM [25] , CSD [59] , SSD [48] . We find that most of them exhibit over-saturation during over-trained steps, as depicted in Figure 3 .To address this issue, we investigated the distribution characteristics of 𝛿 CG and 𝛿 SG . We observed that over 30% of the steps in the iterative optimization process exhibit a negative dot product between 𝛿 CG and 𝛿 SG , indicating obtuse angles between their optimization directions in the high-dimensional space, as shown in Figure 2 . In such instances, simply combining the two directions based on a fixed ratio may lead to the final optimization direction projecting negatively onto one of the term's optimization directions. Consequently, the control of balance becomes ineffective due to the presence of negative optimization, ultimately resulting in over-saturation or over-smoothing.Therefore, we propose to consider score distillation as a multiobjective optimization task, where the optimization objectives are L 1 = -𝜆 log(𝑝 𝑡 (𝑦|x 𝑡 )) and L 2 =log(𝑝 𝑡 (x 𝑡 )), with 𝜆 representing the hyper-parameter that controls the ratio of two guidance terms. The solution of multi-objective optimization tasks will find the Pareto optimal point, where no action can improve one objective without deteriorating another. If generation reaches the Pareto Optimal points, it will stabilize at this optimal point, and the generated content will exhibit rich details while maintaining balanced saturation. In this scenario, the hyper-parameter 𝜆 genuinely controls the final inclination towards both optimization directions, rather than attempting to control the balance at each step.We propose the Balanced Score Distillation, which employs MGDA [8] . Suppose the optimization combination of ∇ x 𝑡 L 1 andEQUATIONFollowing Désidéri [8] , 𝛼 is the solution of minEQUATIONFor binary situation like in Equation ( 20), we have closed form solution for 𝛼, which is 𝛼 = min(max(0, α), 1), whereEQUATIONThus, the final formula for BSD is:EQUATIONEQUATIONAs illustrated in Figure 3 , images generated by the BSD algorithm approximate the saturation of 2D images, showcasing smooth color transitions and detailed contours. In states of over-training, BSD maintains a harmonized saturation level, unlike other methods that suffer from loss of details due to over-saturation. A comparable Text prompt: A bull dog wearing a black pirate hat. method to BSD is VSD [53] , which exhibits fewer instances of oversaturation and finer details compared to other baseline methods.However, as annotated in the figure, VSD requires significantly longer computational time and higher GPU utilization than BSD.We provide an analysis of the relationship between our decomposition and score distillation algorithm with previous works in the Supplementary Material.  
implementation details: 
 The Latent Plane model has 𝑁 = 8 viewpoints. Given the resolution limitation of feature map F (2) to 32 × 32, it becomes imperative to regard the pixel space as continuous, ensuring accurate calculation of boundary correspondence. Even a minor difference of 0.5 pixels along the boundary could lead to notable errors. For all the MLP described in Section 4, we use a single linear layer for computing efficiency. We train the Latent-Plane module with Zero-1-to-3 [29] for 18 hours on 4 × Nvidia RTX A6000 GPUs with a batch size of 32 and loss scale 𝜆 = 0.05 with Objaverse Dataset [7] rendered by SyncDreamer [30] , filtered by RichDreamer [39] and LGM [49] . For more details, please refer to the Supplementary Materials.Quantitative comparison. T3Bench [10] serves as a benchmark for text-to-3D generation, standardizing camera poses for rendering images and utilizing pre-trained models [1, 23, 40, 55] to ensure a fair evaluation. The primary 3D representation employed in this benchmark is a textured mesh, which aids in the convergence of generated 3D samples, forming seamless surfaces. However, extracting meshes from 3D Gaussian representations poses challenges. Consequently, we employ 3D Gaussian splatting to generate RGB images directly. This choice challenges our generation's quality by not eliminating noise in direct RGB image generation. As PlacidDreamer focuses on the generation of individual objects, our evaluations are confined to the single-object part of T3Bench. Considering DreamFusion [37] , Magic3D [26] , Fantasia3D [4] , Pro-lificDreamer [53] , Latent-NeRF [33] , and SJC [51] as baseline methods, all of which were tested by T3Bench authors, He et al. [10] . The outcomes reveal that PlacidDreamer significantly outperforms the baseline methods in both generative quality and accuracy corresponding to the provided text. Our average quality score surpasses the next-highest ProlificDreamer by 11 points. Since this score is a linear transformation of the ImageReward [55] score, we annotate the original ImageReward scores, providing a clearer perspective on the superiority of our generative quality. Qualitative comparison. We also conduct qualitative comparisons on several baseline methods which also builds on 3D Gaussian Splatting [22] , including DreamGaussian [50] , GaussianDreamer [58] , GSGEN [6] , and LucidDreamer [25] . In Figure 4 , we present the results of different methods generating the same prompts. Dream-Gaussian can generate a textured mesh in several minutes, thus, its generation quality is lower compared with other methods. Gaussian-Dreamer and GSGEN leverages end-to-end 3D generation model We perform ablation studies, as illustrated in Figure 5 , and carry out quantitative evaluations using T3Bench, as detailed in Table 1 . Latent-Plane. The "Latent-Plane" module in the pipeline serves the functions of initializing 3D Gaussian points and personalizing the diffusion model. We investigate the effects of removing both functionalities (w/o Latent-Plane), removing only initialization (w/o Initialization), and removing only fine-tuning (w/o Personalization) on the experimental results. Firstly, we remove the Latent-Plane module (w/o Latent-Plane), utilizing the Point-e [35] model as the initialization model for 3D Gaussian points, following Lucid-Dreamer. Point-e struggles to comprehend the complex prompts and fails to generate a complete shape for the dog. Additionally, due to the color deviation introduced by Stable Diffusion during generation, the color of the dog's tie is inaccurate. Subsequently, we only remove the initialization function of the Latent-Plane, retaining the fine-tuning part (w/o Initialization). It can be observed that the Janus problem in the dog's head is resolved. Through extensive experimentation, we find that under the probabilistic supervision of score distillation, 3D Gaussian points do not undergo significant changes in position, making them highly sensitive to initialization. Next, we solely remove the fine-tuning part, keeping the initialization part intact (w/o Personalization). This results in a more complete structure for the dog's bodies, but issues arise with the multi-face problem and tie color. BSD. We replace BSD with SDS [37] and observe severe oversaturation and some blue artifacts. Compared with BSD, SDS makes 3D Gaussian points harder to converge. To further investigate BSD, we conduct ablation experiments on 𝜆 to validate its effectiveness in controlling the ratio of 𝛿 CG and 𝛿 SG . It can be observed that when 𝜆 = 5, the dog's color is over-saturated. As 𝜆 increases to the range of 15-25, over-saturation alleviates without significant loss of details. Only when 𝜆 reaches the value of 35 does the loss of details become apparent. However, at this point, the dog's features are still recognizable. It is worth noting that in the SDS algorithm, CFG can only be set to a large value to get recognizable outcomes. These results prove that each proposed component significantly contributes to the overall effectiveness of the framework.BSD is a general score distillation method independent of the choice of NeRF, camera perspectives, diffusion models, etc. To test its distillation ability, we integrate BSD into various 3D generation pipelines, ensuring a fair evaluation of score distillation capabilities. This included exclusively substituting alternative score distillation algorithms, without introducing additional modifications to the pipeline. Our results demonstrate that BSD enhances the output of each pipeline. Further details and results are provided in the Supplementary Material. 
conclusion: 
 We have focused on resolving conflicts in current text-to-3D approaches, including conflicts within a single model's guidance and conflicts arising from guidance provided by different models. To address these conflicts, we introduced a novel framework Placid-Dreamer with the newly designed Latent-Plane module and the Balanced Score Distillation algorithm to achieve mutual optimization. As a result, we have achieved a more harmonious and balanced text-to-3D generation process, leading to high-fidelity 3D outcomes. We hope our work will inspire further investigations into more harmonious methodologies for 3D generation.  
 
 Appendix:  


Decision: The paper presents a well-written and primarily clear description of the new PlacidDreamer framework, which focuses on harmonizing initialization, multi-view generation, and text-conditioned generation in 3D asset creation. Key to this approach is the integration of the Latent-Plane module and the novel Balanced Score Distillation (BSD) algorithm, addressing challenges like conflicting optimization directions and over-saturation. The BSD algorithm treats score distillation as a multi-objective optimization problem, resulting in richer details and improved saturation balance in generated 3D assets. The paper also provides extensive qualitative and quantitative evaluations, showcasing the superior performance and effectiveness of the PlacidDreamer framework compared to existing methods.  

For the issue raised by reviewer WJz2, the AC has double-checked Table 1 in the rebuttal; the authors have added new baselines from the T3bench and incorporated a new method, LucidDreamer, that employs Gaussian for rendering. For all the comparisons, the approach performs better than the compared approaches.  It appears that the authors also aim to highlight methods that do not require mesh conversion, like LucidDreamer, in Table 1. 

Considering all the merits, the AC recommends accepting this paper.  The Authors should clearly address the Reviewers' concerns in the final version.

Reviews: review: paper_summary: paper_summary: The paper presents a framework for text-to-3D generation. The key idea is to integrate the Tri-Plane model into the prior multiview diffusion model. The paper also designs a balanced score distillation sampling strategy by leveraging multi-objective optimization theories. Experiments are conducted on T3Bench to validate the effectiveness of the proposal.
strength_weakness: strength_weakness: S1. Well-written and mostly clear. 

S2. The proposed balanced score distillation is interesting and new.
limitations: limitations: L1: Integrating volume density field into a multi-view diffusion model is not new. It has been explored by SyncDreamer. The author should discuss the differences between them. 

L2. The comparison in Table 1 is not fair. Baseline methods convert the optimized NeRF or DMTet to a textured mesh and then render RGB images from the textured mesh to evaluate generation quality. However, the paper directly uses the rendered image from 3D Gaussian Splatting (3DGS) for evaluation. Note that extracting textured mesh from the original implicit representation (NeRF, DMTet, 3DGS) always has a huge quality degradation. Thus, it is not fair to compare the proposed PlacidDreamer and baseline methods in Table 1. This may be the reason why the average score of PlacidDreamer is significantly higher than baseline methods in Table 1. 

L3. The ablation study in Table 1 is also confusing. It seems that the baseline of PlacidDreamer is utilizing Zero-1-to-3 as a prior model and using SDS to optimize the underlying 3DGS. Building upon this, the paper introduces Latent-Plane, Initialization, Personalization, and BSD. Correct me if I am wrong. 

L4. The paper trains the Latent-Plane module with Zero-1-to-3 to obtain the prior multi-view diffusion model. The input of the prior multi-view diffusion model is an image or a text?

L5.  Where is the definition of Personalization? I can’t find any description of the proposed Personalization in Section 4.

L6. In Lines 792-793, the authors claim that PlacidDreamer focuses on the generation of individual objects. However, the PlacidDreamer seems can’t generate multiple objects or objects with surroundings. This is because the prior multi-view diffusion model is trained on Objaverse. This is a drawback of PlacidDreamer.
paper_summary: paper_summary: The authors claim that there are two primary issues in text-to-3D generation: 1) conflicts in generation direction when different diffusion models work in the same framework. 2) over-saturation problem. To overcome the conflicts in generation direction, the authors propose a Latent-Plane module to align the initiation and score distillation with the multi-view diffusion model. To alleviate the over-saturation problem, the author proposes a Balanced Score Distillation algorithm, providing a Pareto Optimal solution.
strength_weakness: strength_weakness: - This paper is well-written.
- The idea of introducing MGDA to adaptively balance classifier guidance and smoothing guidance is interesting.
- The proposed Latent Plane for lifting the multi-view diffusion model to 3D coarse reconstruction may inspire future research.
limitations: limitations: - Several studies have demonstrated the benefits of incorporating a negative prompt term in SDS, like NFSD[1]. I wondering if BCD can handle a ternary-objective optimization problem (classifier, smooth, negative) and how.
- The coarse reconstruction results from Zero-1-to-3 w. Latent-Plane modules are encouraged to be included for a better understanding of the benefits of Latent-Plane.
- Fig. 3 illustrates the efficacy of BSD in addressing over-saturation problems when over-trained. Nevertheless, BSD performs comparably to VSD, indicating the need for a metric to quantify over-saturation for improved comparisons.
- More qualitative comparisons are needed. 
- I appreciate the quantitative comparison on T3Bench. However, many methods (e.g., LucidDreamer, MVDream, etc.) are missing in Table 1.

[1] Noise-Free Score Distillation. 

Typos: 
- Eq. (18): redundant parentheses;
paper_summary: paper_summary: This paper introduces PlacidDreamer, a novel framework designed to address limitations in current text-to-3D generation methods. The primary contributions include the Latent-Plane module for harmonizing multi-view and text-conditioned generation and the Balanced Score Distillation (BSD) algorithm to manage over-saturation in generated images. By integrating these innovations, PlacidDreamer aims to unify the generation direction and achieve balanced saturation, enhancing the overall quality and consistency of 3D assets generated from textual descriptions. Extensive experiments validate the framework’s effectiveness, demonstrating superior performance over existing methods.
strength_weakness: strength_weakness: Innovative Framework: PlacidDreamer introduces a comprehensive approach to harmonizing initialization, multi-view generation, and text-conditioned generation. The integration of the Latent-Plane module and BSD algorithm addresses key challenges in the field, such as conflicting optimization directions and over-saturation in score distillation.

Balanced Score Distillation Algorithm: The BSD algorithm treats score distillation as a multi-objective optimization problem, ensuring generated 3D assets have rich details and balanced saturation, which is a significant improvement over previous methods.

Sufficient Evaluation: The paper provides extensive qualitative and quantitative evaluations, including comparisons with baseline methods and ablation studies, demonstrating the superior performance and effectiveness of PlacidDreamer.
limitations: limitations: The complexity of Implementation: The framework's integration of multiple modules and algorithms may introduce complexity, making it challenging to implement and reproduce the results without extensive computational resources.

Dependence on Pre-Trained Models: PlacidDreamer relies heavily on pre-trained 2D and 3D models, which may limit its adaptability and performance in scenarios where such models are not available or not well-suited.

Generalization Across Different Domains: The framework's effectiveness in generating diverse 3D assets across various domains is not thoroughly explored, raising questions about its generalizability and versatility in different application scenarios.
paper_summary: paper_summary: The paper introduces PlacidDreamer,  a framework that harmonizes initialization, multi-view generation, and text-conditioned generation with a single multi-view diffusionmodel, while simultaneously employing a novel score distillation algorithm to achieve balanced saturation. To unify the generation direction, the authors introduce the Latent-Plane module, a plug-in extension that enables multi-view diffusion models to provide fast geometry reconstruction for initialization and enhanced multi-view images to personalize the text-to-image diffusion model. To address the over-saturation problem, they propose to view score distillation as a multi-objective optimization problem and introduce the Balanced Score Distillation algorithm, which offers a Pareto Optimal solution that achieves both rich details and balanced saturation.
strength_weakness: strength_weakness: PlacidDreamer advances a more harmonious generation process through two novel approaches: the Latent-Plane module, which enhances multi-view diffusion models, and the Balanced Score Distillation algorithm, which optimizes detail richness and saturation control. 

Extensive experiments indicate the superiority of our method over previous methods.
limitations: limitations: This statement needs support: "In more than 30% of cases, the angle between these two
guidance vectors are obtuse.

Figure 3 lacks a detailed explanation, such as the meaning of "1.8× Forward" in it.

Line 797-800: "The outcomes reveal that PlacidDreamer significantly outperforms the baseline methods in both generative quality and accuracy corresponding to the provided text. Our average quality score surpasses the next-highest ProlificDreamer by 11 points." What metrics are referred to here? Where are the experimental results? Specific explanations are lacking.