DreamLCM: Towards High Quality Text-to-3D Generation Via Latent Consistency Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 17:52:44

Abstract: Recently, the text-to-3D task has developed rapidly due to the appearance of the SDS method. However, the SDS method always generates 3D objects with poor quality due to the over-smooth issue. This issue is attributed to two factors: 1) the DDPM single-step inference produces poor guidance gradients; 2) the randomness from the input noises and timesteps averages the details of the 3D contents. 
   In this paper, to address the issue, we propose DreamLCM which incorporates the Latent Consistency Model (LCM). DreamLCM leverages the powerful image generation capabilities inherent in LCM, enabling generating consistent and high-quality guidance,~\ie, predicted noises or images. Powered by the improved guidance, the proposed method can provide accurate and detailed gradients to optimize the target 3D models.
   In addition, we propose two strategies to enhance the generation quality further. Firstly, we propose a guidance calibration strategy, utilizing Euler solver to calibrate the guidance distribution to accelerate 3D models to converge. Secondly, we propose a dual timestep strategy, which helps DreamLCM to increase the consistency of guidance and optimize 3D models from geometry to appearance. Experiments show that DreamLCM achieves state-of-the-art results in both generation quality and training efficiency.

Main paper: 
 related work 2.1 diffusion models: 
 Diffusion Models(DMs) have emerged as powerful tools for image generation [10, 29, 31, 34, 40, 42] , excelling in denoising noisecorrupted data and estimating data distribution scores. The stable ability of DMs for generating high-quality images led to multiple applications in various domains, including video [8, 11, 19] and 3D [32, 44] , 𝑒𝑡𝑐. During inference, these models employ reverse diffusion processes to gradually denoise data points and generate samples. In comparison to Variational Autoencoders (VAEs) [20, 39] and Generative Adversarial Networks (GANs) [7] , diffusion models offer enhanced training stability and likelihood estimation. However, their sampling efficiency is often hindered. Discretizing reversetime SDE [6, 42] or ODE [42] are proposed to handle the challenge, various techniques such as ODE solvers [24, 25, 40, 48] , adaptive step size solvers [14] , and predictor-corrector methods [42] have been proposed. Notably, the Latent Diffusion Model(LDM) [35] conducts forward and reverse diffusion processes in the latent data space, leading to more efficient computation. The Consistency model [26, 41] demonstrates promising potential as a rapid sampling generative model for generating high-quality images in a single-step inference. In this paper, we transfer the ability of LCM to text-to-3D task to generate high-quality guidance. Meanwhile, we use Euler Solver as the numerical ODE Solver to further calibrate the guidance for higher quality.This task targets generating 3D contents from given text prompts. The 3D contents are parameterized by various 3D representations, including implicit representations [1, 2, 5, 16] , 3D Gaussians [4, 21, 22, 43, 46] , 𝑒𝑡𝑐. Existing methods includes 3D generative methods [15, 30] . However, these methods can only generate objects within limited categories due to the lack of large-scale datasets. Our method uses DMs to guide the 3D optimization. DreamField [13] represent pioneering efforts in training Neural Radiance Fields (NeRF) [28] with guidance from CLIP [33] . Dreamfusion [32] firstly employs Score Distillation Sampling (SDS) to distil 3D assets from pretrained text-to-image diffusion models. SDS has become integral to subsequent works, with endeavours aiming at enhancing 3D representations, addressing inherent challenges such as the Janus problem, and mitigating the over-smooth effect observed in SDS. Recent studies like ProlificDreamer [45] , HiFA [49] , and LucidDreamer [22] have made significant strides in improving the SDS loss. Concurrent methods such as CSD [47] and NFSD [17] provide empirical solutions to enhance SDS. In our novel approach, DreamLCM, we resolve the over-smooth problem in a new perspective of the guidance model, showing that it is possible to generate high-quality 3D models without any alterations to SDS. 
revisiting consistency models: 
 The core idea of the Consistency Model (CM) and Latent Consistency Model (LCM) is to learn a function that maps any points on a trajectory of PF-ODE [42] to that trajectory origin, i.e., the solution of PF-ODE. The trajectory origin indicates the real data distribution region, which has the highest data density. Besides, LCM extends the denoising process to the latent space. LCM predicts the solution of PF-ODE by introducing a consistency function in a single-step inference. LCM is a text-to-image DM 𝒇 𝜙 parameterized by 𝜙. The objective is to fulfill the mapping: 𝒇 𝜙 (𝒙 𝑡 , 𝒚, 𝑡) → 𝒙 0 , where 𝒙 𝑡 is the noisy latent while 𝒚 is the text prompt. The self-consistency property of LCM is expressed in Eq. ( 1) asEQUATIONwhere 𝛿 is a fixed small positive number. The formula shows the consistency of the perturbed images between different timesteps.Overall, LCM has two benefits: 1) generating 𝒇 𝜙 (𝒙 𝑡 , 𝒚, 𝑡) with high quality in a single-step inference. 2) different 𝒙 𝑡 between different 𝑡 generate consistent guidance. We attribute the over-smooth issue in SDS loss to two factors in Sec. 1. The first is the low-quality guidance issue, which can be resolved by utilizing LCM to generate high-quality guidance. The second is the inconsistency issue. The issue is mitigated because the guidance generated via LCM is consistent between different timesteps. Therefore, we incorporate LCM into the text-to-3D task as the guidance model. The guidance generated by LCM exhibits high quality and high consistency. 
method: 
 In this section, we present DreamLCM for high-quality text-to-3D synthesis. First, we formulate the entire 3D generation process and analyze the issues in recent works. Then, we propose DreamLCM, Guidance Calibration, and Dual Timestep Strategy. We explain how these methods benefit the generation quality. In the initial phase, we directly generate guidance via a single LCM network. In the refinement phase, we utilize another LCM network and an Euler Solver to calibrate the guidance. We calculate the original SDS loss to update 𝜃 .Dreamfusion [32] proposes a general framework for the text-to-3D generation task. It has two important components. The first is a 3D representation, e.g., NeRF [28] , 3D Gaussian Splatting [18] , that is parameterized by 𝜃 for depicting the target 3D object Θ. The second is a pretrained text-to-image diffusion model for providing guidance information and supervising the target Θ. To bridge the 3D object and its guidance model, a differentiable renderer 𝒈 is utilized to obtain the rendered image 𝒙, which is formulated as 𝒙 = 𝒈(𝜃, 𝑐) with camera pose 𝑐. Then 𝒙 is fed into a VAE encoder [20] and perturbed by noise 𝜖. Here, we denote the latent embedding as 𝒙 for simplification. Given noisy latent 𝒙 𝑡 , timestep 𝑡, and text prompt 𝒚 as inputs, the guidance model predicts guidance gradients for updating the 3D object. The guidance prediction of DM can be expressed by two equivalent forms, i.e., 𝜖-prediction ε𝜙 (𝒙 𝑡 , 𝒚, 𝑡) and 𝑥-prediction x𝑡 0 in Eq (2) following SDS [32] .EQUATIONHere x𝑡 0 =𝒙 𝑡 - √ 1-ᾱ𝑡 ε𝜙 (𝒙 𝑡 ,𝒚,𝑡 ) √ ᾱ𝑡 , √ᾱ𝑡 is the noise weight,which shows that ε𝜙 (𝒙, 𝒚, 𝑡) and x𝑡 0 are equivalent, and we consider them both as guidance. 𝜔 (𝑡) is a weighting function that depends on the timestep𝑡. 𝛾 (𝑡) = √ 1-ᾱ𝑡 √ ᾱ𝑡. The gradient leads the 3D model closer to the corresponding text prompt.Previous works ProlificDreamer [45] and LucidDreamer [22] observe that SDS generates over-smooth 3D models. We attribute this issue to two factors: 1) DMs [35, 37] generate low quality guidance because x𝑡 0 are obtained from DDPM single-step inference [9] , as shown in Fig. 1(a) ; 2) DMs are sensitive to the randomness in noise 𝜖 and timestep 𝑡. Especially, a large 𝑡 is hard to generate x𝑡 0 consistent with 𝒙 0 , which averages the appearance of the 3D models during optimization. Overall, the weakness in DMs generates poor guidance, resulting in over-smooth outcomes.The proposed DreamLCM method aims at resolving the aforementioned over-smooth issue by incorporating LCM and further enhancing the generation quality by proposing two effective strategies, i.e., Guidance Calibration and a dual timestep strategy. The overall framework is shown in Fig. 2 . The entire DreamLCM approach is depicted in Algorithm 1.For the low-quality guidance issue, DreamLCM utilizes the powerful ability of LCM to generate high-quality guidance. LCM trains a function 𝒇 𝜙 in Eq. 1, which can be seen as guidance, to map any 𝒙 𝑡 to its PF-ODE trajectory origin, i.e.. Consequently, LCM is capable of generating high-quality guidance in a single-step inference.For the inconsistency issue, we utilize the important property of LCM in Eq. 1, highlighting the consistency of guidance between different timesteps. When guided by DMs, rendered image 𝒙 0 is added random noise 𝜖, which is further weighted by different timesteps 𝑡. The randomness in 𝜖 and 𝑡 directly results in inconsistent x𝑡 0 , eventually resulting in a feature-average outcome. However, due to LCM's property, LCM can generate consistent x𝑡 0 regardless of the randomness.Overall, given a timestep 𝑠, we integrate LCM and SDS by generating guidance ε𝜙 (𝒙 𝑠 , 𝒚, 𝑠) via LCM to calculate the the SDS loss :EQUATIONwhere ε𝜙 (𝒙 𝑠 , 𝒚, 𝑠) is obtained in a single-step inference with high quality and fine details. It enables the 3D model to have fine details, mitigate the over-smooth issue, and save training costs.Unfortunately, it is difficult to resolve the two issues perfectly due to the nature of diffusion models. The images generated by LCM with a single-step inference are always blurred, and the highquality images are derived from four-step inferences iteratively. This fact is also stated in LCM [26] . For the first issue, if we directly utilize LCM's single-step inference results, the guidance would be blurred and not conducive to generating high-quality 3D models. Therefore, we further resolve this weakness by proposing a guidance calibration strategy. For the second issue, during the training of LCM, the two noisy latents in Eq. ( 1) are limited to be on the same PF-ODE trajectory, rather than two arbitrary noisy latents. We follow this protocol during the inference by fixing the noise 𝜖 ′ to perturb the rendered image, reducing the randomness in noise. Besides, we propose a decreasing timestep strategy, where we utilize monotonically decreasing timesteps during training, to reduce the randomness in timesteps. Guidance Calibration. We further dive into the principle when LCM generates guidance. We first review that in DMs, the denoising process follows a reverse stochastic differential equation(SDE):EQUATIONwhere 𝑝 𝑡 (𝒙 𝑡 ) ∼ N (𝒙 0 , 𝜎 2 𝑡 I), 𝜎 𝑡 varies along timestep 𝑡, 𝜎 𝑡 is the time derivative of 𝜎 𝑡 , w is the standard Wiener process and ∇ log 𝑝 𝑡 (x) is the score function which indicates the direction towards highest data density. And there exists a corresponding reverse ordinary deterministic equation(ODE) defined below: dx = -𝜎 𝑡 𝜎 𝑡 ∇ log 𝑝 𝑡 (x)d𝑡 .(EQUATIONwhere ∇ log 𝑝 𝑡 (x) is estimated as -1 √ 1-ᾱ𝑡 ε𝑡 0 . LCM can map any 𝒙 𝑡 on a trajectory of the PF-ODE to the origin 𝒙 * 0 , which indicates the highest-data-density region. However, the mapped origin, i.e., x𝑡 0 derived from single-step inference is always shifted. We attribute the shifting to the insufficient training of 𝒇 𝝓 . Our goal is to calibrate x𝑡 0 to get closer to 𝒙 * 0 . We consider the insufficient training in LCM and rationally assume that the denoising process of LCM follows a smooth PF-ODE trajectory with a small slope. This assumption allows us to calibrate the guidance from the perspective of PF-ODE.Based on the analysis, we propose our guidance calibration strategy, which is a two-stage strategy. We repeat the perturbing and denoising steps to calibrate the guidance distribution. In the first stage, given a perturbed sample 𝒙 𝑠 at timestep 𝑠, we first predict guidance ε𝜙 (𝒙 𝑠 , 𝒚, 𝑠), where 𝑠 is set to a small number to make the guidance more consistent with 𝒙 0 than large 𝑠. In the second stage, since the denoising process of LCM follows PF-ODE, we run a discretization step of a numerical ODE solver. we use Euler Solver to get another sample 𝒙 𝑡 :EQUATION)where 𝑡 > 𝑠. We then fed 𝒙 𝑡 to LCM network to obtain the final calibrated guidance ε𝜙 (𝒙 𝑡 , 𝒚, 𝑡). Compared to ε𝜙 (𝒙 𝑠 , 𝒚, 𝑠), the guidance ε𝜙 (𝒙 𝑡 , 𝒚, 𝑡) has two advantages: 1) it is consistent with the original rendered image 𝒙 0 because the Euler solver makes 𝒙 𝑡 and 𝒙 𝑠 on the same PF-ODE trajectory; 2) large timestep 𝑡 provides more reasonable optimization direction conditioned on 𝒚, leading x𝑡 0 closer to 𝒙 * 0 . Overall, x𝑡 0 is ensured to be consistent with both the rendered image and the PF-ODE trajectory origin conditioned on the text prompt, effectively improving the quality of the guidance. We optimize 𝜃 using the final guidance x𝑡 0 to calculate SDS loss following Eq. ( 7) asEQUATIONDual Timestep Strategy. In this paper, we incorporates 3D Gaussians [18] as the 3D representation, which requires initialization models, i.e., PointE [30] , Shap-E [15] to initialize the geometry. However, these models sometimes initialize badly, especially when given complex text prompts. Thus, properly updating the geometry positions of the 3D model is crucial in 3D generation. We propose a two-phase strategy, optimizing the geometry and appearance of 3D objects in separate phases. In the initial phase, we use large timesteps to predict large deformations to the 3D model since large timesteps keep less information in the rendered image. As a result, the guidance includes global geometry features, leading DreamLCM to optimize the geometry, where the position of Gaussian Splatting is greatly updated. In the refinement phase, we use small timesteps to optimize the appearance because small timesteps keep more information on the rendered image, generating guidance with fine local features. We propose a dual timestep strategy combining the decreasing timestep strategy with the two-phase strategy. Specifically, we define a cut-off iteration 𝑇 𝑐𝑢𝑡 and a cut-off timestep 𝑡 𝑐𝑢𝑡 , in each EQUATIONwhere 𝑇 and 𝑁 are the current iteration and total iteration. For the first 𝑇 𝑐𝑢𝑡 iterations, we optimize geometry using timesteps larger than 𝑡 𝑐𝑢𝑡 . For the remaining iterations, we use timesteps less than 𝑇 𝑐𝑢𝑡 to optimize appearance. Overall, we can see that the timestep strategy in HiFA [49] and ProlificDreamer [45] are two special cases of our timestep strategy. Experiments demonstrate that the strategy can generate high-quality 3D models with fine geometry and appearance, as shown in Fig. 5 . 
discussion: 
 Similar to the proposed DreamLCM method, ProlificDreamer [45] and LucidDreamer [22] targets resolving the over-smooth issue in SDS. They refine the SDS loss with different loss functions to alleviate the over-smoothed and over-saturated results based on SDS. We will revisit these two losses to show the relationship between DreamLCM and the two works and demonstrate that our work is more effective than theirs.ProlificDreamer is based on the SDS loss. It handles the oversmooth issue by training an additional LoRA [12] network denoted as 𝜖 𝐿𝑜𝑅𝐴 . ProlificDreamer optimizes multiple 3D models simultaneously. It aggregates and estimates their distributions by finetuneing 𝜖 𝐿𝑜𝑅𝐴 . The VSD loss for the 𝑖 𝑡ℎ 3D model is as follows:∇ 𝜃 (𝑖 ) L VSD (𝜃 (𝑖 ) ) = E 𝑡,𝝐,𝑐 𝜔 (𝑡) ε𝜙 𝒙 (𝑖 ) 𝑡 , 𝒚, 𝑡 -ε𝐿𝑜𝑅𝐴 𝒙 (𝑖 )𝑡 , 𝒚, 𝑡, 𝑐 𝜕𝒈 (𝜃 (𝑖 ) ,𝑐 ) 𝜕𝜃 (𝑖 ) , (9) where 𝒙𝑡 is the rendered image of the 𝑖 𝑡ℎ 3D model and 𝑐 is the camera condition. ε𝐿𝑜𝑅𝐴 𝒙 (𝑖 ) 𝑡 , 𝒚, 𝑡, 𝑐 indicates the distribution of the rendered image. When ProlificDreamer optimizes one 3D model, the distribution of the rendered image can be estimated as 𝜖, where 𝜖 is the noise added to the rendered image. We consider the SDS gradient as the vector starting from 𝜖 and ε𝐿𝑜𝑅𝐴 to ε𝜙 . Since ε𝐿𝑜𝑅𝐴 contains information from multiple 3D models, ε𝐿𝑜𝑅𝐴 is a steadier and more robust starting point than 𝜖, averaging the random and inconsistent features in the optimization process of each 3D model.We observe that the essential problem is the randomness and inconsistency when optimizing a single 3D model. Besides, ε𝐿𝑜𝑅𝐴 introduces extra parameters and trains several 3D models simultaneously, leading to high training costs. However, the proposed DreamLCM method incorporates LCM as the guidance model, greatly mitigating the inconsistent issue when optimizing one 3D model. As a result, there is no need for DreamLCM to train another ε𝐿𝑜𝑅𝐴 to decrease the training costs for generating high-quality 3D models. LucidDreamer proposes ISM [22] loss, which employs DDIM Inversion to enhance the quality and consistency of the guidance. Specifically, it predicts a invertible noisy latent trajectory 𝒙 𝛿 𝑇 , 𝒙 2𝛿 𝑇 , . . . , 𝒙 𝑡 , iteratively following Eq. ( 10),EQUATIONwhere 𝑠 = 𝑡 -𝛿𝑡. The guidance is obtained by a multi-step DDIM denoising process i.e., iteratingEQUATIONwhere𝜂 (𝑡) = 1- √ ᾱ𝑡 √ ᾱ𝑡. Next, by replacing x𝑡 0 in Eq. ( 2) with x0 , the SDS loss can be rewrote as∇ 𝜃 L (𝜃 ) = E 𝑐 𝜔 (𝑡 ) 𝛾 (𝑡 ) 𝒙 0 -x𝑡 0 𝜕𝒈 (𝜃,𝑐 ) 𝜕𝜃. LucidDreamer then unifies the iterative process in Eq. (10) and Eq. ( 11), proposing ISM loss as follows:EQUATIONwhere 𝜂 𝑡 includes a series of neighboring interval scores with opposing scales, which can be disregarded. ISM essentially substitutes DDPM single-step inference [9] for DDIM multi-step inference [40] to generate high-quality and high-fidelity guidance x𝑡 0 . However, the multi-step inference needs to forward the U-Net [36] in DMs multiple times, increasing training costs. Moreover, we can see that a key improvement in ISM is the quality and consistency of the guidance. Compared to LucidDreamer, DreamLCM is capable of generating high-quality and high-consistency guidance in a single-step inference. Consequently, DreamLCM is more effective with fewer training costs.To sum up, we observe that the principal cause of the oversmooth issue in SDS is the inadequate quality of the guidance. These two methods tackle the issue by utilizing extra resources, e.g., training multiple NeRFs and DDIM Inversions, which is time-consuming. Differently, DreamLCM can resolve the over-smooth issue by taking full advantage of LCM, while saving training costs. A portrait of The Wolverine, blonde hair, head, HDR, photorealistic, 8K.A portrait of Harry Potter, white hair, head, HDR, photorealistic, 8K.A portrait of Elsa, the princess from Frozen, icy blue eyes, fair complexion, head, HDR, photorealistic, 8K.A knight holding a lance and sitting on an armored horse.Viking axe, fantasy, weapon, blender, 8k, HDR.A portrait of Groot, head, HDR, photorealistic, 8K.A portrait of captain Jack Sparrow, buzz cut, head, HDR, photorealistic, 8K.A bear dressed in medieval armor.A blue motorcycle.A DSLR photo of a old-fashioned clock decorated with roses.A phoenix, reborn from flames, adorned with vibrant plumage of crimson and gold.A vine-crafted lantern, intricately woven with delicate tendrils, HDR, photorealistic, 8K.A corgi wearing a top hat. A wide angle DSLR photo of a colorful rooster.An mini garden. A forbidden castle high up in the mountain.An astronaut riding a horse Mount Fuji, snow-capped peak, HDR, photorealistic, 8K.A round marble fountain with multiple tiers, spraying arcs of water.A massive ocean-going yacht, with polished hull and towering decks.Sydney opera house be brightly lit at night, aerial view.  
experiments 6.1 implementation details: 
 We train our end-to-end network for 5000 iterations overall. We employ 3D Gaussian Splatting [18] as our 3D representation and 3D point cloud generation models Shap-E [15] and Point-E [30] for initialization. The rendering resolution is 512 × 512. As for the guidance calibration strategy, we use it in appearance optimization. We practically consider 𝑠 = 350 as the cut-off timestep. Unless stated otherwise, we train the first 1000 iters for geometry optimization using timesteps 𝑠 fulfilling 350 ≤ 𝑠 ≤ 980 and the remaining 4000 iters for appearance optimization using timesteps 𝑠 fulfilling 20 ≤ 𝑠 ≤ 350. Since we assume that LCM follows a smooth PF-ODE, the interval between 𝑠 and 𝑡 is less limited. Practically, we choose 𝑡 = 2𝑠. We use SDS with a normal CFG scale of 7.5. All experiments are performed and measured with an RTX 3090 (24G) GPU. We train about 50 min per sample.6.2 Text-to-3D Generation.In Fig. 4 , we show the generated results of DreamLCM. We generate all examples using the original LCM without LoRA and any finetuned checkpoints. We can see that DreamLCM can generate photo-realistic 3D objects with fine details. The 3D objects are creative and highly consistent with the text prompts. Especially, weA portrait of a unicorn, head, shimmering silver horn, gentle eyes, and a flowing mane of pink hues.A white seashell adorned with a brilliant blue sapphire.A sleek, aerodynamic race car adorned with vibrant colors.A silver platter piled high with fruits.An airplane made out of wood.A Spanish galleon. can see that DreamLCM can generate different and amazing avatar heads conditioned on text prompts, such as "A portrait of Harry Potter, white hair, head, HDR, photorealistic, 8K". Besides, the proposed DreamLCM method is good at generating objects conditioned on complex text prompts, like "the fuji mountain", "the massive yacht", and "the fountain". These examples demonstrate that DreamLCM well resolves the over-smooth issue in SDS. Besides, these examples show great potential in generating all kinds of complex objects with different LCM finetuned checkpoints.We compare our method with the current SoTA baselines which generate 3D Gaussian Objects [22, 43, 46] . As shown in Fig. 4 , our model generates more photo-realistic results than other works, exhibiting high quality and fine details. For example, "A portrait of a unicorn" is more photo-realistic, and the fur is more silky than the results from the other three approaches. As for the results conditioned on the text "A Spanish galleon", our model generates the most intact galleon, and the details of the hull are the finest 6) We use fixed noise to perturb the samples to reduce the randomness in noises to improve the details. We highlight some improved details in cyan. The prompts corresponding to the four examples are "a green dragon breathing fire", "a squirrel in samurai armor wielding a katana", "a delicious hamburger" and "A warrior with red cape riding a horse".among all the shown methods. Notably, compared to LucidDreamer, we generate higher quality objects with less training costs.Fig. 5 depicts the ablation experiments of different baseline methods. In Fig. 5 (b) and (c), we utilize timesteps between 20 and 500 to generate high-quality images. Other settings are the same as the final settings 6.1. Starting from the original SDS loss, guided by Stable Diffusion [35] , with a large CFG scale(100). We first incorporate LCM as the guidance model to demonstrate that LCM is a superior guidance model to DMs [35] . We can see that LCM makes a huge improvement in generation quality. We then add our Dual Timestep Strategy. We divide the strategy into two parts. We demonstrate the effectiveness of Decreasing Time Strategy and the Two-phase Strategy, as shown in Fig. 5 (c) and (d). We can see that the hamburger adding Decreasing Time strategy shows a quality improvement. Based on (c), the hamburger adding the two-phase strategy shows a geometric advancement of the bread at the bottom of the hamburger. Besides, due to the two-phase strategy, the worrier example is deformed to be less close to the horse, since this 3D model is initialized by the prompt "a wolf". Then, we add the guidance calibration strategy which smooths the appearance and improves the details, making the objects more photo-realistic. Finally, we add fixed noises to improve the consistency of guidance between different timesteps. As shown in Fig. 5 , the ability to improve details is demonstrated in cyan, such as the eye, legs, and cushion on the horse back in the worrier sample, the katana in the squirrel sample and the shadow in the dragon sample. 
conclusion: 
 In this paper, we propose DreamLCM method to improve the text-to-3D object task. We incorporate LCM as a guidance model to generate high-quality guidance to resolve the two factors that cause the oversmooth issue. Besides, we introduce two techniques, i.e., Guidance Calibration and Dual Timestep Strategy, to further improve the generation quality. Experiments show superior performance of our method. Our method achieves state-of-the-art results in both generation and training efficiency.  
: 
 (c) The proposed DreamLCM method incorporates LCM as the guidance model. We also propose a guidance calibration strategy that uses Euler Solver to refine the guidance ε𝑠 0 to ε𝑡 0 . Our method generates higher-quality guidance compared to (a) and (b).space. This latent guidance can be utilized to supervise the carving process of the target 3D objects. Thus, this alternative approach tackles the challenge of 3D object generation without large-scale 3D models for training. For example, the Score Distillation Sampling (SDS) objective is introduced in DreamFusion [32] to leverage the robust prior knowledge acquired by text-to-image diffusion models [35, 37] . The SDS backpropagates the gradients from the 2D diffusion model to 3D objects and bridges the gap between the diffusion models and the 3D representations, as shown in Fig. 1(a) . The acquired prior knowledge is utilized to optimize the 3D objects represented by Neural Radiance Fields (NeRF) [28] conditioned on a single text prompt.However, SDS is limited in generating fine details as it produces over-smooth results. This effect has been noted by previous works [17, 22, 45] . These works attempt to improve SDS and achieve good results in increasing the quality of 3D models. For instance, ProlificDreamer [45] optimizes multiple 3D models simultaneously. These models are mutually benefited and merged by finetuning a LoRA model [12] . The LoRA thus reserves the details of the 3D model. Nevertheless, extra resources are needed to regenerate the lost details. In this paper, we think that the problem of the oversmooth issue stems from two factors. Firstly, the guidance of SDS is derived by the DDPM [9] single-step inference, which leads to low-quality guidance and blurred details. Secondly, the rendered images of the target 3D object act as conditions and are fed into DMs after adding random noises. Besides, DMs need to sample timesteps randomly for the diffusion process. The randomness from both the added noises and timesteps directly results in the inconsistent guidance between different iterations. This inconsistency ultimately averages the details of 3D models and leads to the over-smooth issue.This paper endeavors to tackle the over-smooth issue by handling the factors above accordingly. We propose a novel approach to generate clear guidance for the low-quality guidance issue. Inspired by Latent Consistency Model (LCM) [26] , we propose DreamLCM by incorporating LCM as a guidance model to fully utilize the capability to generate high-quality guidance in a single-step inference. Notably, LCM generates high-quality images in a single-step inference, rather than gradually approaching to the origin along the probably flow ODE (PF-ODE) trajectory [42] by multi-step inference like DDIM [40] . Therefore, DreamLCM merely predicts guidance of a rendered image by directly denoising the noisy latent from an arbitrary timestep along the PF-ODE trajectory to keep fine details of the target 3D models. For the inconsistency issue, we observe that LCM generates consistent guidance regardless of the randomness. To solve the issue, a similar method has been proposed in LucidDreamer [22] , which uses the DDIM inversion technique [40] to improve the consistency of the guidance. However, different from LucidDreamer, the proposed DreamLCM method provides two merits: 1) DreamLCM only needs a single-step inference to compute the guidance while LucidDreamer forwards the U-Net [36] in DM multiple times; 2) DreamLCM keeps the original SDS loss. Since LCM can resolve the two issues causing the over-smooth issue, there is no need to change the loss forms. On the contrary, LucidDreamer utilizes a complex objective function to adapt the DDIM Inversion method. The difference is shown in Fig. 1 In addition, to further improve generation quality, we propose two novel strategies, i.e., Guidance Galibration and Dual Timestep Strategy. For Guidance Calibration, we propose a two-stage strategy that repeats the perturbing and denoising steps to calibrate the distribution of the guidance. In this way, the disturbing information can be gradually removed. In the first stage, we perturb a rendered image and predict the corresponding guidance. This guidance is consistent with the rendered image, i.e., the 3D object, as the added noise and timestep are small. In the second stage, we run a discretization step of a numerical ODE solver, where we use the Euler Solver to obtain a latent with relatively large noises. The large noises and timestep can provide a more reasonable optimization direction. Consequently, the calibrated guidance is ensured to be consistent with both the rendered image and the highest data density conditioned on the text prompt, effectively improving the guidance's quality. We calculate SDS loss and back-propagate the gradient using the calibrated guidance to optimize the 3D models. For Dual Timestep Strategy, we utilize the timestep sampling strategy to enable dreamLCM to optimize the geometry and appearance of 3D objects in separate phases. In particular, in the initial phase, we apply large timesteps to guide the 3D model in producing large deformations. In this case, DreamLCM tends to optimize geometry, where the position of Gaussian Splatting is greatly updated. In the refinement phase, we use small timesteps to optimize the appearance because small timesteps help DreamLCM generate guidance with fine details. Besides, we sample monotonically decreasing timesteps to increase the consistency of the guidance. Overall, our proposed dual timestep strategy is the combination of the timestep strategy in HiFA [49] and ProlificDreamer [45] .We apply the Gaussian Splatting [18] as the 3D representation to form the 3D target objects. The proposed DreamLCM achieves the state-of-the-art results. As shown in Fig. 3  
 
 Appendix:  


Decision: This paper introduces LCM into optimize-based 3D generation pipeline, which is novel and effective. All the reviewers agree to accept the manuscript. I recommend a decision of acceptance. In the final version, the authors should include the discussions in the rebuttal to address the concerns of reviewers.

Reviews: review: paper_summary: paper_summary: This paper is the first to apply the CM method to SDS, introducing a novel approach. The authors claim that this method resolves the over-smoothing problem.
strength_weakness: strength_weakness: 1) This paper leverages the LCM model to incorporate more information, resulting in improved generation.

2) The paper is well-written and clearly articulated.
limitations: limitations: 1) The application of LCM is quite strange. Firstly, the SDS loss originates from the training loss of diffusion models, which involves taking the derivative with respect to the image to obtain the SDS loss. Directly switching the model from a diffusion model to an LCM model is strange because the LCM loss does not follow this approach. Secondly, for LCM, it maintains the trace, but  $x_0$  obtaining  $x_s$  does not follow the ODE, which means the predicted target of LCM from $x_s $ is not $x_0 $, resulting in inconsistency. Lastly, to perform Euler Solver, diffusion models are still needed, leading to double memory requirements.

2) Algorithm 1, $ L_{SDS} $, lacks a term.

3) The results do not appear to be better than those of LucidDreamer.

If you can address these concerns, I will raise my score.
paper_summary: paper_summary: This paper introduces LCM into the optimize-based 3D generation pipline to address the over-smooth problem. Firstly, in order to improve the quality of guidance during refinement, this paper proposes guidance calibration module, which aligns the noisy latents and reference image latents on the same PF-ODE trajectory. Secondly, this paper proposes dual time strategy, which optimizes geometry and appearance with large and small timesteps respectively when decreasing the timestep at the same time.
strength_weakness: strength_weakness: 1.This paper introduce LCM into optimize-based 3D generation pipeline for the first time.

2.The proposed guidance calibration are well-motivated and effective.

3.The experiment results on 3D generation are convincing.
limitations: limitations: 1.Further analysis. In guidance calibration module, authors demostrate that the shifting of images is caused by the insufficient training and assume that the denoising process of LCM follows a smooth PF-ODE trajectory with a small slope. Could authors provide further theoretical analysis on the relationship between the insufficient training and the smooth PF-ODE trajectory?

2.Effectiveness of dual time strategy. As shown in Fig.5, the results w/o dual time strategy(column2) actually have more geometry details than the results with the strategy(column3&4), such as the tail of the dragon and the front leg of the horse. Perhaps the authors need to further demonstrate the effectiveness of the strategy with experiment results.

3.Necessity of two-phase strategy. As far as I'm concerned, NeRF is more stable than gaussian splatting during the optimizing process. Is the two-phase strategy still necessary when training on NeRF, as the decreasing timestep strategy is widely used in previous work?
paper_summary: paper_summary: This paper proposes a new text-to-3D framework, DreamLCM, which combines Latent Consistency Model (LCM) with its powerful image generation ability to generate consistent and high-quality 3D results, avoiding the problem of over-smoothing which leads to quality imbalance in SDS loss. In addition, two strategies are proposed to further improve the generation quality: one is a guided calibration strategy, using the Euler solver to calibrate the guided distribution, and the other is a dual time-step strategy, optimizing the 3D model from geometry to appearance
strength_weakness: strength_weakness: 1. **An effective combination with LCM model instead of conventional diffusion**: It is effective to use LCM instead of the diffusion model to reduce the excessive smoothing problem caused by SDS loss, and improve the quality and fineness of the generated model.

2. **Clear writing with explainable figures**: The writing and figures are clear to understand the motivation and implementation of the work. Also, it provides an insightful discussion with ProlificDreamer and LucidDreamer, showing clearer intuition.

3. **High-fidelity Results**: The experiments show high-quality results in text-to-3D generation without Janus Problem.
limitations: limitations: 1. **geometric inconsistencies**: Note that in the model generated by the ablation experiment, the horse appears to have only three legs, which miraculously disappear after the addition of certain components (such as the time-step reduction strategy), but reappear after the addition of the two-step strategy. This seems to be the case with the horse in Figure 3.

2. **Lack of baseline for comparison**: Now there are many text-to-3D models, many of which are dedicated to solving the problem of excessive SDS smoothness, such as ProlificDreamer mentioned in the article, you can compare with such models to generate results, operation time, cost and so on. Rather than a model like DreamGaussian that addresses the build speed issue. In addition, it is not necessary to limit the model generated by the Gaussian splatting method.

3. **Some ablation experiments are missing**: The ablation experiment lacks a model with guidance calibration and no two-time-step strategy. In addition, it may be possible to add background information and formulas on SDS losses.

4. **More discussion**: Choosing a more advanced model (e.g., MVDream), which is a multi-view aware diffusion at the start of LCM may be a more reasonable approach. But it is a minor problem.
paper_summary: paper_summary: This paper presents a training strategy for high-quality text-to-3D generation. The introduction of the Latent Consistency Model (LCM) can improve training stability. Guidance calibration has also been proven to be useful. Keeping the noise added in the rendered images constant is also an interesting and useful trick. The experiments show that the whole training pipeline produces impressive results.
strength_weakness: strength_weakness: 1. This paper is well-written, which makes readers understand the claims and method easily.
2. DreamLCM produces impressive generation quality.
3. Many tricks are introduced into the training pipeline to ensure the generation quality.
4. The introduction of LCM is useful, and the guidance calibration is also an effective patch for the LCM in the text-to-3D pipeline.
5. The ablation studies clearly show the effectiveness of different parts of the pipeline.
limitations: limitations: 1. This paper ignores an important reference, DreamTime [1]. DreamTime has proposed the monotonically non-increasing functions to control the timestep t. This is significantly similar to one of the tricks used in the DreamLCM. However, there is a lack of citations for this paper.
2. In Line 143, the authors mentioned that one of the merits is to keep the original SDS. And they mainly illustrate this point from the complexity of the loss function. The logic looks strange.


References

[1] Huang Y, Wang J, Shi Y, et al. DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation[C]//The Twelfth International Conference on Learning Representations. 2023.