Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching

## Abstract

Abstract In this work, we propose a novel Trajectory Score Matching (TSM) method that aims to solve the pseudo ground truth inconsistency problem caused by the accumulated error in Interval Score Matching (ISM) when using the Denoising Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the inversion process of DDIM to calculate on a single path, our TSM method leverages the inversion process of DDIM to generate two paths from the same starting point for calculation. Since both paths start from the same starting point, TSM can reduce the accumulated error compared to ISM, thus alleviating the problem of pseudo ground truth inconsistency. TSM enhances the stability and consistency of the model’s generated paths during the distillation process. We demonstrate this experimentally and further show that ISM is a special case of TSM. Furthermore, to optimize the current multi-stage optimization process from high-resolution text to 3D generation, we adopt Stable Diffusion XL for guidance. In response to the issues of abnormal replication and splitting caused by unstable gradients during the 3D Gaussian splatting process when using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping method. Extensive experiments show that our model significantly surpasses the state-of-the-art models in terms of visual quality and performance. Code: https://github.com/xingy038/Dreamer-XL .

## 1 Introduction

In recent years, Virtual Reality (VR) and Augmented Reality (AR) have increasingly become a part of our daily lives, and the demand for high-quality 3D content has increased significantly. 3D technology has become extremely important, allowing us to visualize, understand and interact with complex objects and environments. It also plays a key role in various fields such as architecture, animation, gaming and virtual reality. In addition, 3D technology shows broad application prospects in retail [], online meetings [], education [] and other fields [, ]. Despite its wide application, the complexity of creating 3D content poses considerable challenges: generating high-quality 3D models requires computional time, effort, and expertise. Given these challenges, methods for generating 3D from text have become particularly important in recent years [, , , ]. These methods create accurate 3D models directly from natural language descriptions, thereby reducing manual input in traditional 3D modeling processes. Once the text-to-3D method can efficiently generate large amounts of data, it will not only shorten the production time of 3D content, but also reduce costs and improve production efficiency.

Figure: Figure 1: Example of text-to-3D content generated from scratch by our Dreamer XL. Our Dreamer XL is based on 3D Gaussian splatting using stable diffusion XL. Please zoom in for details.
Refer to caption: x1.png

Typically, text-to-3D generation methods utilize pre-trained text-to-image diffusion models [] as an image prior to training neural parametric 3D models such as Neural Radiance Fields (NeRF) [] and 3D Gaussian splitting []. These approaches enable the rendering of consistent images that are aligned with the text. This process essentially relies on Score Distillation Sampling (SDS) []. Through SDS, the model can distill the capabilities of the pre-trained 2D diffusion model to obtain rendered images, and optimize the parameters of the 3D model through backpropagation, so that the 3D model can be effectively trained even without actual image data. However, since random noise generates inconsistent pseudo-baselines, the results obtained by SDS optimization of 3D models tend to be averaged, leading to problems such as over-smoothing.

Although some recent work [, ] devote themselves to solving the over-smoothing problems, they inevitably lead to the generation of low-resolution and average results due to the inherent limitations of stable diffusion models and their sampling methods. For example, [], inspired by the DDIM inversion process, proposed interval score matching, which can generate relatively consistent results. However, due to the inherent cumulative errors in the DDIM inversion process, it may lead to the averaging of results in certain regions. Furthermore, most existing methods do not yet support the new high-resolution Stable Diffusion XL (SDXL) [, ]. To achieve an output of 1024x1024 high resolution, multi-stage optimization is necessary. The main reason is the inherent instability of the Variational Autoencoder (VAE) in the SDXL [] architecture, which is particularly evident during the optimization process of the 3D Gaussian Splatting. In this process, the gradients directly affect the duplication and deletion of point clouds in 3D space. The anomalous gradients introduced by SDXL severely hinder the optimization process of 3D Gaussian Splatting, leading to generated 3D models that lose complex texture details, have blurred appearances, and exhibit abnormal colors. In severe cases, this can cause the 3D models to fail to converge.

In this work, we aim to overcome the above limitations. The reverse process of DDIM was adopted by [], effectively reducing the higher reconstruction errors generated by the diffusion model’s one-step reconstruction and generating relatively consistent pseudo-ground truth. However, the inherent accumulation errors in the reverse process of DDIM still result in semantic changes in these pseudo-ground truths, leading to partially averaged reconstruction results in certain regions, resulting in erroneous and unrealistic outcomes in these regions. To address this issue, we propose a novel method called Trajectory Score Matching (TSM). We through simple improvements to Interval Score Matching (ISM) [], effectively alleviate the average effect of inconsistent pseudo-ground truths caused by inherent accumulation errors. We demonstrate that our TSM has smaller accumulation errors compared to ISM, and ISM can be considered as a special case of TSM. Through experiments, we prove that the effects produced by our TSM are superior to ISM, yielding highly realistic and detailed results. In order to achieve high-resolution output, previous methods need to undergo multi-stage training. Our model directly uses SDXL that supports high resolution as guidance without going through multi-stage training. This not only reduces training costs, but also simplifies the training process. However, we still need to solve the inherent gradient instability problem of SDXL. Therefore, we propose a pixel-by-pixel gradient clipping method, which effectively alleviates the inherent gradient instability of SDXL. In summary, the contributions of our work are as follows:

- •
We investigate and analyze the inherent accumulated errors produced by DDIM inversion process in interval score matching (ISM), resulting in the presence of inconsistent pseudo-ground truth.
- •
To address the aforementioned limitation, we introduce a novel Trajectory Score Matching (TSM) method. Unlike the single path of ISM, TSM improves ISM to a dual path, effectively alleviating the inconsistent pseudo-ground truth issue generated from the inherent accumulated error of DDIM.
- •
To simplify the training process and generate high-resolution, high-quality text-to-3D results. We are the first to leverage SDXL for guidance based on 3D Gaussian splatting. In addition, we also introduce a novel gradient clipping method, which effectively solves the problem of SDXL in gradient stability. Extensive experiments demonstrate that our method significantly outperforms the current state-of-the-art methods.

## 2 Related Work

#### 2D diffusion.

Score-based generative models and diffusion models [, , , , , ] have shown excellent performance in image synthesis [], especially by introducing latent diffusion models (LDM) [] into stable diffusion to generate high-resolution images in latent space. Podel et al. [] further extended this model to a larger latent space in SDXL, VAE and U-net, achieving higher resolution (1024×1024). Zhang et al. [] enhanced the functionality of these models by generating controllable images for different input types. At the same time, the diffusion model also showed impressive performance in converting text into image synthesis, which opens up the possibility of using this technology to directly generate 3D images from text [, , , , , ].

#### Text-to-3D Generation.

Early attempts from text-to-3D are mainly guided by the use of multi-modal information from CLIP [] to achieve information conversion from text-to-3D, with DreamField [] being a pioneer in this direction. However, the multi-modal information of CLIP can only provide rough alignment, and the results of using it for 3D distillation are often unsatisfactory. Zero-1-to-3 [] first introduces camera parameters to fine-tune the 2D pre-trained stable diffusion model, enabling it to generate multi-view images, and then use the multi-view images for 3D reconstruction. This improvement not only facilitates more accurate 3D reconstructions, but also inspires a wealth of derivative research [, , , , ]. In addition, another research direction explores the possibility of using pre-trained 2D diffusion models to directly optimize 3D representations. These methods often combine differentiable 3D representation techniques, such as NeRF [], NeuS [], and 3D Gaussian Splatting [], and optimize model parameters through backpropagation techniques. Dreamfusion [] first introduces SDS to optimize 3D representations directly from pre-trained 2D text-to-image diffusion models. Similarly, Score Jacobian Chaining [] proposes an alternative method that achieves parameterization effects similar to SDS. ProlificDreamer [] conducted an in-depth analysis of the objective function of SDS and proposed a particle-based variational framework called Variational Score Distillation (VSD), which significantly improves the quality of generated content. The latest research combines SDS with Gaussian Splatting to accelerate the optimization process. Consistent3D [] analyzes SDS from the latest perspective of ordinary differential equations (ODE) and proposes a method called Consistency Distillation Sampling (CSD) to solve the challenges of SDS in over-smoothing and inconsistency issues. Similarly, LucidDreamer [] analyzed the loss function of SDS and proposed interval score matching (ISM), which is very similar to the idea of CSD. However, ISM utilizes the reversible diffusion trajectory of DDIM [] when calculating the two interval steps. DDIM will inevitably produce inherent accumulated errors in this process, resulting in inconsistent reconstruction results. In this work, we empirically follow the mature mainstream architecture method of 3D Gaussian Splatting [, , ] as the baseline of our approach. On this basis, inspired by the recent Consistency trajectory model [], we propose to optimize the 3D model from two trajectories, and use the less noisy trajectory to guide another noisier trajectory to alleviate the inconsistency problem.

## 3 Methods

This section presents the preliminaries on the inverse DDIM process, SDS and ISM (see Section 3.1). We then propose the Trajectory Score Matching (TSM) method (see Section 3.2), which generates dual paths from the same starting point using the reverse process of DDIM. This enhances the stability and consistency of the model along the entire generative path during the distillation process. We further indicate that ISM is a special case of TSM (see Section 3.3). Additionally, we investigate the challenges of optimizing 3D models using the SDXL architecture (see Section 3.4).

### 3.1 Preliminaries

#### Review of DDIM inversion

We first consider the most common sampling scheme is that of DDIM [] where intermediate steps are calculated as:

$$ $x_{t-1}=\sqrt{\alpha_{t-1}}\left(\frac{x_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\phi }(x_{t},t,\emptyset)}{\sqrt{\alpha_{t}}}\right)+\sqrt{1-\alpha_{t-1}}\epsilon_ {\phi}(x_{t},t,\emptyset),$ (1) $$

where $x_{t}$ and $x_{t-1}$ represent the noisy latent, $\{\alpha_{t}\}_{t=0}^{T}$ (where $a_{0}=1,a_{T}=0$) indicates a set of time steps indexing a strictly monotonically increasing noise schedule. The $\epsilon_{\phi}(x_{t},t,\emptyset)$ denotes the predicted denoising direction (by stable diffusion model) with the given condition $y$ (Here, the condition is null, i.e., unconditioned $\emptyset$).

As noted in DDIM [], the above denoising process can approximate the inverse transition from $x_{t}$ to $x_{t-1}$, which can be expressed as:

$$ $\displaystyle x_{t}$ $\displaystyle=\sqrt{\alpha_{t}}\left(\frac{x_{t-1}-\sqrt{1-\alpha_{t}}\epsilon _{\phi}(x_{t},t,\emptyset)}{\sqrt{\alpha_{t-1}}}\right)+\sqrt{1-\alpha_{t-1}} \epsilon_{\phi}(x_{t},t,\emptyset)$ (2) $\displaystyle\approx\sqrt{\alpha_{t}}\left(\frac{x_{t-1}-\sqrt{1-\alpha_{t}} \epsilon_{\phi}(x_{t-1},{t-1},\emptyset)}{\sqrt{\alpha_{t-1}}}\right)+\sqrt{1- \alpha_{t-1}}\epsilon_{\phi}(x_{t-1},t-1,\emptyset),$ $$

where the approximation is a linearization assumption that $\epsilon_{\phi}(x_{t},t,\emptyset)\approx\epsilon_{\phi}(x_{t-1},t-1,\emptyset)$. This approximation inevitably introduces errors, resulting in inconsistencies between the diffusion states in the forward and backward processes.

Figure: Figure 2: ISM example []. We notice that using the same initial value $x_{0}$ but under different noise $\{\epsilon_{1},\epsilon_{2},\epsilon_{3},\epsilon_{4}\}$, the generated results still show certain inconsistencies. This is due to the error accumulation inherent in the DDIM inversion process. These inconsistencies can lead to errors or inconsistencies in some areas during the optimization of the 3D model.
Refer to caption: extracted/5604290/motivation.png

#### Text-to-3D generation by interval score matching (ISM)

The concepts of the ISM is first introduced by LucidDreamer [] to address the issues of over-smoothness and inconsistency inherent in the original SDS method. The 3D model leverages a differentiable function $x=g(\theta,c)$ to render images, where $\theta$ represents the trainable 3D parameters and $c$ is camera parameter. The gradient of the ISM loss for $\theta$ is expressed as follows:

$$ $\nabla_{\theta}\mathcal{L}_{\text{ISM}}(\theta):=\mathbb{E}_{t,c}\left[\omega( t)(\epsilon_{\phi}(x_{t},t,y)-\epsilon_{\phi}(x_{s},s,\emptyset))\frac{ \partial x}{\partial\theta}\right],$ (3) $$

where $0<s<t$, the noisy latent $x_{t}$ and $x_{s}$ are calculated by DDIM inversion process, the $\epsilon_{\phi}(x_{t},t,y)$ is the predicted denoising direction given the conditioned $y$, and $\omega(t)$ is a time-dependent weighting function. In DDIM inversion process, the generation of accumulated error is inevitable. Especially for conditional models, these errors are further magnified. Therefore, inconsistent pseudo-ground truths will be generated when optimizing the 3D model, thus affecting the optimization quality of the final result.

### 3.2 Trajectory Score Matching

To address the inherent accumulated error in the DDIM inversion process, which leads to the production of inconsistent pseudo-ground truths and consequently suboptimal 3D models. Inspired by recent work [], we propose a new approach, Trajectory Score Matching (TSM), which utilizes dual paths originating from the same starting point to minimize error accumulation during iterations. Specifically, similar to ISM [], our TSM also utilizes the DDIM inversion to predict an invertible noisy latent trajectory. For a given timestep $s$ (where $0<s<t$), the corresponding noise latent $x_{s}$ can be obtained using Equation 2. Considering $x_{s}$ as the starting latent, it is possible to approximate two noise latents, $x_{\mu}$ and $x_{t}$, on the latent trajectory, where $0<s<\mu<t\leq T$. This can be expressed as follows:

$$ $x_{\mu}=\sqrt{\alpha_{\mu}}\left(\frac{x_{s}-\sqrt{1-\alpha_{\mu}}\epsilon_{ \phi}(x_{s},s,\emptyset)}{\sqrt{\alpha_{s}}}\right)+\sqrt{1-\alpha_{s}} \epsilon_{\phi}(x_{s},s,\emptyset),$ (4) $$

$$ $x_{t}=\sqrt{\alpha_{t}}\left(\frac{x_{s}-\sqrt{1-\alpha_{t}}\epsilon_{\phi}(x_ {s},s,\emptyset)}{\sqrt{\alpha_{s}}}\right)+\sqrt{1-\alpha_{s}}\epsilon_{\phi} (x_{s},s,\emptyset).$ (5) $$

Then, we can integrate DDIM inversion and DDIM denoising with the same step size. We define the naive objective of 3D distillation as follows:

$$ $L_{\text{TSM}}(\theta):=\mathbb{E}_{t,c}\left[\omega(t)||\epsilon_{\phi}(x_{t} ,t,y)-\epsilon_{\phi}(x_{\mu},\mu,\emptyset)||^{2}\right],$ (6) $$

where $x_{t}$ and $x_{\mu}$ is generated through DDIM inversion from $x_{0}$. Followiing [], the gradient of TSM loss over $\theta$ is:

$$ $\nabla_{\theta}\mathcal{L}_{\text{TSM}}(\theta):=\mathbb{E}_{t,c}\left[\omega( t)(\epsilon_{\phi}(x_{t},t,y)-\epsilon_{\phi}(x_{\mu},\mu,\emptyset))\frac{ \partial x}{\partial\theta}\right].$ (7) $$

The optimization goal of TSM is to maintain the consistency of $x_{0}$ updates as much as possible to reduce the error introduced by DDIM inversion. Since TSM uses the same noise latent during the inversion process, its cumulative error is relatively small. The algorithm flow of TSM is shown in the Algorithm 1. Among them, the blue part marks the differences from ISM.

Figure: Algorithm 1 Trajectory Score Matching

### 3.3 Comparison with ISM

We now analyze the differences between TSM and ISM theoretically.

#### TSM has smaller error

For ISM, the optimized goal is the minimum predicted noise from noise latent $x_{s}$ and $x_{t}$. The noise latent can be obtained from Equation 2 and then using the pre-trained stable diffusion model to predict noise. During the inversion process of DDIM, $x_{s}$ is approximated by the prediction noise at $s-1$, and $x_{t}$ is approximated by the prediction noise at $s$. For simplicity, we can regard the optimization goal of ISM as minimizing noise latent, thus the accumulated error generated by ISM during the optimization process can be expressed as:

$$ $\displaystyle x_{t}-x_{s}$ $\displaystyle=\sqrt{\alpha_{t}}\left(\frac{x_{s}-\sqrt{1-\alpha_{t}}\epsilon_{ \phi}(x_{s},s,\emptyset)}{\sqrt{\alpha_{s}}}\right)+\sqrt{1-\alpha_{s}} \epsilon_{\phi}(x_{s},s,\emptyset)$ (8) $\displaystyle-\sqrt{\alpha_{s}}\left(\frac{x_{s-1}-\sqrt{1-\alpha_{s}}\epsilon _{\phi}(x_{s-1},{s-1},\emptyset)}{\sqrt{\alpha_{s-1}}}\right)+\sqrt{1-\alpha_{ s-1}}\epsilon_{\phi}(x_{s-1},{s-1},\emptyset)$ $$

For our TSM, we can also give similar expresses as:

$$ $\displaystyle x_{t}-x_{\mu}$ $\displaystyle=\sqrt{\alpha_{t}}\left(\frac{x_{s}-\sqrt{1-\alpha_{t}}\epsilon_{ \phi}(x_{s},s,\emptyset)}{\sqrt{\alpha_{s}}}\right)+\sqrt{1-\alpha_{s}} \epsilon_{\phi}(x_{s},s,\emptyset)$ (9) $\displaystyle-\sqrt{\alpha_{\mu}}\left(\frac{x_{s}-\sqrt{1-\alpha_{\mu}} \epsilon_{\phi}(x_{s},s,\emptyset)}{\sqrt{\alpha_{s}}}\right)+\sqrt{1-\alpha_{ s}}\epsilon_{\phi}(x_{s},s,\emptyset)$ $$

Compared with ISM, which is optimized for a single path, TSM is optimized for a dual path starting from the same noise latent. We show that the accumulated error is smaller in TSM, as shown below.

#### Theorem 1

Consider three timesteps $0<s<\mu<t$, where $\mu$ is defined as $\mu=\gamma(t-s)+s$, with $\gamma\in[0,1]$. Then, we have:

$$ $x_{t}-x_{s}>x_{t}-x_{\mu}\Rightarrow x_{\mu}>x_{s},$ (10) $$

Proof: Assume $x_{\mu}\leq x_{s}$. Utilizing Equation 8 and Equation 9, we aim to demonstrate that:

$$ $\displaystyle\sqrt{\alpha_{\mu}}\left(\frac{x_{s}-\sqrt{1-\alpha_{\mu}} \epsilon_{\phi}(x_{s},s,\emptyset)}{\sqrt{\alpha_{s}}}\right)+\sqrt{1-\alpha_{ s}}\epsilon_{\phi}(x_{s},s,\emptyset)$ (11) $\displaystyle\leq\sqrt{\alpha_{s}}\left(\frac{x_{s-1}-\sqrt{1-\alpha_{s}} \epsilon_{\phi}(x_{s-1},{s-1},\emptyset)}{\sqrt{\alpha_{s-1}}}\right)+\sqrt{1- \alpha_{s-1}}\epsilon_{\phi}(x_{s-1},{s-1},\emptyset).$ $$

Where $\alpha$ represents a set of timesteps with a strictly monotonically increasing noise schedule, hence $\alpha_{s-1}<\alpha_{s}<\alpha_{\mu}$. Considering the $\alpha_{s}<\alpha_{\mu}$, the term involving $\sqrt{\alpha_{\mu}}$ should yield a smaller value compared to the term involving $\sqrt{\alpha_{s}}$, which presents a contradiction to our initial assumptions. Considering the $\epsilon_{\phi}$ terms, due to the inversion process of DDIM, $\epsilon_{\phi}(x_{s},s,\emptyset)\approx\epsilon_{\phi}(x_{s-1},s-1,\emptyset)$ and $\sqrt{1-\alpha_{s-1}}>\sqrt{1-\alpha_{s}}$. This also contradicts the assumption. Therefore, our initial assumption $x_{\mu}\leq x_{s}$ must be false. Consequently, we conclude that $x_{\mu}>x_{s}$, thus proving the theorem.

#### ISM is a special case of TSM

Theoretically, the optimization objectives of ISM and TSM are the same, however, TSM considers optimization steps that are closer together compared to ISM. Specifically, we can regard ISM as a special case of TSM. Between time steps $s$ and $t$, we choose any time step $\mu=\gamma(t-s)+s$, where $\mu=s$ if and only if $\gamma=0$. Therefore, our hypothesis is confirmed, and ISM is indeed a special case of TSM when $\mu=s$.

### 3.4 The abnormal gradient from advanced pipeline

Previous methods have shown that increasing rendering resolution and training batch size can significantly improve visual quality. Although increasing the resolution of rendering can significantly improve the visual quality, most text-to-3D generation methods mainly use guidance based on Stable Diffusion 2.1 and only support $512\times 512$ resolution. Due to the impact of low resolution, local details are still blurred. Consequently, we experimented with using Stable Diffusion XL as guidance, which supports $1024\times 1024$ resolutions. The more advanced model Stable Diffusion XL has a different architecture from the previous one, and the VAE of this model is unstable. Although it has a certain impact on NeRF-based methods, it is not serious. However, this instability poses significant challenges for methods that employ 3D Gaussian splatting. In 3D Gaussian splatting, the reliability of operations like copying and deleting point clouds is heavily dependent on gradient stability. If the average positional gradient $g$ of the Gaussian view space exceeds a preset threshold, regions with under- or over-reconstruction of color $c$ and depth $d$ are intensively corrected. SDXL gradients are usually large and unstable, and high average gradient values in this case may cause normal areas to still be densified. An intuitive method is leveraging the gradient clip technical to handle this issue, previous related work [] has explored for NeRF-based method, which is not very suitable for 3D Gaussian splitting. Thus, we propose an improved gradient clip method for 3D Gaussian splitting. Specifically, we still use the [] method for gradient clipping of color $c$. For depth $d$, we calculate its scaling factor independently for each depth element and perform pixel-by-pixel pruning. The pruning gradient of depth $\boldsymbol{\hat{g}}_{d}$ can be expressed as:

$$ $\boldsymbol{\hat{g}}_{d}=\boldsymbol{g}_{d}\cdot\min\left(\min\left(\frac{s}{| \boldsymbol{g}_{d}|},c\right)\right),$ (12) $$

where $\boldsymbol{g}_{d}$ is the gradient of depth, $s$ is scale of Gaussian and $c$ is the threshold. In this way, we can ensure that the updated direction of the depth gradient remains unchanged and has no effect on the gradient of the normal region.

Figure: Figure 3: Comparison with state-of-the-art baseline methods in text-to-3D generation. Experimental results show that our method can generate 3D content that is more consistent with input text prompts and has more detailed details. All results of this work are generated on a single A100 GPU. Please zoom in to see more details.
Refer to caption: x2.png

## 4 Experiments

### 4.1 Qualitative Results

#### Text-to-3D Generation

We show the generated results of Dreamer XL in Figure 1. The results show that Dreamer XL is capable of generating high-quality 3D content accurately based on the input text, and it performs exceptionally well in producing realistic and complex appearances, effectively avoiding common issues such as excessive smoothing or oversaturation. For example, it can finely reproduce the texture details of objects like teacups. Moreover, our framework can generate objects that are close to reality and create imaginary ones. This flexibility offers possibilities for various application scenarios.

#### Comparison with State-of-the-Art Methods

We compare our approach with four state-of-the-art text-to-3D baselines: DreamFusion [] proposes Score Distillation Sampling (SDS) leveraging a pre-trained 2D text-to-image diffusion model for text-to-3D synthesis; Fantasia3D [] disentangle geometric and appearance attributes to simulate real-world physical environments; ProlificDreamer [] introduces Variational Score Distillation (VSD), a particle-based variational framework to address issues of oversaturation, oversmoothing, and low diversity; LucidDreamer [] introduces Interval Score Matching (ISM), utilizing deterministic diffusion trajectories and interval-based score matching to alleviate oversmoothing problems, and employs a 3D Gaussian splatting for 3D representation. The comparison results are shown in Figure 3. The results generated by our method are significantly clearer than other baseline results. For example, the crown shows a more precise geometric structure and a more realistic color, and the Schnauzer’s hair texture and overall body shape show obvious advantages. We can observe that our method significantly outperforms existing methods in both visual quality and consistency.

Figure: Figure 4: Comparison with the generation results of different stable diffusion models. Compared with ISM, our TSM performs better in the clarity and consistency of local details. Please zoom in to see the circled region for more details.
Refer to caption: x3.png

#### Comparison with ISM in detail

As shown in Figure 4, we show the generation results of ISM and TSM using the same prompt on different stable diffusion models. In Iron Man, it can be seen that the ISM has significant inconsistencies on the left and right sides of the neck, while our TSM maintains consistency in this region. In Joker, the ISM has shallower wrinkles on the head compared to our TSM, which is due to the averaging effect caused by error accumulation. Furthermore, ISM also shows significant inconsistency in the neck region.

**Table 1: Quantitative evaluation. We compare with recent text-to-3D conversion methods. CLIP-score is used to measure the alignment between text and 3D content, while A-LPIPS is used to evaluate the degree of artifacts caused by inconsistencies in 3D content.**
| Methods | CLIP-Score $\uparrow$ | A-LPIPS $\downarrow$ |  |  |
| --- | --- | --- | --- | --- |
| CLIP-L/14 | OpenCLIP-L/14 | VGG | Alex |  |
| DreamFusion | 0.232 | 0.165 | 0.081 | 0.080 |
| Fantasia3D | 0.233 | 0.207 | 0.077 | 0.082 |
| ProlificDreamer | 0.255 | 0.221 | 0.178 | 0.103 |
| LucidDreamer | 0.278 | 0.234 | 0.065 | 0.059 |
| Ours | 0.297 | 0.243 | 0.052 | 0.041 |

Figure: Figure 5: Ablation on offset rate. $\gamma=0.3$ achieves optimal visual quality and ensures high consistency between the generated results and the original text.
Refer to caption: x4.png

### 4.2 Quantitative Results

Currently, there are no standardized evaluation metrics specifically dedicated to text-to-3D. This is primarily due to the subjective nature of the task and the presence of multiple dimensions that are difficult to quantify. To maintain consistency with existing text-to-3D evaluation methods, we adopt CLIP-based metrics for quantitative analysis. Specifically, we employ variants of the CLIP model, including OpenCLIP ViT-L/14 and CLIP ViT-L/14, to calculate the average CLIP score between the text and its corresponding 3D render. Furthermore, considering the importance of view consistency, we follow previous work calculating A-LPIPS to determine view consistency, quantifying visual artifacts caused by view inconsistency through calculating the average LPIPS score between adjacent 3D scene images. We adopt A-LPIPS as an alternative metric to quantify view consistency, and present it alongside the CLIP scores in our report.

Consistent with the qualitative results, we compare our method with four state-of-the-art text-to-3D methods. Compared with the current best-performing LucidDreamer, our method improves CLIP-Score (CLIP-L/14) and CLIP-Score (OpenCLIP-L/14) by 6.83% and 3.85% respectively. At the same time, our method reduces the performance by 20.00% and 30.51% respectively on the A-LPIPS (VGG) and A-LPIPS (Alex) evaluation metrics, showing significant advantages in image authenticity and visual consistency. This improvement is mainly due to our adoption of the more advanced SDXL as a guidance model, which has lower accumulated error. Overall, these results highlight the superior performance of our approach in terms of image quality and text consistency.

Figure: Figure 6: Ablation on pixel-by-pixel gradient clipping.
Refer to caption: x5.png

### 4.3 Ablation Study

#### Ablation on offset rate γ 𝛾 \gamma italic_γ

We investigate the impact of the offset rate $\gamma$ on the generated results (Figure 5), and the best results are achieved when $\gamma$ is set to 0.3. If $\gamma$ is set too low, it will result in a loss of color details; if it is set too high, it may destroy the consistency between the generated results and the text. That is, when $\mu$ is too close to $s$, the average effect is too heavy, and its effect is similar to that of ISM. When $\mu$ is too close to $t$, although the cumulative error is reduced, the updated gradient will become very small, easily causing the model to fall into a local optimal state. However, for simple scenes, the results are best when $\mu$ is close to $t$, and the analysis in Section A.3.

#### Ablation on pixel-by-pixel gradient clipping

As shown in Figure 6, when pixel-by-pixel gradient clipping is not applied, the instability of the gradients causes abnormal splitting and duplication in normal areas, filling the depth map with noise and making it rough and uneven, thus making the entire facial appearance abnormal. However, after applying pixel-by-pixel gradient clipping, it is clearly observed that the depth map becomes smoother, the texture returns to normal, and the normal facial features are displayed. This comparison demonstrates the effectiveness of our method.

## 5 Conclusion

In this work, we investigate the inconsistency problem produced by ISM during the generation of 3D results. To alleviate this problem, we introduce TSM, which leverages dual paths to reduce error accumulation and thereby improve inconsistency. In addition, to simplify the generation of a high-resolution training process, we adopt SDXL as guidance and propose a pixel-by-pixel gradient clipping method to alleviate the abnormal splitting of normal regions in 3D Gaussian splatting caused by SDXL gradient instability. Our experimental results demonstrate that our method can effectively generate high-resolution, high-quality 3D results.

## Appendix A Supplemental material

### A.1 Implementation details

During the optimization process, we train the model for 2500 iterations. To optimize the 3D Gaussian model, we set the learning rates for opacity, scaling, and rotation to 0.05, 0.005, and 0.001 respectively. Furthermore, the learning rate of the camera encoder is set to 0.001. During training, RGB images and corresponding depth maps from 3D Gaussians are used for rendering. Gaussian densification and pruning processes are performed between 100 and 1500 iterations. We select the publicly available stable diffusion of text to images as a guidance model and choose the checkpoint of Stable Diffusion XL(^1^11[https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)). The guidance scale is 7.5 for all diffusion guidance models.

### A.2 More Qualitative Results

As shown in Figure 7, we present additional generation results. It can be observed that our Dreamer XL is capable of generating 3D models that are visually high-quality, closely approximate reality, and maintain good consistency.

Figure: Figure 7: More results generated by our Dreamer XL framework. Please zoom in for details.
Refer to caption: x6.png

### A.3 More Ablation Study

#### Ablation on offset rate γ 𝛾 \gamma italic_γ for simple scene

Here, we provide more ablation on offset rate $\gamma$ for simple scene. As shown in Figures 8 and 9, if $\gamma$ is set too low, the similarity between text and images decreases, and the images appear unrealistic. Raising $\gamma$ can mitigate the problem of accumulated errors, but it also leads to a loss of some details.

Figure: Figure 8: More Ablation on offset rate $\gamma$ for simple scene.
Refer to caption: x7.png

Figure: Figure 9: More Ablation on offset rate $\gamma$ for simple scene.
Refer to caption: x8.png

#### Ablation on pixel-by-pixel gradient clipping

Here, we present additional result in Figures 10 and 11. In Figure 10, once gradient clipping is not applied, the generated 3D model tends to be darker in color and lacks realism. In Figure 11, without gradient clipping, the generated objects appear more blurry, and it is evident from the depth map that there are numerous noise and spikes.

Figure: Figure 10: More Ablation on pixel-by-pixel gradient clipping.
Refer to caption: x9.png

Figure: Figure 11: More Ablation on pixel-by-pixel gradient clipping.
Refer to caption: x10.png

### A.4 Limitations

While our method can generate high-quality and relatively realistic 3D models, qualitative results show a significant limitation in our approach regarding light handling. Specifically, we have observed anomalous blue reflections in many scenes. Through our experiments, we have identified this problem as primarily caused by our use of SDXL. When SDXL is applied, the blue channel values in rendered images tend to be large, resulting in numerous areas exhibiting abnormal blue hues after normalization. Despite our attempts, including parameter adjustments and different normalization methods, we have yet to find a viable solution. We speculate that this may be attributed to the gradient or training data of SDXL. Additionally, it’s worth noting that while our research aims to enhance the quality of generated models, it may inadvertently contribute to the advancement of deepfake technology.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Please give clear examples of failure modes caused by ISM, and clear improvement from the same setting (prompt, seed, SD version) using TSM, as well as improved ablation studies and more abundant comparison to other SDS-based text-to-3D generation works.
soundness: 2
strengths: - The paper is well-written and easy to follow.
- The paper offers sound theoretical background on trajectory score matching, in which they modify ISM so that instead of conducting matching between intervals of a trajectory, two trajectories starting from same latent of a certain timestep are compared and used as a loss for gradient distillation into the 3D scene. Their explanation that this reduces error accumulation from DDIM inversion is intuitive and makes sense.
summary: This paper introduces TSM, which they argue is an improved for of Interval Score Matching (ISM) introduced by LucidDreamer, in which DDIM inversion process is leveraged to generate two paths for score distillation. The paper also introduces a pixel-by-pixel gradient clipping method, which according to their statement alleviates abnormal gradient from advanced pipeline.
weaknesses: - **Unclear motivation.** This paper's states that the main problem in previous text-to-3D framework (ISM) is "pseudo ground truth inconsistency" that derives from accumulation of error within the DDIM inversion process. However, the term "pseudo ground truth" is never clearly defined within the paper, failing to convince the reader of the necessity of their task. 
- **Lack of justification for the method.** Furthermore, the paper fails to clearly deliver critical failure modes caused by ISM due to the error accumulation caused by DDIM inversion: in order for its motivation to be meaningful, cases of failure or degradation resulting from ISM-based method, from which TSM improves upon, must be given. Such comparisons are crucial to the credibility of the paper, but comparison of results with those of ISM are not given within the paper, making the reader question the necessity of their contribution.
- **Lack of experiment.** The paper gives two contributions, TSM and pixel-level gradient clipping, but I believe the ablation studies to be lacking, with the results given in Appendix showing marginal performance improvement at best. In fact, most qualitative results in the paper show very little to marginal improvement from the results given in LucidDreamer (ISM).
- **Clarification on performance gain from using SDXL.** Table 1 shows quantitative evaluation and comparison to previous methods: however, major improvement in performance in the given table seems to be resulting from changing of diffusion model from SD 2.1 to SDXL, which is given and not related to any form of contribution stated in the paper. I believe all compared works utilize SD 2.1 or earlier version of Stable Diffusion, and in this setting TSM seems to show very marginal quantitative improvement.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to Weaknesses.
soundness: 3
strengths: The paper introduces a novel Trajectory Score Matching (TSM) method to address the accumulated error issue in Interval Score Matching (ISM), which is an interesting technical innovation. The authors demonstrate the advantages of TSM in reducing accumulated error and improving the stability of model-generated paths, supported by experiments. The paper is well-structured, and the discussion of related work helps readers understand the progress in the field. The authors' approach of using Stable Diffusion XL as guidance and proposing a pixel-by-pixel gradient clipping method to address the unstable gradient issues during 3D Gaussian splatting is practical.
summary: The paper presents a promising new method to enhance the visual quality and performance of text-to-3D generation. But there are some limitations.
weaknesses: 1. Although the paper proposes TSM to reduce the accumulated error in ISM, it remains to be further validated whether TSM can fully address the issue of accumulated errors, especially when dealing with more complex 3D models where the impact of accumulated errors might be more significant.
2. This paper use the powerful base model SDXL for comparison with the SD model-based methods is unfair because this paper is employing a stronger prior base model.
3. This paper utilizes the SDXL model, which is significantly slower than SD2-1 in 2D generation, suggesting it would be extremely slow for 3D generation. The paper does not discuss comparative experiments on time consumption.
4. The paper briefly mentions limitations, such as issues with light handling, but does not provide a detailed analysis of when the model fails or its limitations in complex scenarios.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Besides the optimization based methods, another line of work is feed-forward generation method, which directly train 3D generative models from large-scale 3D datasets. This work focuses on the optimization based method. The related work section can be made more clear on this and be more specific. 

In Figure 3, the difference between ISM and TSM appears to be primarily a matter of scale. Could you provide further analysis on this?

What kind of dataset is used for the quantitative evaluation in Table 1? It should be a commonly used benchmark, such as the T^3 benchmark.

In Table 1, what diffusion model is used by LucidDreamer? Is an eye-to-eye comparison ensured?
soundness: 2
strengths: The paper is well organized and clearly presented.
summary: This paper targets optimization based text-to-3D generation, aiming to resolve the inconsistent images generated by 2D DDPM used for supervising the training of 3D neural parameters in SDS. Specifically, it introduces Trajectory Score Matching (TSM), a method designed to improve upon the Interval Score Matching (ISM) technique introduced in LucidDreamer, addressing the inconsistent supervision issues present in SDS. Compared to ISM, TSM adds an additional noise latent on the trajectory, claiming it will help reduce the accumulated errors in the DDIM inversion. As the paper uses SDXL as the 2D diffusion prior, it additional proposes a pixel-by-pixel gradient clipping method to resolve the gradient instability issue.
weaknesses: The motivation for using the dual paths appears somewhat ad-hoc.

In Fig.2, notable inconsistencies still exist for the proposed TSM, such as variations in hair, face color, and neck.

Comparisons between TSM and ISM are shown only in Figure 5, which is insufficient to demonstrate the effectiveness of TSM.

There are no video demos showcasing the generated 3D results.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Regarding weakness 1, did the authors account for CFG when computing the "reconstruction term" in Figure 3?
   
2. Regarding weakness 3, what are the base diffusion models used for the baselines?

3. What exactly does the A-LPIPS metric in Section 4.2 refer to? This metric does not seem common in the field. The authors should cite relevant previous work clearly and provide a detailed explanation of this metric.

4. How many prompts and seeds did the authors experiment with in Section 4.2? The authors should report these numbers in the experimental details. Conducting experiments with at least 25 samples would provide a more robust measurement.
soundness: 1
strengths: 1. The paper clearly distinguishes the differences between the proposed TSM and ISM methods.
2. The introduction of depth gradient pruning could be a valuable engineering approach for practical applications.
summary: This paper proposes Trajectory Score Matching (TSM) to address the accumulated error of DDIM inversion used in ISM. Specifically, the authors added an additional inversion step to reduce the error in the loss to reduce the error. The authors also proposed depth gradient clipping to stablize the training.
weaknesses: 1. The theoretical foundation presented in this paper is not robust and sound. Specifically, the naive distillation objective in Eq. (7) lacks intuitiveness and requires further explanation in the context of this paper. Additionally, the proof of Proposition 1 appears to be incorrect, as it directly compares vectors (instead of their norm lengths). Without a complete proof, it is difficult to accept that TSM can effectively reduce the "errors." Figure 3 does not clarify the differences, and the main benefit of TSM seems to be the reduction of gradient variance, which is not well justified by quantitative measurements.

2. The motivation for reducing prediction error is inadequately explained. The inconsistencies shown in Figure 2 are unconvincing, as both ISM and TSM produce mostly identical results with only minor differences. The advantages of TSM in terms of reduced error are not clearly demonstrated, and when the difference between $\boldsymbol{\epsilon} _\phi(\boldsymbol{x} _t)$ and $\boldsymbol{\epsilon} _\phi(\boldsymbol{x} _\mu)$ is small, the impact of classifier-free guidance may be amplified. Moreover, if the goal is merely to reduce error, setting $t = \mu$ would eliminate the "error" to 0. Reducing the tunable parameter $\delta_T = t - s$ in ISM could also achieve error reduction. An experiment comparing ISM with TSM using the same $t - \mu = \delta_T$ is necessary.

3. There is a lack of fair comparison. The comparisons presented appear unfair, as the authors use two different base diffusion models in Table 1, with TSM using SDv2.1 showing only minimal improvements over ISM. This improvement may also originate from the increase in the impact of CFG as noted in weakness 2. As shown in figure 5, the differences between ISM and TSM is also minor. A fair comparison utilizing the same diffusion model is essential.