BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis

## Abstract

Abstract Text-to-3D synthesis has recently seen intriguing advances by combining the text-to-image models with 3D representation methods, e.g . , Gaussian Splatting (GS), via Score Distillation Sampling (SDS). However, a hurdle of existing methods is the low efficiency, per-prompt optimization for a single 3D object.
Therefore, it is imperative for a paradigm shift from per-prompt optimization to one-stage generation for any unseen text prompts, which yet remains challenging. A hurdle is how to directly generate a set of millions of 3D Gaussians to represent a 3D object .
This paper presents BrightDreamer , an end-to-end single-stage approach that can achieve generalizable and fast ( 77 ms ) text-to-3D generation.
Our key idea is to formulate the generation process as estimating the 3D deformation from an anchor shape with predefined positions .
For this, we first propose a Text-guided Shape Deformation ( TSD ) network to predict the deformed shape and its new positions, used as the centers (one attribute) of 3D Gaussians.
To estimate the other four attributes ( i.e . , scaling, rotation, opacity, and SH coefficient), we then design a novel Text-guided Triplane Generator ( TTG ) to generate a triplane representation for a 3D object.
The center of each Gaussian enables us to transform the triplane feature into the four attributes.
The generated 3D Gaussians can be finally rendered at 705 frames per second .
Extensive experiments demonstrate the superiority of our method over existing methods.
Also, BrightDreamer possesses a strong semantic understanding capability even for complex text prompts. The project code is available at https://vlislab22.github.io/BrightDreamer/ .

## 1 Introduction

Text-to-3D generation has recently received considerable attention in the computer graphics and vision community owing to its immersive potential across diverse applications, such as virtual reality and gaming [].

Recently, with the emergence of diffusion models [, ] and neural rendering techniques [, ], text-to-3D has witnessed an unprecedented technical advancement.
In particular, pioneering methods, such as DreamFusion [], LatentNeRF [], SJC [], have sparked significant interest in the research community, catalyzing a trend toward developing techniques for creating 3D assets from texts. The follow-up methods then focus on either quality improvement [, , , ] or geometry refinement [, ] or training efficiency [, ].

The dominant paradigm of these methods is to randomly initialize a 3D representation model, *e.g*., Neural Radiance Fields (NeRF) [] or Gaussian Splatting [], and optimize such a model to align with a specific text prompt, as depicted in Fig. 1 (a). Unfortunately, these methods suffer from two critical challenges. Firstly, as per-prompt optimization usually requires several tens of thousands of iterations, this inefficiency brings a considerable obstacle to broader applications. It is significantly different from the mainstream training paradigm in the field of 2D image generation [, ] or 3D-aware image generation [, , , , ]: a generative model is trained with a collection of text-image pairs or images, and the model can generate the desired content from any input at the inference stage.
Secondly, as demonstrated in Fig. 2(a), existing methods often fail to accurately process the complex texts. For example, the mainstream methods all fail to generate 3D content that input prompt contains complex interaction between multiple entities.
This limitation arises from the models being trained on a single text prompt, which results in a degraded capability in comprehensive semantic understanding.

Therefore, it is urgently needed for a paradigm shift from per-prompt optimization to develop a generic text-to-3D generation framework. Once trained, the generative framework should be able to generate content from any text prompts in the inference stage, as depicted in Fig. 1(b).
Previously, some research efforts, *e.g*., ATT3D [] and Instant3D [], have been made grounded in NeRF representation. The core insight is to add a large number of texts and take them as conditional inputs to generate explicit spatial representations, such as triplane []. Nonetheless, in stark contrast to the volume rendering in NeRF, 3D GS representation for an object usually consists of millions of 3D Gaussians. Consequently, there exists an inherent and natural difficulty in converting the generation representations into 3D GS ones in their framework.

Figure: Figure 2: DreamGaussian [] and LucidDreamer [] are both optimized for a single text. Our result is the direct generation. And for the display of our generalization, all the prompts do not appear in our training set. (a) is for showing the complex text understanding. (b) is for demonstrating our capability of generalization. It is noteworthy that light purple, deep purple, and light yellow don’t appear in the training set. (c) Interpolation between two prompts from color and shape perspective.
Refer to caption: x2.png

In this paper, we propose BrightDreamer, an end-to-end single-stage framework that, for the first time, can achieve generalizable and fast (77 ms) text-to-3D GS generation.
BrightDreamer exhibits a robust ability for complex semantic understanding (Fig. 2 (a)), and it demonstrates a substantial capacity for generalization (Fig. 2 (b)).
In addition, same as traditional generative models [], our generator can interpolate between two inputs (Fig. 2 (c)), which enables users to fully engage their imagination and creativity, expanding the potential for novel and nuanced design exploration.
As stated before, the 3D GS representation of an object usually consists of several millions of 3D Gaussians.
Consequently, directly generating such an extensive collection, is impractical.
Our key idea is to redefine this generation problem as its equal problem, *i.e*., 3D shape deformation.
Specifically, we place and fix some anchor positions to form the initial shape.
Then, it can be deformed to the desired shape by giving different input prompts through our designed Text-guided Shape Deformation (TSD) network (Sec. 3.1).
After the deformation, the new positions can be set to the centers of the 3D Gaussian.
Upon establishing the basic shape, we elaborately design a Text-guided Triplane Generator (TTG) to generate a spatial representation of the 3D object (Sec. 3.2).
Subsequently, we utilize the spatial feature of each center of 3D Gaussian to represent its whole feature and translate it into the remaining attributes (including scaling, rotation, opacity, and SH coefficient) through our well-designed Gaussian Decoder (Sec. 3.3).
For TTG, grounded on our re-analysis of the previous convolution-based triplane generation process, we have identified and solved two primary deficiencies that necessitate rectification.
One issue pertains to the spatial inhomogeneity observed during the calculation process, as illustrated in Fig. 4.
The other issue arises from the single-vector style control mechanism similar to StyleGAN [], which complicates the management of relationships between multiple entities.

Our contributions can be summarized as follows: (I) We propose BrightDreamer, the first 3D Gaussian generative framework to achieve generalizable and fast text-to-3D synthesis. (II) We design the Text-guided Shape Deformation (TSD) network to simplify the difficulty of direct generation of 3D Gaussians. We design the Text-guided Triplane Generator (TTG) to generate the object’s spatial features and then decode them as the 3D Gaussians. For TTG design, we re-analyze and solve the existing problems in the mainstream triplane generator, including spatial inhomogeneity and text understanding problems. (III) Extensive experiments demonstrate that BrightDreamer not only can understand the complex semantics (while the per-prompt optimization methods fail) but also can utilize its generalization capability to achieve generation control.

## 2 Related Works

Text-to-3D Generation.
Existing methods can be grouped into two categories.
1) Optimization-based methods typically commence with a randomly initialized 3D model, such as NeRF [], and subsequently employ text-image priors [, ] to guide and optimize its parameters.
After undergoing thousands of iterative refinements, this predefined 3D model progressively morphs to embody the shape described by the corresponding text input.
DreamField [] represents the inaugural foray into text-to-3D methodology, utilizing the pre-trained text-image model, CLIP [], as a guiding mechanism for the optimization process of a predefined NeRF model.
DreamFusion [] proposes the Score Distillation Sampling (SDS) to transfer the prior of the 2D diffusion model [] into a 3D representation model [, ], which achieves impressive performance and ignites the research enthusiastic for the text-to-3D task.
VSD [] and ISM [] are devoted to re-designing the SDS loss [], enabling much more local details of the 3D model.
MVDream [] and PerpNeg [] attempt to solve the Janus problem, *i.e*., multi-face problem in some text prompts.
2) Generation-based methods, by contrast, aim to directly generate a 3D model from a given text, streamlining the process of text-to-3D generation.
ATT3D [] is the first attempt to train a NeRF model with multiple texts.
Instant3D [] designs some modules to map the text input to the EG3D model [] and then use SDS to train this model. Nonetheless, different from the volume rendering in NeRF, 3D GS representation for an object usually consists of millions of 3D Gaussians. Therefore, it is inherently difficult to convert the NeRF representations into 3D GS ones. We propose BrightDreamer, a generic framework that, for the first time, can achieve fast (77 ms) text-to-3D GS generation. BrightDreamer exhibits a robust ability for complex semantic understanding (see Fig. 2).

3D Gaussian Splatting (GS).
Recently, 3D GS [] has become the most popular 3D representation method of 3D objects or scenes. 3D GS shows a faster rendering speed and higher application potential than NeRF [].
Within a short time, a large number of methods have been proposed to leverage 3D GS for diverse tasks, *e.g*., anti-aliasing novel view synthesis [, ], SLAM [, , , ], human reconstruction [, , , , , ], dynamic scene reconstruction [, , , , ], and 3D content generation [, , , , , ].
Our work is also based on 3D GS; however, we aim to develop a generic text-to-3D generation framework that can generate 3D Gaussians at a low latency (77 ms).

3D Generative Model.
The 3D generative model can be divided into two categories.
In a broad sense, NeRF [], 3D GS [], and their subsequent works [, , , , , ] can be seen as the novel view image generation from a set of input views.
From a narrow perspective, the objective of 3D-aware image synthesis and 3D object generation is to directly generate the 3D representation itself, without any input views.
For example, conditional NeRF [, , ] and conditional SDF [, ] take a latent code (Noise) as a condition to obtain different content.
EG3D [] and Instant3D [] directly generate a triplane to represent the space.
Differently, our BrightDreamer exclusively employs texts to train our 3D generative model, leveraging SDS.
And it outputs a real 3D GS model, which is different from a conditional NeRF network.
In addition, we introduce a better triplane design, which is different from EG3D and Instant3D.

Figure: Figure 3: An overview of our BrightDreamer. The details of Spatial Transformer, ResConv Block and Upsample Block are shown in Fig. 5.
Refer to caption: x3.png

## 3 Method

Overview. The objective of BrightDreamer is to directly generate 3D Gaussians in response to text prompts. After training, BrightDreamer is capable of generating the 3D Gaussians with a remarkably low generation latency (about 77ms). And, the generated 3D Gaussians can be rendered at an impressive inference speed of over 700 frames per second. 3D Gaussians can be defined by five attributes, namely the center $p^{\prime}$, scaling $S$, rotation $R$, opacity $\alpha$, and SH coefficient $SH$.
To directly generate 3D Gaussians, our key idea is two-fold: 1) Defining anchor positions, *i.e*., predefined positions, to estimate the center of 3D Gaussians; 2) Building implicit spatial representation, which can be decomposed to estimate the other four attributes of 3D Gaussians.

Intuitively, we propose BrightDreamer, and an overview is depicted in Fig. 3. Given a text prompt as input, we transform it to a $77\times 1024$ embedding through the frozen CLIP text encoder. Next, the TSD network (Sec. 3.1) transforms the fixed anchor positions to the desired shape with text guidance. The new positions are used as the centers of 3D Gaussians. We then design the TTG (Sec. 3.2) to separately generate three feature planes to construct the implicit spatial representation. Based on the centers of Gaussians, we can obtain their spatial features, which are then transferred to the other attributes through the Gaussian Decoder (Sec. 3.3). Finally, we render 3D Gaussians to 2D images and use the SDS Loss [] to optimize the whole framework. We now describe our BrightDreamer in detail.

### 3.1 Text-guided Shape Deformation (TSD)

The goal of TSD is to obtain the center (one attribute) of each 3D Gaussian.
Considering that directly outputting a huge number of center coordinates is extremely difficult, we overcome this hurdle by deforming the anchor positions instead of generating them.

Anchor Position.
The anchor position is predefined in this paper, which is a fixed coordinate.
It serves as one of the inputs for the TSD.
Specifically, we place the anchor positions on the vertices of a 3D grid, as represented by the gray points in Fig. 3.
Then, we design the TSD network to predict their deviation to deform the initialized shape of the 3D grid, guided by the text prompt input.

Network Design.
As shown in Fig. 3 (a), the inputs of TSD are text prompts and anchor positions.
Firstly, the text prompts are encoded as the text embedding by an off-the-shelf text encoder, *e.g*., CLIP [] or T5 [].
Considering the possibility of the complex input sentence, it remains non-trivial how to bridge each position and word in the sentence.
The cross-attention [] can quantify the correlation degree between each point and each word within a sentence. We then employ the cross-attention to design a module to obtain the deviation from the anchor position.
It consists of the Layer Normalization [], Multi-Head Attention, Feed-Forward Network, and shortcut connection [].
Consequently, certain positions, correlating more closely with corresponding words in the sentence, are assigned with higher attention scores.
This process enables the aggregation of features that more accurately reflect the characteristics of the corresponding words.
The detailed computation process is formulated as follows:

$$ $score=softmax(\frac{W_{Q}(p)W_{K}(y)}{\sqrt{d}}),$ (1) $$

$$ $h=score\cdot W_{V}(y),$ (2) $$

$$ $output=FFN(h),$ (3) $$

where $p\in\mathbb{R}^{3}$ is the 3D coordinate of the anchor position, $y\in\mathbb{R}^{77\times 1024}$ is the text embedding of the input prompt, 77 and 1024 are the sentence length and the embedding dimension, $W_{Q}(\cdot)$, $W_{K}(\cdot)$ and $W_{V}(\cdot)$ are the query, key, value transformation function, $d$ is the feature dimension, $score$ represent the attention score between words and points, $h$ is the intermediate feature, $FFN(\cdot)$ is the feed-forward network.

The output of the TSD network is the offset $\Delta\in\mathbb{R}^{3}$ of the anchor positions.
To ensure the stability of the training, we control the maximum extent to which a point can deviate from the anchor position.
Specifically, given the degree of freedom $\beta\in\mathbb{R}$, we use the following equation to adjust the range of output into interval $(-\beta,\beta)$:

$$ $\Delta=2\cdot\beta\cdot sigmoid(output)-\beta.$ (4) $$

Finally, we have the deformed position $p^{\prime}\in\mathbb{R}^{3}$, *i.e*. centers of 3D Gaussian, corresponding to input prompt, as follows:

$$ $p^{\prime}=p+\Delta.$ (5) $$

### 3.2 Text-guided Triplane Generator (TTG)

Upon determining the centers of the 3D Gaussians, we need to obtain the other four attributes.
To efficiently assign features to each Gaussian, the objective of TTG is to generate an implicit spatial representation in space, represented by the triplane.
Therefore, we design a novel and highly efficient triplane generator, whose input is also the text prompts.

One challenge is that the previous triplane generation approaches, such as EG3D [] and Instant3D [], exhibit the problem of spatial inhomogeneity, as shown in Fig. 4.
Since they directly segment a feature map into three feature maps along the channel dimension, only a few areas are computed together.
For example, the position (0, 0) in the 2D space is unfolded to (0, 0, :), (0, :, 0), and (:, 0, 0), denoted by blue color in Fig. 4.
Taking $1\times 1$ Conv as an example, only these three areas are calculated together.
On the contrary, (0, 0, :) is hardly possible to be calculated with (0, :, 1), because they do not appear at the same pixel in the 2D feature map.
The same applies to the $3\times 3$ Conv.
This means that only a few areas share the same spatial information, while the others do not, thus causing spatial inhomogeneity.
For this, a simple yet effective way is to apply three generators (without sharing weights).

Figure: Figure 4: The visualization of expanding 2D convolution kernel (cyan area) to 3D and its moving process in previous convolutional triplane generator []. We use $1\times 1$ convolutional kernel as an example. Only several positions are interacted, which leads to spatial inhomogeneous.
Refer to caption: x4.png

Another challenge is that given the complex prompts, squeezing a sentence into a single style feature vector to apply StyleGAN’s AdaIN operation [, , ] could result in a loss of local details.
Therefore, we need a more fine-grained generation method guided by the word level, thus can retain more information of text encoder trained on large-scale dataset.
Naturally, calculating cross-attention between the pixels of the feature map and words in the sentence is a better choice.

To address these two challenges, we design the Text-guided Triplane Generator (TTG), as shown in Fig. 3 (b).
Our TTG is designed with the inspiration from the spatial transformer block and residual convolutional block in Stable Diffusion []. Considering the increased computational demand associated with pixel-wise self-attention in the feature map of [], we do not incorporate this layer into our network.
Instead, we find that interleaved convolutional layers can sufficiently facilitate the interaction within the feature map.
The detailed designs are shown in Fig. 5.
For the whole pipeline, we first initialize a 2D input query according to its 2D trigonometric function position encoding.
Three ResConv Blocks and Spatial Transformer blocks are stacked to assemble the prompt word features at low resolution through cross-attention.
We then gradually increase the resolution of the feature map through the stacks of ResConv Blocks, Spatial Transformer blocks, and Upsample Block by five times, as depicted in Fig. 3 (b).
Finally, we use a Conv layer to output the plane feature. We describe the design of the Spatial Transformer block, ResConv block, and Upsample block in detail.

Figure: Figure 5: A detailed illustration of specific blocks. (a) Spatial Transformer Block. (b) Residual Convolutional Block. (c) Upsample Bock.
Refer to caption: x5.png

Spatial Transformer Block. As Fig. 5 (a) shows, the Spatial Transformer Block comprises two multi-head cross-attention modules and a feed-forward network [].
The process is initiated by flattening the 2D feature map into a 1D structure, thereby transforming the dimensions from $(H,W,C)$ to $(H\times W,C)$, with each pixel’s feature considered as the input query embedding.
Subsequent to this transformation, the features undergo normalization via Layer Normalization [].
The normalized features serve as queries, while the text embeddings act as keys and values in the computation of the cross-attention feature.
This cross-modality attention mechanism is designed to align the feature map with the corresponding words in the input sentence.
Following the application of two cross-attention modules, the features are further refined through a feed-forward network.
This sequence of operations also incorporates the use of skip connections, mirroring the original transformer architecture [], to facilitate effective feature processing and integration.

ResConv Block. As illustrated in Fig. 5 (b), the Residual Convolutional Block consists of a sequence of components: a Layer Normalization layer, followed by the application of the SiLU activation function [], and
$3\times 3$ convolutional layers.
Then a skip connection is established between input and output.

Upsample Block. As depicted in Fig. 5 (c), the Upsample Block begins with the interpolation of the feature map, enlarging it by a factor of $2\times$.
Following this upscaling, the enlarged feature map is further processed through a $3\times 3$ convolutional layer.

### 3.3 3D Gaussians Decoder

It aims to obtain the other four attributes of 3D Gaussian for achieving the generation.
Upon generating the triplane, comprised of the three feature planes $\pi_{xy}$, $\pi_{xz}$, $\pi_{yz}$, we can obtain the feature vector $\mathcal{F}\in\mathbb{R}^{32}$ of each Gaussian based on its center $p^{\prime}$.
Subsequently, the feature vector needs to be converted into additional attributes of 3D Gaussian, including opacity $\alpha\in\mathbb{R}$, scaling $S\in\mathbb{R}^{3}$, rotation $R\in\mathbb{R}^{4}$, and SH coefficient.

Specifically, we first project the 3D coordinate onto the three planes, X-Y, X-Z, and Y-Z.
Based on the projected 2D coordinates, we can derive the features $\mathcal{F}_{xy}$, $\mathcal{F}_{xz}$, $\mathcal{F}_{yz}$ according to the interpolation with their four vertex in the 2D feature maps.
To ensure that the gradient back-propagation is evenly distributed across all three planes, we utilize an averaging operation to aggregate these features, thereby obtaining the 3D Gaussian’s feature $\mathcal{F}$.
Given that the attributes of 3D Gaussian can be categorized into two groups, *i.e*., shape and color, we develop two distinct transformation modules $F_{shape}$ and $F_{color}$.
Each module is a lightweight, two-layer Multi-Layer Perceptron (MLP) network.
To enhance the gradient back-propagation to the TSD, the center of 3D Gaussian $p^{\prime}$ is additionally inputted into both modules:

$$ $\alpha,S,R=F_{shape}(\mathcal{F},p^{\prime}),\;SH=F_{color}(\mathcal{F},p^{ \prime}).$ (6) $$

In the training process, we observe the scaling $S$ is not a stable variation, and the memory consumption of 3D Gaussian rendering is extremely sensitive to it.
Therefore, we use the following equation to control it to interval $(a,b)$:

$$ $S=(b-a)\cdot sigmoid(S)+a.$ (7) $$

Upon obtaining all attributes of 3D Gaussian, our generation process is completed.
We can render it from arbitrary view direction to 2D images.

### 3.4 Optimization

Our training commences with the selection of $B$ prompts from the training set.
These prompts are then fed into our 3D Gaussians generator, which is tasked with generating the corresponding 3D GS representation of the objects.
Following this, we proceed to randomly sample $C$ view directions to render $C$ 2D images.
The $B\times C$ rendered images are supervised through the Score Distillation Sampling (SDS) loss function [], as Eq. 8 shows, in conjunction with the Perp-Neg [].
In this way, our generator can gradually construct a mapping relationship between text and 3D.

$$ $\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\phi,\mathbf{x}=g_{\theta}(prompt)) \triangleq\mathbb{E}_{t,\epsilon}\left[w(t)\left(\hat{\epsilon}_{\phi}\left( \mathbf{z}_{t};y,t\right)-\epsilon\right)\frac{\partial\mathbf{x}}{\partial \theta}\right],$ (8) $$

where $prompt$ is the input prompt of the generator, $\theta$ is the 3D Gaussians generator’s trainable parameters, $\phi$ is the parameters of denoising network, $x$ is the generated image by process $g_{\theta}(\cdot)$, $w(t)$ is the weighting function with time step $t$ in the denoising schedule, $\epsilon$ is the random noise, $z_{t}=x+\epsilon$, $y$ is the adjusted text according to the sampling view direction, $\hat{\epsilon}_{\phi}(\cdot)$ is the predicted noise.

## 4 Experiments

Figure: Figure 6: Generation Demonstration. All prompts do not appear in the training set.
Refer to caption: x6.png

### 4.1 Implementation Details

All our experiments are conducted on a server with 8 GPUs with 80GB memory.
Our codebase is constructed on the PyTorch framework [] with Automatic Mixed Precision Training (AMP).
And we introduce the Gradient Checkpointing technology [] to save the GPU memory.
We use the Adam optimizer [] to update the parameters of our generator with a constant learning rate of $5\times{10}^{-5}$, $\beta_{1}$ of $0.9$ and $\beta_{2}$ of $0.99$.
We train our generator using the images rendered at $512\times 512$ resolution by original rasterization of 3D Gaussian Splatting [] and use the DeepFloyd IF [] UNet to calculate the SDS Loss (Eq. 8).
The prompt batch size is set to $64$ in total and the camera batch size is set to $4$.
We set the freedom $\beta$ (Eq. 4) to $0.2$, and the range of scaling $(a,b)$ (Eq. 7) to $(-9,-3)$.
The anchor position is placed as a $64^{3}$ 3D grid, and the resolution of the generated triplane is $256\times 256$.

### 4.2 Datasets

Vehicles.
We construct a vehicle’s prompts set using ChatGPT(^1^11https://chat.openai.com/) to generate a description about car details, *e.g*., "vintage convertible, cherry red, chrome bumpers, white-wall tires".
There are 3,026 prompts containing 1,856 unique words in total.

Daily Life.
This dataset is constructed by Instant3D [].
They use the ChatGPT^1 to generate more than $17,000$ prompts containing $3,135$ unique words.
For example, "A tall man with a beard and wearing a leather jacket is riding a motorcycle".

Animals.
This prompt set is introduced by Instant3D [].
And the structure is "a {species} sitting {item} and wearing {gadget} and wearing a {hat}".
Species contains [’wolf’, ’dog’, ’panda’, ’fox’, ’civet’, ’cat’, ’red panda’, ’teddy bear’, ’rabbit’, ’koala’].
Item contains [’in a bathtub’, ’on a stone’, ’on books’, ’on a table’, ’on the lawn’, ’in a basket’, ’null’].
Gadget contains [’a tie’, ’a cape’, ’sunglasses’, ’a scarf’, ’null’].
Hat contains [’beret’, ’beanie’, ’cowboy hat’, ’straw hat’, ’baseball cap’, ’tophat’, ’party hat’, ’sombrero’, ’null’].
There are $3,150$ prompts in total.

### 4.3 Demonstration of BrightDreamer

To show the quality and view consistency of BrightDreamer, we show some results in Fig. 6.
All prompts do not appear in our training set.
Additionally, we show much more visual results in our supplementary details.

**Table 1: The generation latency (millisecond) and rendering speed (FPS, Frames Per Second) of our generator.**
| Device | Generation Latency | Rendering Speed |
| --- | --- | --- |
| RTX 3090 24GB | 79 ms | 698 FPS |
| A800 80GB | 77 ms | 705 FPS |

Figure: Figure 7: Vehicle generation comparison. All prompts don’t appear in the training set. The result of Dreamfusion is reproduced by ThreeStudio(^3^33https://github.com/threestudio-project/threestudio).
Refer to caption: x7.png

### 4.4 Comparison with Other Methods

Inference time.
In Tab. 1, we show the inference latency on a single A800 GPU and a single RTX3090 GPU.
BrightDreamer can generate a 3D GS representation for any text prompt in less than 0.1 seconds, which shows a large margin improvement, compared to optimization-based methods, which need several hours to optimize for one single prompt.
The generated 3D Gaussians can be rendered at over 700 FPS speed.

Figure: Figure 8: Visual comparison of daily life. All prompts don’t appear in training set.
Refer to caption: x8.png

Visual comparison results on vehicle prompts.
In Fig. 7, we show the comparison results. It’s noteworthy that other methods all need to train different models for different prompts.
In contrast, our method can directly infer one-time for a single prompt and all the test prompts do not appear in our training prompt set.
Because there is no open-source codebase of Instant3D [] and ATT3D [], we only show our result compared to other optimization-based methods.
Due to the car generation limitations remaining in Stable Diffusion, we use DeepFloydIF denoising UNet in Eq. 8.
However, it can only provide the $64\times 64$ supervision, so there remains some unfairness.
We first compare with SoTA NeRF optimization methods, dreamfusion [] and ProlificDreamer [].
As you can see, the generation result is extremely unreasonable and chaotic, which shows the difficulty when working on relatively complex prompts.
Though DreamGaussian’s training is relatively fast, 5 minutes, the generation result is still relatively bad.
Since LucidDreamer [] and GSGEN [] introduce the Point-E [] as their prior or supervision, they can generate the correct shape of a vehicle.
However, the LucidDreamer has abnormal light spots, which cause unreal sensing.
Our method can deal with arbitrary text prompts in 77 ms, which shows an extremely strong application value.
Importantly, you can scale up to check the second two columns in Fig. 7.
Looking through the window at the seats inside, there is an irregular phenomenon in the seat of GSGEN and LucidDreamer.
But ours looks reasonable, which shows that joint training can make the network have a stronger capability of semantic understanding.

Visual comparison results on daily life prompts.
In Fig. 8, we compare our method to Instant3D [] and other per-prompt optimization methods.
As you can see, nearly all per-prompt methods fail to generate reasonable content.
Either all entities in the prompt appear, but the interaction relationship is significantly wrong.
Either they lack some entities.
Though ProlificDreamer [] and LucidDreamer [] improve the original SDS loss, which allows them can get more details in the generated content, the consistency to the prompt is damaged a lot.
Our BrightDreamer not only uses 77ms to generate but also surpasses other methods in the aspect of rationality far away.
Additionally, the generated 3D Gaussians can be rendered over 700 FPS, which is 30 times than other NeRF-based methods, *e.g*. Instant3D.

Figure: Figure 9: The visual comparison of animal generation.
Refer to caption: x9.png

Visual comparison results on animal prompts.
In Fig. 9, we compare our method to Instant3D [] and other per-prompt optimization methods.
As you can see, if we only substitute some words in the sentence, our method still can remain the similar style.
Specifically, in the first and second two columns, we only change the thing which the teddy bear sitting on.
It is hard for per-prompt optimization-based methods to maintain the similarity of other parts.
And it is difficult for them to generate all entities that appeared in the input prompts.
There are also some wrong understandings of the input prompt, *e.g*., the rabbit case in GSGEN [].
In general, our BrightDreamer achieves the best ability of understanding and the highest speed in generation and rendering.

### 4.5 Ablation Studies

In Fig. 10, we demonstrate the influence of our network design for training.
Compared between Fig. 10 (a) and Fig. 10 (b), our divided triplane generator can reduce the degree of chaos in the space significantly, which shows the necessity of our division.
As Fig. 10 (a) and Fig. 10 (c) demonstrate, it is necessary to pass the coordinate into the $F_{shape}$ and $F_{color}$.
This design can construct a gradient pathway toward the TSD Network, ensuring more accurate shape formulation.

Figure: Figure 10: The key components in our design. All the models are trained to 10, 000 iterations with the same configuration. (a) Our completion design. (b) Replace the three separate generators with a single generator. (c) Don’t input the coordinate to $F_{shape}$ and $F_{color}$.
Refer to caption: x10.png

## 5 Conclusion

In this paper, we introduced the first text-driven 3D Gaussians generative framework, BrightDreamer, capable of generating 3D Gaussians within a remarkably low latency of 77ms.
To address the challenge of directly generating a vast quantity of Gaussians, millions of them, we innovatively deform anchor positions and use the new positions as the centers of 3D Gaussians, in response to the input prompt.
This approach effectively circumvented the obstacle of generating a large number of positions.
Regarding network architecture, we conducted a thorough reevaluation of the triplane generation process and introduced an improved alternative strategy.
Our largest contribution is poised to significantly advance the field of generalized 3D generation, offering a novel and efficient pathway to creating 3D assets from text prompts immediately.
A large number of experiments prove that our BrightDreamer has a strong complex semantic understanding capability and strong generalization ability.

Future work.
Both Instant3D and our approach are constrained to generating a singular 3D content for each input, which is different from GAN and diffusion models.
These traditional methods offer the flexibility to vary the generation outcomes by altering the input noise.
We aim to address this challenge in our future research endeavors, with the goal of introducing variability and diversity for a single input.

## Appendix 0.A Training Details of BrightDreamer

For a clearer illustration of our training pipeline, we provide the details in Algorithm. 1.
We train 36 hours for the vehicle prompts set, 60 hours for the daily life prompts set, and 30 hours for the animal prompts set on a server with 8 80GB GPUs.

Figure: Algorithm 1 Training Procedures of BrightDreamer

## Appendix 0.B Inference Procedures of Our BrightDreamer

In Algorithm. 2, we provide the inference details of our BrightDreamer.

Figure: Algorithm 2 Inference Procedures of Our BrightDreamer

## Appendix 0.C More Visual Results of Ablation Studies

In Fig. 12, We provide more visual results in ablation studies.

Figure: Figure 11: More ablation visual results. All models were trained to $10,000$ iterations with the same other configurations. (a) Our completion design. (b) Replace the three separate generators with a single generator. (c) Don’t input the coordinate to $F_{shape}$ and $F_{color}$. The prompt of the first line is "Muscle car, silver moon, hot hatch performance ". The prompt of the second line is "Vintage station wagon, cobalt blue, space-efficient design, economical ".
Refer to caption: x11.png

## Appendix 0.D Discussion about Generalization

In Fig. 2 of the main paper, we show the different combinations that don’t appear in the training prompts, *e.g*., ’deep purple’ and ’light purple’.
Here, we show the word that doesn’t appear in the training prompts may also be understood.
For example, ’banana’ doesn’t appear in the training process anymore.
However, as shown in Fig. 20 and Fig. 28, it can generate the corresponding color accurately.

## Appendix 0.E More Visual Results

From Fig. 12 to Fig. 51, we provide more multi-view results of our BrightDreamer.
Our method shows strong detail control ability.

Figure: Figure 12: Electric luxury SUV, apple red, spacious, advanced tech
Refer to caption: extracted/5476653/supple_img/Electric_luxury_SUV,_apple_red,_spacious,_advanced_tech.png

Figure: Figure 13: Electric luxury SUV, yellow, spacious, advanced tech
Refer to caption: extracted/5476653/supple_img/Electric_luxury_SUV,_yellow,_spacious,_advanced_tech.png

Figure: Figure 14: Electric luxury SUV, forest green, spacious, advanced tech
Refer to caption: extracted/5476653/supple_img/Electric_luxury_SUV,_forest_green,_spacious,_advanced_tech.png

Figure: Figure 15: Electric luxury SUV, cyan, spacious, advanced tech
Refer to caption: extracted/5476653/supple_img/Electric_luxury_SUV,_cyan,_spacious,_advanced_tech.png

Figure: Figure 16: Electric luxury SUV, deep blue, spacious, advanced tech
Refer to caption: extracted/5476653/supple_img/Electric_luxury_SUV,_deep_blue,_spacious,_advanced_tech.png

Figure: Figure 17: Electric luxury SUV, light purple, spacious, advanced tech
Refer to caption: extracted/5476653/supple_img/Electric_luxury_SUV,_light_purple,_spacious,_advanced_tech.png

Figure: Figure 18: Racing car, deep red, lightweight aero kit, sequential gearbox
Refer to caption: extracted/5476653/supple_img/Racing_car,_deep_red,_lightweight_aero_kit,_sequential_gearbox..png

Figure: Figure 19: Racing car, blaze orange, lightweight aero kit, sequential gearbox
Refer to caption: extracted/5476653/supple_img/Racing_car,_blaze_orange,_lightweight_aero_kit,_sequential_gearbox.png

Figure: Figure 20: Racing car, banana, lightweight aero kit, sequential gearbox
Refer to caption: extracted/5476653/supple_img/Racing_car,_banana,_lightweight_aero_kit,_sequential_gearbox..png

Figure: Figure 21: Racing car, green, lightweight aero kit, sequential gearbox
Refer to caption: extracted/5476653/supple_img/Racing_car,_green,_lightweight_aero_kit,_sequential_gearbox..png

Figure: Figure 22: Racing car, cyan, lightweight aero kit, sequential gearbox
Refer to caption: extracted/5476653/supple_img/Racing_car,_cyan,_lightweight_aero_kit,_sequential_gearbox..png

Figure: Figure 23: Family minivan, purple, large capacity, economical
Refer to caption: extracted/5476653/supple_img/Family_minivan,_purple,_large_capacity,_economical.png

Figure: Figure 24: Family minivan, yellow, large capacity, economical
Refer to caption: extracted/5476653/supple_img/Family_minivan,_yellow,_large_capacity,_economical.png

Figure: Figure 25: Family minivan, apple red, large capacity, economical
Refer to caption: extracted/5476653/supple_img/Family_minivan,_apple_red,_large_capacity,_economical.png

Figure: Figure 26: Family minivan, orange, large capacity, economical
Refer to caption: extracted/5476653/supple_img/Family_minivan,_orange,_large_capacity,_economical.png

Figure: Figure 27: Urban microcar, orange, ideal for city life, fuel-efficient
Refer to caption: extracted/5476653/supple_img/Urban_microcar,_orange,_ideal_for_city_life,_fuel-efficient.png

Figure: Figure 28: Urban microcar, banana, ideal for city life, fuel-efficient
Refer to caption: extracted/5476653/supple_img/Urban_microcar,_banana,_ideal_for_city_life,_fuel-efficient.png

Figure: Figure 29: Electric coupe, forest green, sleek design, autonomous features
Refer to caption: extracted/5476653/supple_img/Electric_coupe,_forest_green,_sleek_design,_autonomous_features.png

Figure: Figure 30: Electric coupe, pearl white, sleek design, autonomous features
Refer to caption: extracted/5476653/supple_img/Electric_coupe,_pearl_white,_sleek_design,_autonomous_features.png

Figure: Figure 31: Vintage convertible, orange, chrome bumpers, white-wall tires
Refer to caption: extracted/5476653/supple_img/Vintage_convertible,_orange,_chrome_bumpers,_white-wall_tires.png

Figure: Figure 32: Vintage convertible, apple red, chrome bumpers, white-wall tires
Refer to caption: extracted/5476653/supple_img/Vintage_convertible,_apple_red,_chrome_bumpers,_white-wall_tires.png

Figure: Figure 33: Vintage convertible, yellow, chrome bumpers, white-wall tires
Refer to caption: extracted/5476653/supple_img/Vintage_convertible,_yellow,_chrome_bumpers,_white-wall_tires.png

Figure: Figure 34: a man wearing a backpack is climbing a mountain
Refer to caption: extracted/5476653/supple_img/a_man_wearing_a_backpack_is_climbing_a_mountain.png

Figure: Figure 35: a woman wearing a backpack is climbing a mountain
Refer to caption: extracted/5476653/supple_img/a_woman_wearing_a_backpack_is_climbing_a_mountain.png

Figure: Figure 36: an elderly man wearing a backpack is climbing a mountain
Refer to caption: extracted/5476653/supple_img/an_elderly_man_wearing_a_backpack_is_climbing_a_mountain.png

Figure: Figure 37: an elderly woman wearing a backpack is climbing a mountain
Refer to caption: extracted/5476653/supple_img/an_elderly_woman_wearing_a_backpack_is_climbing_a_mountain.png

Figure: Figure 38: a man is trimming his plants
Refer to caption: extracted/5476653/supple_img/a_man_is_trimming_his_plants.png

Figure: Figure 39: a fat man is trimming his plants
Refer to caption: extracted/5476653/supple_img/a_fat_man_is_trimming_his_plants.png

Figure: Figure 40: a fat and elderly man is trimming his plants
Refer to caption: extracted/5476653/supple_img/a_fat_and_elderly_man_is_trimming_his_plants.png

Figure: Figure 41: an elderly man is trimming his plants
Refer to caption: extracted/5476653/supple_img/an_elderly_man_is_trimming_his_plants.png

Figure: Figure 42: a man is playing with a dog
Refer to caption: extracted/5476653/supple_img/a_man_is_playing_with_a_dog.png

Figure: Figure 43: a man wearing a backpack is playing with a dog
Refer to caption: extracted/5476653/supple_img/a_man_wearing_a_backpack_is_playing_with_a_dog.png

Figure: Figure 44: a man is playing with a dog on the lawn
Refer to caption: extracted/5476653/supple_img/a_man_is_playing_with_a_dog_on_the_lawn.png

Figure: Figure 45: a man is playing with a dog on the beach
Refer to caption: extracted/5476653/supple_img/a_man_is_playing_with_a_dog_on_the_beach.png

Figure: Figure 46: a man is mowing the lawn
Refer to caption: extracted/5476653/supple_img/a_man_is_mowing_the_lawn.png

Figure: Figure 47: a man wearing a hat is mowing the lawn
Refer to caption: extracted/5476653/supple_img/a_man_wearing_a_hat_is_mowing_the_lawn.png

Figure: Figure 48: a woman is mowing the lawn
Refer to caption: extracted/5476653/supple_img/a_woman_is_mowing_the_lawn.png

Figure: Figure 49: a woman in a long dress is mowing the lawn
Refer to caption: extracted/5476653/supple_img/a_woman_in_a_long_dress_is_mowing_the_lawn.png

Figure: Figure 50: A glamorous woman in a cocktail dress is dancing in the park
Refer to caption: extracted/5476653/supple_img/A_glamorous_woman_in_a_cocktail_dress_is_dancing_in_the_park.png

Figure: Figure 51: A glamorous woman in a cocktail dress is dancing on the beach
Refer to caption: extracted/5476653/supple_img/A_glamorous_woman_in_a_cocktail_dress_is_dancing_on_the_beach.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	How many iterations are needed to train this model?
2.	What are the parameter counts for the TSD and TTG modules, respectively?
soundness: 2
strengths: 1.	The model achieves fast 3D Gaussian generation, requiring only 77 ms.
2.	The paper finds that re-parameterizing the Gaussian’s scaling helps stabilize optimization.
3.	The architecture and model design are reasonable.
summary: The paper presents an amortized text-to-3D Gaussian generator trained with SDS loss. The framework consists of two modules: TSD, responsible for center deformation, and TTG, which generates other Gaussian attributes. The model achieves fast 3D Gaussian generation.
weaknesses: 1.	In Table 1, the timing for Latte3D should be less than 1 second. Additionally, the method is not ‘3D diffusion’ but a model trained with amortization. ATT3D is missing from Table 1.
2.	For the baseline comparison in Figures 7 and 8, it would be better to include SDS with the MVdream prior, which is much more robust and produce higher-quality shapes.
3.	The experimental results are not fully convincing. Specifically: 1) the baseline comparison does not include the improved version of SDS methods with MBdream; 2) in Figure 9, only one pair of results is presented to demonstrate the advantage of the complete design. For this ablation study, running an FID or CLIP score evaluation on a holdout set would better illustrate the complete design’s effectiveness compared to other settings.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See the weaknesses section.
soundness: 2
strengths: 1. This paper proposes a real-time text-to-3D synthesis method based on 3DGS, which does not require 3D data for training.
2. The proposed Text-guided Shape Deformation (TSD) and Text-guided Triplane Generator (TTG) are novel and provide a certain level of technical contribution.
summary: This paper introduces a real-time text-to-3D synthesis method called BrightDreamer, where a Text-guided Shape Deformation (TSD) network and a Text-guided Triplane Generator (TTG) are proposed to predict the attributes of a 3DGS representation.
weaknesses: 1. The quality of the generated content by the proposed method is not high enough. The generated texture looks over-smoothed and lacks realism, e.g., the rabbit in Figure 8. The generated geometry is incomplete, e.g., the car in Figure 9(a).
2. The comparison experiments are not fair enough. (a) The methods compared in Section 4.4 are insufficient. Some of the latest 3D diffusion methods should also be included in the comparison, e.g., Latte3D [1] and LGM [2]. (b) The author needs to ensure the quality of the generated content. For instance, although ProlificDreamer may have issues with multiple facets, the generated content in Figures 7 and 8 is clearly below its typical standard.
3. The summary of existing methods in Table 1 is not comprehensive. For example, there are GAN-based generators that also support text input [3-9].

[1] LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis. [2] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. [3] Text2shape: Generating shapes from natural language by learning joint embeddings. [4] CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation [5] Shapecrafter: A recursive text-conditioned 3d shape generation model. [6] Towards implicit text-guided 3d shape generation. [7] Autosdf: Shape priors for 3d completion, reconstruction and generation. [8] CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language [9] Hierarchical Text-Conditional Image Generation with CLIP Latents.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could the authors provide a comment on the quality-speed trade-off? How can this issue be potentially addressed? 

2. Could the authors comment on the diversity and the generalization of the model? 

3. Could the authors explain the choice of training data? How are the current training prompts selected? What happens if the prompts are just randomly sampled?
soundness: 2
strengths: The paper addresses an important problem in text-to-3D generation, where the current speed to generate a 3D object using optimization-based methods are relatively low, taking minutes on average to generate an object. By making the generation process entirely feed forward, the method directly samples a 3D object in milliseconds. 

The paper is quite well written. I appreciate Table 1 which clearly demonstrates the differences of the method with previous work.
summary: The paper presents a generic text-to-3D generation method. Instead of performing score distillation to optimize a single 3D representation one at a time, the proposed method aims for a single-stage solution, where a text prompt is used to generate a set of deformations from predefined anchored points as well as three feature planes. The generated deformed points are then used to query the feature planes to construct 3D Gaussians, where the positions of the Gaussians are at the deformed points themselves, and the remaining four attributes of the Gaussians (scale, rotation, opacity, spherical harmonic coefficients) are established from the feature planes. The results show that the proposed method can generate 3D objects from text prompts in just ~77 ms.
weaknesses: 1) The technical content suffers from two notable issues.  

- The training seems to be done for each class of object, e.g., vehicle, animal, etc. For per-prompt optimization methods, the benefit is that no such class definition is required. It remains unclear whether why such categorization is required, or minimum how many samples per class is required for training. Additionally, some data must be prepared for training. 

- While the inference speed is at milliseconds, there seems to have some fidelity-speed tradeoff. The generated 3D model quality is not as good as per-prompt optimization case.  

From the above points, my guess is that training a generative framework like the proposed method is actually a trade-off. It leads to some limitations in the generative process (limited to object categories) as well as some degradations in object quality. This makes it tricky to handle arbitrary prompts like in optimization case. 


2) The experiment results appear to be quite limited. Some results are not particularly convincing. 

- The qualitative results appear to be more blurry than optimization-based text-to-3D Gaussian methods. 
     The 3D models also do not have a lot of details, and the colors appear to be very saturated. 
   
- The generalization of the model (to unseen prompts) are not well demonstrated. In terms of diversity and unseen objects it seems the generalization is worse than an image diffusion model as the current training scale is way smaller. 
   
- Fig. 9: I do not see clear improvement between a, b, c. 
   
- The training is quite expensive, with 30+ hours on 8 x 80GB GPUs. 


3) The paper writing can be further improved by addressing the following issues: 

- Some newer works for per-prompt optimization methods can be cited. Currently some 2024 methods are missing. 
  [A] Taming Mode Collapse in Score Distillation for Text-to-3D Generation, CVPR 2024
  [B] DiverseDream: Diverse Text-to-3D Synthesis with Augmented Text Embedding, ECCV 2024

- The discussion of network blocks in 3.2 is quite long. Some of the details are not directly relevant to 3D generation and can be moved to appendix (Spatial Transformer Block, ResConv, UpSample). 

- Reduce the use of bold, italic text in the writing. Too much emphasis diluted the emphasis. 

- Some paragraphs are particularly dense. Some spaces between paragraphs should be reserved.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: As noted in the Weaknesses section, I believe the authors should clarify the motivation behind their approach and the network structure similarty with triplane meet gaussians. Therefore, I am giving a rating of 3. If the authors address these points, I would be open to raising my score.
soundness: 2
strengths: This paper consist of following strengths:

- **Introduction of 3D Gaussians to Amortized Text-to-3D Generation:**

This paper is the first to apply 3D Gaussians to the amortized Text-to-3D generation task, demonstrating higher efficiency compared to traditional methods that require training from scratch.

- **Reframing the 3D Gaussian Generation Problem:**

The paper transforms the challenge of generating 3D Gaussians into a deformation problem. This approach addresses the difficulty of generating complex objects that would typically require millions of Gaussian points, making direct generation infeasible.

- **Proposal of a Triplane Generator for Spatial Features:**

A novel triplane generator is introduced to produce spatial features, which can be decoded into Gaussian attributes. The method is further refined with several specific design improvements to enhance performance.

- **Experimental Validation of Efficiency:**

Extensive experiments validate the efficiency of the proposed approach compared to original Text-to-3D Gaussian generation methods.
summary: This paper introduces a novel amortized Text-to-3D Gaussian generation method, enabling the creation of 3D Gaussian structures from text prompts in a single forward pass. The proposed approach incorporates two key innovations:

- Anchor-based Deformation:
Traditional 3D Gaussian representations require numerous Gaussian points to model complex scenes, making direct generation challenging. To address this, the authors introduce a strategy where a small number of anchor points are fixed and subsequently deformed through a text-conditioned transformer. These deformed points act as the central points for generating the 3D Gaussian structures.
- Text-guided Triplane Generator (TTG):
The paper further proposes the use of a Text-guided Triplane Generator (TTG) to create a triplane structure. The anchor points are then utilized to query Gaussian features from the generated triplane.

To train the models, the Score Distillation Sampling (SDS) method is applied to rendered images, following the practice used in previous amortized Text-to-3D models. This ensures effective training of the networks. The authors conducted extensive experiments, demonstrating the effectiveness of their approach compared to traditional train-from-scratch Text-to-3D Gaussian methods.
weaknesses: Though this method achieves promising results through its proposed designs, the paper presents several weaknesses that need to be addressed:

**1.Novelty:**

While the paper is the first to reformulate 3D Gaussian generation as a deformation problem, this concept is not new in the field of explicit 3D generation. Similar deformation-based approaches have been explored in previous works, such as:

- Wang, Nanyang, et al. "Pixel2mesh: Generating 3d mesh models from single rgb images." Proceedings of the European conference on computer vision (ECCV). 2018.
- Wen, Chao, et al. "Pixel2mesh++: Multi-view 3d mesh generation via deformation." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

Since 3D Gaussians represent a novel but explicit 3D format, the deformation approach is straightforward to apply, suggesting that this work is more of an incremental adaptation than a fundamentally novel idea.

It is accessable to bring an old idea to a novel field. However, another critical issue lies in the representation ability. While Gaussian splatting typically involves millions of points to accurately capture scene details, this paper proposes using a fixed, smaller number of anchor points. The lack of discussion on how many anchor points are required for convincing results raises concerns about the expressiveness of the generated models. If the number is reduced too much, the method may struggle to represent complex scenes effectively. The authors should address this limitation by providing a more thorough analysis and justification for their approach.

**2.Network Structure:**

The network architecture closely resembles the one proposed by Zou et al. in:

- Zou, Zi-Xin, et al. "Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers." Proceedings of the IEEE/CVF CVPR. 2024.

Both networks feature a triplane decoder and a point-based querier. Although the authors claim to introduce improvements, the paper does not provide an ablation study to evaluate the impact of these modifications. A more in-depth discussion and comprehensive ablation studies are necessary to demonstrate the value of these improvements.

**3.Experiments:**

The experimental results focus only on comparisons with text-to-3D Gaussian generation methods. However, since the paper's task aligns more closely with amortized Text-to-3D generation, it should also be compared against those methods to ensure a fair evaluation.

Additionally, as mentioned earlier, an ablation study is needed to assess the effectiveness of the network design choices. This would clarify the role of individual components and provide insights into which modifications contribute to the improved performance.

**4.Writing:**

The paper introduces new terms, such as "Triplane Generator Division" and "Coordinate Shortcut," in ablation study, but fails to explain them clearly. As a result, it becomes difficult to understand the exact meaning and purpose of these concepts. Providing detailed descriptions of these definitions would improve clarity and make the paper more accessible to readers.