Progressive Multi-scale Triplane Network for Text-to-3D Generation

Authors: Han Yi, Zhedong Zheng, Xiangyu Xu, Tat-Seng Chua
Primary area: generative models
Keywords: Text-to-3D Generation, Diffusion Model

Abstract:
The challenge of text-to-3D generation lies in accurately and efficiently crafting 3D objects based on natural language descriptions, a capability that promises substantial reduction in manual design efforts and offers an intuitive interface for user interaction with digital environments. Despite recent advancements, effective recovery of fine-grained details and efficient optimization of high-resolution 3D outputs remain critical hurdles. Drawing inspiration from the efficacious paradigm of progressive learning, we present a novel Multi-scale Triplane Network (MTN) architecture coupled with a tailored progressive learning strategy. As the name implies, the Multi-scale Triplane Network consists of four triplanes transitioning from low to high resolution. This hierarchical structure allows the low-resolution triplane to serve as an initial shape for the high-resolution counterparts, easing the inherent complexity of the optimization process. Furthermore, we introduce the progressive learning scheme that systematically guides the network to shift its attention from prominent coarse-grained structures to intricate fine-grained patterns. This strategic progression ensures that the focus of the model evolves towards emulating the subtlest aspects of the described 3D object. Our experiment verifies that the proposed method performs favorably against contemporary methods. Even for the complex and nuanced textual descriptions, our method consistently excels, delivering robust and viable 3D shapes where other methods falter.

Decision: reject
Reviewer scores: 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730293778257

thread_type:
official_review

<thread_object>
creator: reviewer_kywy

<content>
summary:
This paper introduces a Multi-scale Triplane Network coupled with a progressive learning strategy to generate 3D objects with fine-grained details that align with input text. The generated 3D objects are suitable for 3D printing with minimal additional post-processing.

strengths:
1. The generated meshes are suitable for 3D printing, as shown in Figure 9.

2. The progressive learning strategy can be generalized to other SDS-based text-to-3D generation models. In SDS-based methods, it is a common phenomenon that 3D objects generated with a closer camera radius exhibit more fine-grained but less plausible overall geometry, whereas a larger camera radius results in the opposite.

weaknesses:
1. The results presented in Figure 2 are not satisfactory. For example, the "corgi" lacks fine-grained texture. 

2. Figure 4 lacks comparisons with more recent works, such as GSGEN [1]. The DreamGaussian model performs poorly in text-to-3D tasks, making it an unsuitable baseline. The performance of ProlificDreamer in Figure 4 appears inferior to that reported in the original paper.

3. The paper lacks references to other related works on Triplane-based 3D generation.

References: 

[1] Chen, Zilong, et al. "Text-to-3d using gaussian splatting." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

questions:
1. In Figure 2, are there results with more realistic color for "motorcycle" and "a cake in the shape of a train"? What specifically does "automatic color rendering is applied when a common color is applicable for such a category" mean?

2. Why is random sampling timestep applied at the end of training? The claim that "It reintroduces randomness to maintain the vibrancy of the coloration of the 3D model" needs supporting evidence. Please provide specific examples or experimental results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730383515204

thread_type:
official_review

<thread_object>
creator: reviewer_us97

<content>
summary:
In this work, the authors propose a text-to-3D generation method based on Score Distillation Sampling (SDS). Their key contribution is leveraging a multi-scale triplane representation, which is progressively upsampled during the optimization process. They also introduce a progressive learning scheme that samples different diffusion time steps across training iterations. The authors claim their proposed framework achieves favorable performance compared to existing methods.

strengths:
- The experimental results (both quantitative and qualitative comparisons) demonstrate that the proposed framework achieves comparable, and in some cases slightly better, performance than Latent-NerF, Stable-Dreamfusion, and DreamGaussian.
- The ablation studies highlight the effectiveness of various components in this work, including the multi-scale triplane network (MTN) and the progressive training schemes.

weaknesses:
Despite the paper's contributions, several weaknesses and concerns remain:

- Limited Technical Novelty:
    - Multi-scale representation in 3D generation is already well-explored:
        - XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. CVPR 2024.
        - Locally Attentional SDF Diffusion for Controllable 3D Shape Generation. SIGGRAPH 2023.
    - Two-stage optimization for SDS-based 3D generation was previously explored in:
        - Magic3D: High-Resolution Text-to-3D Content Creation. CVPR 2023.
    - Given these existing works, the multi-scale representation approach does not appear to be a significant contribution. Moreover, these works are not sufficiently discussed in the manuscript.
    - The progressive learning scheme, explored in Dreamtime [ICLR 2024], lacks novelty.
- Inadequate Comparisons:
    - The evaluation omits important baselines, including some cited in the related work section. High-quality 3D generation frameworks like Fantasia3D and MVDream are not compared. This omission weakens the claim that the proposed framework achieves favorable performance compared to existing works.
    - The comparison with ProlificDreamer is only qualitative, and the difference seems minimal. Including evaluation metrics as in Figure 5 and Table 1 is necessary to support the claim.

questions:
- Unclear Ablation Setting:
    - The resolution of the triplane used in a single triplane is not specified. The number of iterations for training the single triplane network is also unclear.
    - Without this information, it is difficult to assess whether the current ablation adequately validates the effectiveness of the modules. It is recommended to clarify these settings.

Overall, this work presents a new text-to-3D generative framework using Score Distillation Sampling. While the proposed framework achieves comparable, and in some cases better, performance compared to certain baselines, the novelty of employing multi-scale representations appears limited. Furthermore, the experimental results are insufficient to fully support the current claims, and some important settings are not clearly described. Given these concerns, I am inclined to recommend rejection of this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632624163

thread_type:
official_review

<thread_object>
creator: reviewer_svjy

<content>
summary:
The paper introduces the Progressive Multi-Scale Triplane Network (MTN) for text-to-3D generation, a method designed to improve the efficiency and detail quality of 3D objects derived from text descriptions. The MTN operates using a multi-scale triplane network with four hierarchical triplanes progressing from low to high resolution, enabling the model to progressively refine from global to fine details. Additionally, a progressive learning strategy is applied, where both camera radius and time step in the diffusion process adjust dynamically to shift focus from broad structures to intricate details. The approach aims to address the challenges of high-resolution optimization and fine-grained details in 3D object generation. Experimental results indicate that MTN achieves better text alignment, texture quality, and shape precision compared to existing models like Latent-NeRF and DreamGaussian.

strengths:
1.MTN's multi-scale triplane architecture enables efficient generation by progressing from low to high resolutions. This structured approach improves global coherence and local detail quality in the final output, allowing it to achieve a high degree of alignment with textual prompts.

2.The model’s progressively reducing camera radius and time step enables it to refine details incrementally, addressing a common limitation in text-to-3D generation where models often struggle with balancing global structure and fine-grained textures.

weaknesses:
1. Omission of LucidDreamer and Similar Works: The paper does not consider LucidDreamer (CVPR 2024) in its evaluations, a notable omission given that LucidDreamer also applies SDS-based multi-view generation with a focus on geometry extraction. Including LucidDreamer would provide a more complete and comparative assessment of MTN’s effectiveness in capturing fine details.

2. The claim of not using 3DGS as a representation sounds weird. The authors mentioned, " Though faster, these approaches often 
 compromise the high-quality characteristic of NeRF-based methods and require post-processing to  convert Gaussian representations into NeRF or meshes, adding computational overhead." However, triplane representation also needs to be converted to mesh if someone wants to use it as an explicit 3D model. Also, 3DGS-based methods are 10x faster than MTN, which is clearly greater than the time required to convert 3DGS into a mesh (if you use marching-cube as the converting algorithm).

3. In Figure 4, the third case contains a weird prompt "zoomed out DSLR photo". Since the input prompt only controls the rendered image but not the camera position itself, your result shows a smaller pancake pile and a smaller rabbit, which means your result is physically smaller. I think this is a little weird. This result implicitly implies that the generated objects might differ in terms of the object size, and this would hinder the practicability of this work since one will always need to check if your result needs to be resized, while other methods generally generate objects with the same size.

questions:
1. Please address the issue mentioned in the weakness part.
2. Why did the results of DreamGaussian look much worse than the results in their paper and project page? Is there any specific reason?
3. Why are there no limitations to this work discussed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675003622

thread_type:
official_review

<thread_object>
creator: reviewer_bfug

<content>
summary:
This paper proposes a multiscale triplane architecture for coarse-to-fine SDS text-to-3D generation. SDS losses are applied at each scale of the network, with the triplane’s resolution varying at different scales. During training, a specific time-step sampling and reduction of camera radius are used to help generate high-frequency details.

strengths:
1.	The paper presents a multiscale coarse-to-fine triplane architecture for SDS optimization. The coarse-to-fine design is reasonable.

weaknesses:
1.	The baseline comparison is not convincing enough. Using MVDream with the SDS method could yield much better results. It's much more robust and achieve better results compared to the current baseline (stable-dreamfusion). MVdream-SDS could be added to the Table 1 with the R-Precision.   
2.	The texture frequency of the shape still appears insufficient sometimes. In Figure 7, the results seem to exhibit higher texture frequency; however, in Figure 2, many shapes still appear blurry, with some having very uniform color (motorcycle and the train, and the body of the corgi and bunny). Any idea why this the difference of quality is this large?

questions:
1.	If you render the shape from the triplane at different scales, what do they look like? Is the model truly able to disentangle different structural scales into separate triplanes? It may provide more insight if you could visualize the shape at different scales.
2.	Currently, the highest resolution is 512. Could you scale it further to 1024?
3.	What is the resolution of the rendered image at each scale?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method for improving text-to-3D generation, using a coarse-to-fine hierarchical learning strategy built on triplanes. Reviewers comment on the reasonable design choices, leading to improved details and text adherence, but also raise significant concerns about limited technical novelty, as multi-scale representation and progressive learning have been well explored in previous works. The evaluation was criticized for inconsistent quality of results between figures and missing comparisons with important recent baselines. During the discussion phase, the authors addressed some concerns but reviewers remained unconvinced about the method's advantages over existing approaches. The consensus view suggests that while the work has practical merit, it needs more comprehensive evaluation and clearer demonstration of technical novelty to be truly compelling.