A3D: Does Diffusion Dream about 3D Alignment?

Authors: Savva Victorovich Ignatyev, Nina Konovalova, Daniil Selikhanovych, Oleg Voynov, Nikolay Patakin, Ilya Olkov, Dmitry Senushkin, Alexey Artemov, Anton Konushin, Alexander Filippov, Peter Wonka, Evgeny Burnaev
Primary area: generative models
Keywords: Aligned 3D generation, 3D generation and editing, Text-to-image diffusion, Score distillation sampling, Implicit neural surfaces, Radiance fields

Abstract:
We tackle the problem of text-driven 3D generation from a geometry alignment perspective. Given a set of text prompts, we aim to generate a collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain a set of objects aligned with each other. In order to achieve the alignment of the corresponding parts of the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of our method.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730539953888

thread_type:
official_review

<thread_object>
creator: reviewer_dfs8

<content>
summary:
This paper focuses on generating a set of objects with similar structures using one text-to-3D generation model. To this aim, it introduces a  latent code to embed these desired objects into a common space. Besides training the text-to-3D generation model with their individual texts, this work enforces two kinds of regularization on the transitions between the objects. Experiments on multiple aligned 3D objects generation, combining parts of different objects into a ''hybrid'' and preserving transformations of a 3D object verify that the proposed method can generate diverse objects while maintaining structural consistency.

strengths:
1. This paper introduces an interesting problem, how to make one text-to-3D generation model flexibly produce different objects while maintaining structural consistency?
2. The proposed method proposes to embed these objects into a common latent space, allowing them to share similar structures.  
3. Experiments show the potential of the proposed method on different tasks, including generation of multiple objects, hybridization and structure-preserving transformation.

weaknesses:
1. The paper is not well-written. It does not provide an overview to introduce the method architecture. It is better to provide a high-level flowchart of the proposed method, and then introduce the components of the proposed method by referring them to the presented flowchart. In the Method section, many details are given in the appendix.
2. The experiments lack the ablation studies for the two kinds of regularization on the transitions between objects. Can you remove the two regularizations, respectively and conduct corresponding ablation experiments to verify their effectiveness?
3. Can the proposed method be applied to image-to-3D generation models? Can you discuss the potential challenges or modifications needed to adapt their method to image-to-3D models, and how this might impact the structural consistency of the generated objects?

questions:
1. I am curious if the the proposed method can be applied to image-to-3D generation methods. Since one image can be converted into a text embedding, the proposed method should support the image-to-3D setting.
2. What is the training time overhead for the proposed method?  Can you compare the training time with other methods and discuss how the training time scales with the number of objects being aligned?
3. For the ablation study, Table 3 only compares the DIFT distance metric. Can you also compare other metrics like Table 1 and Table 2?
4. I wonder if the proposed method can edit some details. For example, "a happy running man", "a sad running man", "a happy running minotaur" and "a sad running minotaur".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618743587

thread_type:
official_review

<thread_object>
creator: reviewer_ngys

<content>
summary:
The paper proposes a method that enables the joint generation of a collection of 3D objects with semantically corresponding parts aligned across them.  A3D addresses 3D editing and consistent generation by embedding the 3D objects and the transitions between them into a shared latent space and enforcing the smoothness and plausibility of these transitions. This allows for several practical scenarios that benefit from alignment between the generated objects, including 3D editing, object hybridization, and structure-preserving transformations.

strengths:
1. Proposed A3D can jointly generate a collection of 3D objects aligned in structure from a set of text prompts. This method can be applied to various scenarios, such as 3D editing, object mixing, and structure-preserving transformation.
2. Compared to existing methods that generate 3D objects independently, A3D can generate a collection of 3D objects that are structurally consistent, text-aligned, and of high visual quality.
3. A3D can be used in conjunction with different text-to-3D generation frameworks to improve the generation quality while maintaining the structure.
4. The results are interesting.

weaknesses:
1. Lacking a diagram that can illustrate the process of the transition and latent coding process. Only the text description on this part might be not clear enough.

2. The generated results are confined to the formed shape in the latent space, which might result in unnatural and unpleasant results. e.g., in Fig. 6, the crab and ant share the same latent shape but the generated assets are not reasonable enough for both(Too many legs for the ant and no claws for the crab).

questions:
1. Can it be generalized to other diffusion models like Stable Diffusion-based SDS?

2. Is it capable of performing on Gaussian splatting-based paradigms?

3. Can this method generate diverse sets of 3D results given one single set of text prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713234970

thread_type:
official_review

<thread_object>
creator: reviewer_b2fb

<content>
summary:
This paper proposes A3D, a new method to generate structurally-aligned 3D objects from a set of text prompts. A3D can first generate multiple aligned 3D objects, then make hybrids combined of different parts from the generated objects. Another advantage is that A3D can achieve smooth structure-preserving transformation given a coarse mesh.

strengths:
1. The overall idea is interesting. The authors observe that the previous 3D editing method fails to generate structurally-aligned high-quality 3D objects (e.g., either lack of structure alignment or detailed texture). A3D provides an effective way to generate aligned 3D objects by introducing additional latent code conditions.
2. The results in Figure 3 demonstrate that A3D indeed learns a smooth transition between different latent codes and can be decoupled with 3D positions by setting anchor points.
3. The authors provide sufficient experiments and comparisons with previous works and A3D has better performance in generating aligned objects.
4. The presentation is clear and easy to follow. Some visualizations are useful for understanding the core idea of learning smooth transitions.

weaknesses:
1. I noticed that some generated textures from A3D are still poor. Is this caused by the limitation of SDS loss or the framework of A3D?
2. I am wondering whether A3D is sensitive to text prompts. It would be better if the author could evaluate this point. is A3D robust to various text prompts?
3. In section 4.1, A3D uses some regularization strategies to supervise the network, including limiting the depth of the neural network and enforcing the smoothness of the rendered results. which will cause performance degeneration compare with raw SDS-based optimization.

questions:
See the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716407034

thread_type:
official_review

<thread_object>
creator: reviewer_2stx

<content>
summary:
This paper tackles an interesting problem of hybridization and editing regarding geometry-aligned text-to-3D generation, in which they leverage 3D shape and transitional embedding between different prompts to ensure continuous transition within the same 3D representation. By embedding these objects into a common latent space, they achieve both seamless hybridization of different shapes as well as high-quality 3D alignment between the generated scene and template 3D shape.

strengths:
- This paper is well-written and easy to follow.
- The motivation the paper is sound, tackling the task of 3D shape editing through addressing hybridization and alignment of difference 3D scenes corresponding to the given shape, which I believe is an innovative and novel solution to the task at hand.
- The technical solution this paper offers to the problem is simple and effective: training a single NeRF model to handle multiple scene from different prompts during the optimization phase, and then allowing for arbitrary translation between the different scenes with interpolation of latent vectors is a elegant approach to achieve 3D alignment and controllable 3D editing of the scene.

weaknesses:
- This paper uses NeRF for seamless representation of difference scenes, and I understand design choice was to enable seamless transition between different optimized scenes within the representation space of NeRF. However, more explicit 3D representations such as Instant-NGP or 3DGS offer their own advantages, such as speed and controllability, and I believe are a viable candidates for comparison. Can this method be applied to such different forms to 3D representation? If so, how do their performance different from the current model that uses NeRF?

- How much 'deformation' from the given template shape does this method allow for? Figure 4 shows the generated shapes display variations in geometry from the given template - is there any mechanism within the paper that specifically allows for this "loose" alignment (SD operating in the latent space, perhaps)? How does 3D geometry look in the "middle states" of transitioning the entire network from one object to another? 

I am very much willing to raise the score if the mentioned questions are addressed during the discussion phase.

questions:
See the weakness section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717493828

thread_type:
official_review

<thread_object>
creator: reviewer_r6mh

<content>
summary:
This paper tries to tackle the problem of text-driven 3D generation from a geometry alignment perspective. To achieve the alignment of the corresponding parts of the generated objects, this paper tries to optimize the continuous transitions in a shared latent space. Visualization results demonstrate the effectiveness.

strengths:
1. The motivation of optimization in a shared latent space for smooth transitions is reasonable.

2. This writing style is easy to follow.

weaknesses:
1. Fairness issues. This work is built on MVDream, which highly relies on the pre-trained knowledge from SD2.1. This raises the question: compared with other approaches, does the effectiveness of A3D come from the method itself or the pre-trained knowledge of the SD model? Relevant experiments are needed to demonstrate it. Besides, the authors are encouraged to compare different SD versions, like SD1.5 and SDXL.
2. In Tab.2, MVEdit outperforms A3D on CLIP score and DIFT distance. The authors claim that A3D can add or remove significant parts. However, this indirectly indicates that pre-trained SD models may cause unfair knowledge leakage.
3. What is the training cost compared with baseline methods, like MVEdit, LucidDreamer, and GaussianEditor?

questions:
See Weaknesses
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, all reviewers vote for acceptance. After checking the paper, the AC acknowledges the technical novelty proposed in this paper and agrees on the acceptance.

### Pros
1. This paper is well-written and easy to follow.
2. The motivation to tackle the task of 3D shape editing through addressing hybridization and alignment of different 3D scenes corresponding to the given shape.
3. Sufficient experiments and comparisons with previous works to support the effectiveness of generating aligned objects.

Please polish the manuscript, given the comments the reviewers point out. For example,
1. Reporting the computation cost or training/inference time.
2. More applications of the proposed method to other 3D representations.
3. More ablation studies.