A3D: Does Diffusion Dream about 3D Alignment?

## Abstract

Abstract We tackle the problem of text-driven 3D generation from a geometry alignment perspective.
We aim at the generation of multiple objects which are consistent in terms of semantics and geometry.
Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality objects represented by 3D neural radiance fields.
These methods handle multiple text queries separately, and therefore, the resulting objects have a high variability in object pose and structure.
However, in some applications such as geometry editing, it is desirable to obtain aligned objects.
In order to achieve alignment, we propose to optimize the continuous trajectories between the aligned objects, by modeling a space of linear pairwise interpolations of the textual embeddings with a single NeRF representation.
We demonstrate that similar objects, consisting of semantically corresponding parts, can be well aligned in 3D space without costly modifications to the generation process.
We provide several practical scenarios including mesh editing and object hybridization that benefit from geometry alignment and experimentally demonstrate the efficiency of our method. voyleg.github.io/a3d

## 1 Introduction

The process of 3D model design requires substantial manual effort from artists and is known to be labor-intensive.
At the same time, recent advances in generative modeling have demonstrated that 3D asset creation can be greatly simplified.
Recent methods in text-driven 3D generation have made significant progress in generating diverse objects from textual descriptions by leveraging a strong 2D diffusion prior [, , , , , , ].
However, these methods are unsuitable for generating a collection of objects with a similar geometric structure, as these methods do not consider alignment in geometry and semantics when producing multiple objects (see [Figure 2](#S1.F2), top).
In our work, we tackle three related tasks that require the integration of alignment in the text-driven 3D object generation process.

The first one is the generation of a collection of coherent 3D objects that are semantically and geometrically aligned (see [Figure 1](#S0.F1), top).
Having a variety of aligned objects provides the artist with a wider range of options for replacing assets in a scene or using the same animations for a set of objects.
The second task is 3D object hybridization, which refers to the generation of a merged object by combining parts from different aligned objects (see [Figure 1](#S0.F1), middle).
An artist can swap out different parts, experiment with different configurations, and adjust individual elements without affecting the overall structure of an asset.
The third task is pose-preserving 3D object transformation, which aims to transform one 3D model into another, keeping the pose of the object intact (see [Figure 1](#S0.F1), bottom).
A special case of this task is the generation of an object aligned with a very coarse geometric model.
A coarse model is easy to design and manipulate, and pose-conditioned generation allows an artist to quickly modify and experiment with different poses of a generated object while the automatic generation process can fill in more complex geometric details and textures.

We propose *A3D*, a general framework for all these three tasks.
Like the recent methods for text-driven 3D generation, our method represents a 3D object with a Neural Radiance Field (NeRF) [] and trains it with a text-to-image denoising diffusion model via Score Distillation Sampling (SDS) [].
To achieve consistency w.r.t. semantics and geometry, we propose to embed 3D objects into a common higher-dimensional space together with transitions between them.
This enables us to regularize the transitions to make them smooth and thus encourage alignment between the objects.
We parameterize this space using a single NeRF and propose a new approach for training it with SDS.
Specifically, we train it to map the transition trajectories between the elements from the space that embeds their text descriptions into the space that embeds their 3D models.
In contrast, the existing methods learn to map the text descriptions to the 3D models independently for different object-prompt pairs.

We demonstrate the effectiveness of our approach in the three tasks described above.
In summary, our technical contributions are:
(1) an approach for training an implicit neural representation with Score Distillation Sampling that allows us to embed multiple 3D models described with text together with the transitions between them into a common space;
(2) a set of methods for text-driven generation of aligned 3D objects, composition of these objects, and transformation of objects.
Overall, our work advances the state of the art in text-driven 3D generation and opens up new possibilities for applications requiring the generation of semantically and geometrically-aligned objects.

Figure: Figure 2: When generating 3D shapes with existing methods (top row), there is no or only accidental alignment between the 3D shapes. Our framework A3D (bottom row) can generate a set of objects with aligned semantics and geometry.
Refer to caption: /html/2406.15020/assets/x2.png

## 2 Related work

#### Diffusion-based 2D image generation and manipulation.

Recently developed diffusion models, such as Denoising Diffusion Probabilistic Models (DDPM) [] have found wide applications in various computer vision tasks, such as text-driven image synthesis [, , ], video generation [, , , ] and inpainting [, ]. Additionally, they are extensively used in diverse image manipulation tasks [, , , , ]. Various methods leverage these diffusion models for image editing using text prompts. These methods include text inversion [, ], diffusion inversion [, , ], additional diffusion tuning [, ], editing using instructions [] or improving the generation strategy [, ]. Overall, 2D diffusion models have achieved significant success in text-guided image generation and manipulation tasks, offering fine-grained control over image attributes and visually appealing results.

#### Text-driven 3D asset generation.

The utilization of 2D priors for 3D generation has proven to be a successful strategy for 3D generation. The early works tried to apply a pre-trained cross-modal CLIP [] to guide the optimization of 3D neural radiance fields (NeRF) [, , ], vertex-based meshes [, , ] or point clouds []. These works demonstrated that 2D text-to-image models could be employed for 3D generation. However, the results suffered from a lack of quality and realism. DreamFusion [] proposed Score Distillation Sampling (SDS) that unlocked the ability to use a pre-trained 2D diffusion model for NeRF training guidance. Alternatively, [] introduced Score Jacobian Chaining (SGC) for utilizing 2D diffusion priors for text-to-3D generation.
Further methods built on DreamFusion to improve the quality and speed of 3D generation in different ways. Magic3D [] introduced a coarse-to-fine optimization technique to improve the optimization speed and to increase resolution. Fantasia3D [] disentangled the geometry and texture training. ProlificDreamer [] achieved high-quality intricate generation by employing variational SDS. Moreover, adversarial training [], 3D-view conditioned diffusion models [, , , , , ], and Gaussian splatting-based models [, ] were utilized to enhance realism, detail, and optimization speed.
SDS-based 3D generation methods produce results with some degree of shared canonical orientation, due to their view-dependent prompt sampling strategy.
Still, they suffer from the so-called “Janus problem” where the method generates the object with multiple faces.
MVDream [] addressed this issue by modifying a large-scale 2D diffusion model for the multi-view setting and fine-tuning it on a dataset of 3D objects, thus improving the 3D consistency of the results.
This helps to improve the consistency within the generated objects but still does not offer any consistency between different generated objects.
We build our framework on the high visual quality of the existing models, specifically, MVDream.
The main goal of our work is to offer additional mechanisms for better control of the spatial positioning of the 3D output.

#### Text-driven 3D asset editing.

Several methods have been proposed to manipulate NeRF-based scene representations via text [, , , ].
DreamBooth3D [] and Magic3D [] showed the possibility of editing personalized objects.
FocalDreamer [] and Vox-E [] have proven the possibility of local editing restricted to a specific spatial region.
In contrast, we focus on coherent pose-preserving global editing or stylization of the whole object.

Some works show examples of text-driven globally-consistent transformation of one object into another [].
They start with one 3D model and iteratively optimize it towards consistency with the text prompt through Score Distillation Sampling, which does not guarantee the preservation of the pose.
Other works [, ] suggest using the SDS loss in combination with 2D pre-trained InstructPix2Pix network [], specialized in image editing.
MVEdit [] goes one step further, by avoiding SDS and proposing a special mechanism that coordinates 2D editing from different viewpoints.
While this approach performs re-texturing and local geometric deformation relatively well, it struggles with major transformations of the 3D model, as we show in our experiments.
In contrast, our method optimizes the complete transition process between the different objects, which allows us to preserve the alignment while making significant changes to the 3D model.

## 3 Preliminaries

#### Neural radiance fields.

Neural radiance field (NeRF) [] was originally introduced as a fully differentiable volume rendering approach that represents the scene as a continuous radiance function parameterized with a fully-connected neural network.
This network maps a 3D point $\bm{\mathrm{\mu}}\in\mathbb{R}^{3}$ along with a view direction $\bm{\mathrm{d}}\in\mathbb{S}^{2}$ into a volumetric density $\tau\in\mathbb{R}^{+}$ and a view-dependent emitted radiance $\bm{\mathrm{c}}\in\mathbb{R}^{3}$ at that spatial location.
To render an image, NeRF queries 5D coordinates $(\bm{\mathrm{\mu}},\bm{\mathrm{d}})$ along camera rays and uses classic volume rendering techniques to project the output colors and densities into the image.
The ray color $\bm{\mathrm{C}}$ is calculated numerically through quadrature approximation:

$$ $\bm{\mathrm{C}}={\textstyle\sum}_{i}\alpha_{i}T_{i}\bm{\mathrm{c}}_{i},\qquad T_{i}={\textstyle\prod}_{j<i}1-\alpha_{i},\qquad\alpha_{i}=1-\exp(-\tau_{i}\lVert\bm{\mathrm{\mu}}_{i}-\bm{\mathrm{\mu}}_{i+1}\rVert),$ (1) $$

where $\bm{\mathrm{c}}_{i}$ and $\tau_{i}$ are the radiance and density queried at the $i$’th position along the ray, and $\alpha_{i}$ and $T_{i}$ are the transmittance and accumulated transmittance.

Originally, NeRF is iteratively trained from a set of posed images.
At each iteration, a batch of camera rays is randomly sampled from the set of all observed pixels and the photometric deviation between the colors $\hat{\bm{\mathrm{C}}}_{k}$ observed along the $k$’th ray and $\bm{\mathrm{C}}_{k}$ rendered via [Equation 1](#S3.E1) is minimized:

$$ $\mathcal{L}_{\mathrm{c}}={\textstyle\sum}_{k}\lVert\bm{\mathrm{C}}_{k}-\hat{\bm{\mathrm{C}}}_{k}\rVert_{2}^{2}.$ (2) $$

#### Score distillation sampling.

Score Distillation Sampling (SDS) [] was proposed for fitting a NeRF to a text description of the 3D scene, without any input images, using a pre-trained text-to-image diffusion model.
The idea is to iteratively guide the NeRF towards consistency with the text prompt by using the text-conditioned diffusion model as a critic for images rendered from the NeRF.
At each iteration, the image $\bm{\mathrm{x}}$ is rendered for a randomly sampled camera position.
A random Gaussian noise $\bm{\mathrm{\epsilon}}\sim\mathcal{N}(\bm{\mathrm{0}},\bm{\mathrm{I}})$ is added to the image and the output of the denoising diffusion model $\mathcal{E}$ is obtained via
$\hat{\bm{\mathrm{\epsilon}}}=\mathcal{E}(\bm{\mathrm{y}},t,\alpha_{t}\bm{\mathrm{x}}+\sigma_{t}\bm{\mathrm{\epsilon}})$,
where $\bm{\mathrm{y}}$ is the embedding of the text prompt, $t\sim\mathcal{U}(0,1)$ is the diffusion timestep, and $\alpha_{t}$ and $\sigma_{t}$ are weighting factors (see [] for more details).
The weights $\theta$ of the NeRF network are then updated using the gradient of the SDS loss term:

$$ $\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}=\mathbb{E}_{t,\bm{\mathrm{\epsilon}}}\left[w(t)(\hat{\bm{\mathrm{\epsilon}}}-\bm{\mathrm{\epsilon}})\partial_{\theta}\bm{\mathrm{x}}\right],$ (3) $$

where $w(t)$ is another weighting factor.

The authors of SDS use a scene representation slightly different from the original NeRF.
Their NeRF-like network $F$ maps the 3D point $\bm{\mathrm{\mu}}$ into volumetric density $\tau$ and the diffuse RGB reflectance $\bm{\mathrm{\rho}}\in\mathbb{R}^{3}$ (albedo) instead of the emitted radiance $\bm{\mathrm{c}}$, *i.e*., $(\tau,\bm{\mathrm{\rho}})=F\left(\bm{\mathrm{\mu}};\theta\right)$.
To obtain the emitted radiance and render the ray color via [Equation 1](#S3.E1), they introduce lighting to the scene, which they randomly sample at each training iteration, and compute the emitted radiance via

$$ $\bm{\mathrm{c}}=\bm{\mathrm{\rho}}\odot\bm{\mathrm{l}}(\bm{\mathrm{\mu}},\bm{\mathrm{n}}),\qquad\bm{\mathrm{n}}=-\nabla_{\bm{\mathrm{\mu}}}\tau/\lVert\nabla_{\bm{\mathrm{\mu}}}\tau\rVert,$ (4) $$

where $\bm{\mathrm{l}}\in\mathbb{R}^{3}$ is the radiance received by the scene at the point $\bm{\mathrm{\mu}}$ from the light sources, $\bm{\mathrm{n}}$ is “surface normal”, and $\odot$ is the element-wise product.

## 4 Method

### 4.1 Generation of multiple aligned 3D models

We make two modifications to the SDS method to generate aligned 3D objects from a set of $N$ text prompts.
Firstly, we use a single NeRF-like neural network to represent all the objects and embed them into a common space of 3D reflectance fields.
We add an input parameter $\bm{\mathrm{u}}\in\mathbb{R}^{N}$ to the network that parameterizes the transitions between the objects.

Secondly, we train the network to smoothly map transitions between the given text prompts into the transitions between the 3D objects.
To achieve this, we restrict the transition parameter to the $(N-1)$-dimensional probability simplex $\left\{\bm{\mathrm{u}}\in\mathbb{R}^{N}:u_{1}+\dots+u_{N}=1,u_{i}\geq 0\right\}$, and assign the vertices of this simplex $\left\{u_{i}=1\right\}$ to the given text prompts.
We train the network using SDS so that the reflectance fields obtained at the vertices of the simplex represent the individual 3D objects corresponding to the respective text prompts.

Specifically, we train the network with the SDS loss [Equation 3](#S3.E3) where the rendered image $\bm{\mathrm{x}}$ and the text embedding $\bm{\mathrm{y}}$ now depend on the transition parameter $\bm{\mathrm{u}}$ that we randomly sample at each iteration.
We render the image following [Equations 1](#S3.E1) and [4](#S3.E4), where the density and albedo now additionally depend on the transition parameter $(\tau,\bm{\mathrm{\rho}})=F\left(\bm{\mathrm{\mu}},\bm{\mathrm{u}};\theta\right)$.
We interpolate the text embedding between the embeddings of the individual prompts $\left\{\bm{\mathrm{y}}_{i}\right\}$ weighted with the components of the transition parameter $\bm{\mathrm{y}}(\bm{\mathrm{u}})=u_{1}\bm{\mathrm{y}}_{1}+\dots+u_{N}\bm{\mathrm{y}}_{N}$.

The key feature of our approach is that, instead of sampling the transition parameter $\bm{\mathrm{u}}$ from the whole simplex, we sample it from the shortest one-dimensional trajectories between the points corresponding to individual 3D objects, *i.e*., from the edges of the simplex.
Our sampling strategy encourages the network to learn a mapping from these edges into meaningful trajectories between the individual 3D objects in the space of reflectance fields, as we illustrate in [Figure 3](#S4.F3).
This improves the alignment between the 3D objects.
Our method is inspired by several works [, , ] that show that a similar technique improves the interpretability of latent generative models.

Figure: Figure 3: Trajectories in the space of 3D objects obtained with our method for pairs of text prompts. Our method generates aligned 3D objects together with smooth transitions between them.
Refer to caption: /html/2406.15020/assets/x3.png

The described approach on its own encourages the network to align the generated 3D objects.
This is caused by the shallowness of the NeRF-like network, which acts as a regularization of the smoothness of the transition between the objects.
We found that enforcing the smoothness of the transition even more improves the alignment further.
There are multiple ways to enforce this smoothness.
For example, one can penalize the norm of the gradient of the neural field w.r.t. the transition parameter.
We choose a simpler and more efficient approach and apply Spectral Normalization [] to all layers of the NeRF network.

### 4.2 Hybridization: combining the aligned 3D models

As the result of the training process described above, we obtain a neural network that does not only represent multiple aligned 3D objects but also allows us to smoothly interpolate the reflectance field between these objects at each point of the 3D space independently.
This provides us with a natural and simple way to seamlessly combine these objects with each other, fusing specific parts of the individual generated objects into a new object, such as a gopher with the head of a kangaroo shown in [Figure 4](#S4.F4).
To achieve this, we partition the 3D space into regions related to the specific 3D objects and smoothly interpolate the reflectance field between the respective objects at the boundaries of these regions.
Specifically, we define this partitioning with a smooth spatial distribution of the transition parameter $\bm{\mathrm{u}}\left(\bm{\mathrm{\mu}}\right)$ (see the second column in [Figure 4](#S4.F4) for an example).
We render the new combined model, or extract the surface, following [Equations 1](#S3.E1) and [4](#S3.E4) with the reflectance field now depending on the spatially varying transition parameter $(\tau,\bm{\mathrm{\rho}})=F\left(\bm{\mathrm{\mu}},\bm{\mathrm{u}}\left(\bm{\mathrm{\mu}}\right);\theta\right)$.

Figure: Figure 4: Our approach allows us to blend different objects seamlessly. A proper alignment of multiple 3D models provides the ability to replace parts of one object with similar components of the other objects. We manually select spatial anchor points (left) and assign them to a particular model. The NeRF input vector $\bm{\mathrm{u}}$ is linearly interpolated between anchors at every spatial location, resulting in a smooth distribution over 3D space (second column). The resulting objects are shown on the right.
Refer to caption: /html/2406.15020/assets/x5.png

### 4.3 Pose-preserving transformation of 3D models

The process of 3D generation through SDS can be conditioned on a predefined 3D model by fitting the neural 3D representation to this model before starting the SDS optimization.
Together with our approach to the generation of aligned 3D objects, this allows us to transform one model into another while preserving the pose and proportions.
In particular, this makes it possible to generate a 3D object in a predefined desired pose via the transformation of a dummy model in this pose.

To transform one 3D object into another defined by a text prompt, we modify our generation approach as follows.
First, we set up the neural network as described in [Section 4.1](#S4.SS1) for two text prompts $N=2$.
In this case, the transition parameter $\bm{\mathrm{u}}$ is constrained to a one-dimensional segment.
Then, we initialize the network with the input 3D model for all values of the transition parameter uniformly.
This initialization can be done in different ways depending on the representation of the input model.
In our experiments, we obtain the renderings of the input model for a random set of viewpoints and fit the network to these renderings photometrically, by minimizing [Equation 2](#S3.E2).
After that, we pick a text prompt describing the input model (in our experiments we pick it manually for simplicity), and assign the endpoints of the transition segment $u_{1}=1$ and $u_{2}=1$ to this prompt and to the target prompt, respectively.
Finally, we train the network with SDS as described in [Section 4.1](#S4.SS1), additionally keeping the constraint on the photometric consistency with the input model ([Equation 2](#S3.E2)) at the respective endpoint of the transition segment $u_{1}=1$.

## 5 Experiments

### 5.1 Generation of multiple aligned 3D models

We evaluate our approach for the generation of aligned 3D models quantitatively on 15 pairs of prompts describing various kinds of objects including animals, humanoids, furniture, vehicles, and buildings.
We also show qualitative results for 2 triplets and 2 quintets in [Figures 1](#S0.F1) and [2](#S1.F2).

As no existing method targets the joint generation of aligned 3D models, we employ a recent 3D editing method MVEdit [] as a baseline.
MVEdit transforms an input triangular 3D mesh to make it consistent with the input text prompt and also provides a pipeline for text-conditioned 3D generation.
To apply MVEdit for the generation of two aligned models, we first use its text-to-3D pipeline to generate one model from one of the prompts in the pair, and then transform this model into another, described with the second prompt.
After that, we repeat the above process into the inverse direction, from the second prompt to the first one.

#### Metrics.

We evaluate the methods in terms of geometric alignment between the two generated 3D models and the semantic consistency of the models with the respective prompts.
We measure the geometric alignment using Chamfer distance between the surfaces of the generated 3D models.
For our method, we extract the surface from the density field using Marching cubes [].
We measure the semantic consistency between the 3D models and the prompts using CLIP similarity.
Specifically, for each prompt in the pair, we calculate the cosine similarity between the CLIP [] embeddings of the renders of the 3D model and the embeddings of the prompt, averaged across 120 views sampled uniformly around the object.
Then, we calculate the average value for the pair.

With MVEdit, we obtain two different results for each pair of the prompts, using each of the prompts for generation of the initial mesh, and calculate the average value for these two results.

#### Discussion.

We show the quantitative comparison in [Table 1](#S5.T1) and the qualitative comparison in [Figure 5](#S5.F5).
Our method generates geometrically aligned 3D models that are semantically aligned with the text prompts consistently better than the models generated with the baseline.
MVEdit, using just one text prompt for the generation of the initial model, often produces the model that is hard to transform to be consistent with the second prompt.
This is confirmed by a significant drop of the CLIP-similarity w.r.t. the second prompt compared to the similarity w.r.t. the initial prompt (see columns “CLIP, source / target prompt” in [Table 1](#S5.T1)).
This is especially evident in cases where the objects described with the prompts usually have different structures, *e.g*., for the car and carriage, or the animal and lego animal.
Overall, our results clearly demonstrate the advantage of our approach for the joint generation of aligned 3D models.

Table: Table 1: Quantitative comparison of our method with MVEdit for aligned generation and for pose-preserving transformation.

Figure: Figure 5: We compare pairs of aligned 3D objects generated with our method A3D with the results of baseline MVEdit. For MVEdit, we show the initial 3D models generated for each of the prompts in the pair and the respective transformed version.
Refer to caption: /html/2406.15020/assets/x9.png

### 5.2 Hybridization: combining the aligned 3D models

We show examples of 3D models combined from different parts of the objects generated with our approach in [Figure 4](#S4.F4), and in the middle of [Figure 1](#S0.F1).
To obtain these models, we start with the generation of the aligned objects from a set of text prompts.
Then, we manually place several anchor points in the common 3D space of the objects and assign each point to a particular object.
Finally, to define the spatial distribution of the transition parameter $\bm{\mathrm{u}}$, described in [Section 4.2](#S4.SS2), we linearly interpolate this parameter at every spatial location between the values corresponding to the objects associated with the two closest anchors.

These examples demonstrate that the aligned 3D objects generated with our approach can be seamlessly blended in different configurations.
The coherent appearance of the hybridized models demonstrates the high quality of the alignment between the generated objects.
Remarkably, our approach allows us to easily transition between the objects in places with different geometries, *e.g*., the necks of the gopher and kangaroo, which have different diameters, or waists of the seahorse and mermaid, which have different other parts of the object nearby.
This is in contrast to methods like MVEdit that produce triangular meshes, which have to be locally aligned first to be stitched together.

### 5.3 Pose-preserving transformation of 3D models

We demonstrate the application of our approach for pose-preserving transformation in two scenarios.
For the first one, we collect triangular meshes for 8 objects from the web (under an open license) and transform them into other objects described with text.
For the second scenario, we use the SMPL parametric human body model [] to obtain 12 textureless meshes of people in different poses, and transform them, thus performing pose-conditioned 3D generation.
In both scenarios we compare with MVEdit, using the same metrics for quantitative evaluation as described above.

We show the quantitative comparison in [Table 1](#S5.T1) and the qualitative comparison in [Figure 6](#S5.F6), and show more results in [Figure 1](#S0.F1) and in the supplementary material.
Our method consistently outperforms MVEdit w.r.t. consistency with the text prompt while keeping the transformed 3D model aligned with the initial one.
Unlike MVEdit, which is restricted to superficial deformations of the surface, our method is able to add or remove significant parts of the object, *e.g*., adding the axe for the dwarf or the throne for the princess.

Figure: Figure 6: We compare the mesh editing capabilities of previous work MVEdit with our method A3D. For each example, we show the input model, MVEdit results, and our results.
Refer to caption: /html/2406.15020/assets/x10.png

## 6 Ablation

In [Figure 7](#S6.F7) we show the results of the qualitative ablation study of the core idea of our approach.
We compare pairs of 3D objects generated for pairs of text prompts obtained in three different ways.
As a baseline, we train two NeRF-like networks using SDS for each prompt independently.
In our approach, we train the network to smoothly map the transition between the objects from the space of text prompts to the space of 3D objects.
As an intermediate step from the baseline towards our approach, we train a single network to embed both objects but not the transition between them.
Specifically, during training, we sample the transition parameter $\bm{\mathrm{u}}$ just from the endpoints of the transition segment, without interpolation between them.

Our experiments show that the baseline produces unaligned objects with significantly different features and structures.
The intermediate variant slightly improves the alignment but still produces the objects with variations in pose and structure.
Our complete approach yields significantly more consistent results with the generated objects exhibiting good alignment w.r.t. geometry and semantics.

Figure: Figure 7: Ablation Study. Training NeRFs independently (left) does not produce aligned objects. Training our method without interpolation between the objects (middle) improves alignment to some degree. Our complete approach with interpolation (right) produces well-aligned objects.
Refer to caption: /html/2406.15020/assets/x11.png

## 7 Conclusions

We present A3D, the first framework designed to generate a collection of objects aligned w.r.t. semantics and geometry.
This is achieved by encouraging the transitions between the objects, jointly embedded into a shared latent space, to be smooth and meaningful.
We show that, when applied to the tasks of aligned 3D generation and mesh editing, our approach produces closely aligned meshes while surpassing the competitor approaches in terms of quality and semantic alignment with the text description.
We demonstrate that with our method it is possible to solve the novel task of object hybridization, seamlessly combining multiple parts of the different objects.
Our approach is limited to generating static aligned objects, and can not be applied to pose-changing tasks.
Also, it sometimes struggles when aligning objects with significantly different proportions.
In future work, we plan to extend our framework to mesh deformation and 4D video generation by experimenting with different regularization techniques for the transitions between objects.

#### Acknowledgements.

The authors acknowledge the use of Skoltech supercomputer Zhores [] for obtaining the results presented in this paper.
The authors from Skoltech were supported by the Analytical center under the RF Government (subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021).

## Appendix A Additional results

Figure: Figure 8: Our approach allows us to blend different objects seamlessly. A proper alignment of multiple 3D models provides the ability to replace parts of one object with similar components of the other objects. We manually select spatial anchor points (left) and assign them to a particular model. The NeRF input vector $\bm{\mathrm{u}}$ is linearly interpolated between anchors at every spatial location, resulting in a smooth distribution over 3D space (second column). The resulting objects are shown on the right.
Refer to caption: /html/2406.15020/assets/x13.png

Figure: Figure 9: Ablation Study. Training NeRFs independently (left) does not produce aligned objects. Training our method without interpolation between the objects (middle) improves alignment to some degree. Our complete approach with interpolation (right) produces well-aligned objects
Refer to caption: /html/2406.15020/assets/x19.png

Figure: Figure 10: Trajectories in the space of 3D objects obtained with our method for pairs of text prompts. Our method generates aligned 3D objects together with smooth transitions between them.
Refer to caption: /html/2406.15020/assets/x21.png

Table: Table 2: Quantitative comparison of our method with MVEdit for pose-preserving transformation with details for every scene.

Table: Table 3: Quantitative comparison of our method with MVEdit for aligned generation with details for every scene.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: 1. The motivation of optimization in a shared latent space for smooth transitions is reasonable.

2. This writing style is easy to follow.
summary: This paper tries to tackle the problem of text-driven 3D generation from a geometry alignment perspective. To achieve the alignment of the corresponding parts of the generated objects, this paper tries to optimize the continuous transitions in a shared latent space. Visualization results demonstrate the effectiveness.
weaknesses: 1. Fairness issues. This work is built on MVDream, which highly relies on the pre-trained knowledge from SD2.1. This raises the question: compared with other approaches, does the effectiveness of A3D come from the method itself or the pre-trained knowledge of the SD model? Relevant experiments are needed to demonstrate it. Besides, the authors are encouraged to compare different SD versions, like SD1.5 and SDXL.
2. In Tab.2, MVEdit outperforms A3D on CLIP score and DIFT distance. The authors claim that A3D can add or remove significant parts. However, this indirectly indicates that pre-trained SD models may cause unfair knowledge leakage.
3. What is the training cost compared with baseline methods, like MVEdit, LucidDreamer, and GaussianEditor?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness section above.
soundness: 3
strengths: - This paper is well-written and easy to follow.
- The motivation the paper is sound, tackling the task of 3D shape editing through addressing hybridization and alignment of difference 3D scenes corresponding to the given shape, which I believe is an innovative and novel solution to the task at hand.
- The technical solution this paper offers to the problem is simple and effective: training a single NeRF model to handle multiple scene from different prompts during the optimization phase, and then allowing for arbitrary translation between the different scenes with interpolation of latent vectors is a elegant approach to achieve 3D alignment and controllable 3D editing of the scene.
summary: This paper tackles an interesting problem of hybridization and editing regarding geometry-aligned text-to-3D generation, in which they leverage 3D shape and transitional embedding between different prompts to ensure continuous transition within the same 3D representation. By embedding these objects into a common latent space, they achieve both seamless hybridization of different shapes as well as high-quality 3D alignment between the generated scene and template 3D shape.
weaknesses: - This paper uses NeRF for seamless representation of difference scenes, and I understand design choice was to enable seamless transition between different optimized scenes within the representation space of NeRF. However, more explicit 3D representations such as Instant-NGP or 3DGS offer their own advantages, such as speed and controllability, and I believe are a viable candidates for comparison. Can this method be applied to such different forms to 3D representation? If so, how do their performance different from the current model that uses NeRF?

- How much 'deformation' from the given template shape does this method allow for? Figure 4 shows the generated shapes display variations in geometry from the given template - is there any mechanism within the paper that specifically allows for this "loose" alignment (SD operating in the latent space, perhaps)? How does 3D geometry look in the "middle states" of transitioning the entire network from one object to another? 

I am very much willing to raise the score if the mentioned questions are addressed during the discussion phase.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the Weaknesses.
soundness: 3
strengths: 1. The overall idea is interesting. The authors observe that the previous 3D editing method fails to generate structurally-aligned high-quality 3D objects (e.g., either lack of structure alignment or detailed texture). A3D provides an effective way to generate aligned 3D objects by introducing additional latent code conditions.
2. The results in Figure 3 demonstrate that A3D indeed learns a smooth transition between different latent codes and can be decoupled with 3D positions by setting anchor points.
3. The authors provide sufficient experiments and comparisons with previous works and A3D has better performance in generating aligned objects.
4. The presentation is clear and easy to follow. Some visualizations are useful for understanding the core idea of learning smooth transitions.
summary: This paper proposes A3D, a new method to generate structurally-aligned 3D objects from a set of text prompts. A3D can first generate multiple aligned 3D objects, then make hybrids combined of different parts from the generated objects. Another advantage is that A3D can achieve smooth structure-preserving transformation given a coarse mesh.
weaknesses: 1. I noticed that some generated textures from A3D are still poor. Is this caused by the limitation of SDS loss or the framework of A3D?
2. I am wondering whether A3D is sensitive to text prompts. It would be better if the author could evaluate this point. is A3D robust to various text prompts?
3. In section 4.1, A3D uses some regularization strategies to supervise the network, including limiting the depth of the neural network and enforcing the smoothness of the rendered results. which will cause performance degeneration compare with raw SDS-based optimization.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Can it be generalized to other diffusion models like Stable Diffusion-based SDS?

2. Is it capable of performing on Gaussian splatting-based paradigms?

3. Can this method generate diverse sets of 3D results given one single set of text prompts?
soundness: 3
strengths: 1. Proposed A3D can jointly generate a collection of 3D objects aligned in structure from a set of text prompts. This method can be applied to various scenarios, such as 3D editing, object mixing, and structure-preserving transformation.
2. Compared to existing methods that generate 3D objects independently, A3D can generate a collection of 3D objects that are structurally consistent, text-aligned, and of high visual quality.
3. A3D can be used in conjunction with different text-to-3D generation frameworks to improve the generation quality while maintaining the structure.
4. The results are interesting.
summary: The paper proposes a method that enables the joint generation of a collection of 3D objects with semantically corresponding parts aligned across them.  A3D addresses 3D editing and consistent generation by embedding the 3D objects and the transitions between them into a shared latent space and enforcing the smoothness and plausibility of these transitions. This allows for several practical scenarios that benefit from alignment between the generated objects, including 3D editing, object hybridization, and structure-preserving transformations.
weaknesses: 1. Lacking a diagram that can illustrate the process of the transition and latent coding process. Only the text description on this part might be not clear enough.

2. The generated results are confined to the formed shape in the latent space, which might result in unnatural and unpleasant results. e.g., in Fig. 6, the crab and ant share the same latent shape but the generated assets are not reasonable enough for both(Too many legs for the ant and no claws for the crab).

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. I am curious if the the proposed method can be applied to image-to-3D generation methods. Since one image can be converted into a text embedding, the proposed method should support the image-to-3D setting.
2. What is the training time overhead for the proposed method?  Can you compare the training time with other methods and discuss how the training time scales with the number of objects being aligned?
3. For the ablation study, Table 3 only compares the DIFT distance metric. Can you also compare other metrics like Table 1 and Table 2?
4. I wonder if the proposed method can edit some details. For example, "a happy running man", "a sad running man", "a happy running minotaur" and "a sad running minotaur".
soundness: 3
strengths: 1. This paper introduces an interesting problem, how to make one text-to-3D generation model flexibly produce different objects while maintaining structural consistency?
2. The proposed method proposes to embed these objects into a common latent space, allowing them to share similar structures.  
3. Experiments show the potential of the proposed method on different tasks, including generation of multiple objects, hybridization and structure-preserving transformation.
summary: This paper focuses on generating a set of objects with similar structures using one text-to-3D generation model. To this aim, it introduces a  latent code to embed these desired objects into a common space. Besides training the text-to-3D generation model with their individual texts, this work enforces two kinds of regularization on the transitions between the objects. Experiments on multiple aligned 3D objects generation, combining parts of different objects into a ''hybrid'' and preserving transformations of a 3D object verify that the proposed method can generate diverse objects while maintaining structural consistency.
weaknesses: 1. The paper is not well-written. It does not provide an overview to introduce the method architecture. It is better to provide a high-level flowchart of the proposed method, and then introduce the components of the proposed method by referring them to the presented flowchart. In the Method section, many details are given in the appendix.
2. The experiments lack the ablation studies for the two kinds of regularization on the transitions between objects. Can you remove the two regularizations, respectively and conduct corresponding ablation experiments to verify their effectiveness?
3. Can the proposed method be applied to image-to-3D generation models? Can you discuss the potential challenges or modifications needed to adapt their method to image-to-3D models, and how this might impact the structural consistency of the generated objects?

Meta-review: In this paper, all reviewers vote for acceptance. After checking the paper, the AC acknowledges the technical novelty proposed in this paper and agrees on the acceptance.

### Pros
1. This paper is well-written and easy to follow.
2. The motivation to tackle the task of 3D shape editing through addressing hybridization and alignment of different 3D scenes corresponding to the given shape.
3. Sufficient experiments and comparisons with previous works to support the effectiveness of generating aligned objects.

Please polish the manuscript, given the comments the reviewers point out. For example,
1. Reporting the computation cost or training/inference time.
2. More applications of the proposed method to other 3D representations.
3. More ablation studies.

Decision: Accept (Poster)