A3D: Does Diffusion Dream about 3D Alignment?
Authors: Savva Victorovich Ignatyev
Nina Konovalova
Daniil Selikhanovych
Oleg Voynov
Nikolay Patakin
Ilya Olkov
Dmitry Senushkin
Alexey Artemov
Anton Konushin
Alexander Filippov
Peter Wonka
Evgeny Burnaev
Venue: ICLR 2025 Poster (2025)
Keywords: Aligned 3D generation
3D generation and editing
Text-to-image diffusion
Score distillation sampling
Implicit neural surfaces
Radiance fields
Primary area: generative models
TL;DR: We generate a set of structurally aligned 3D models from text prompts by embedding transition trajectories between these models into a multi-dimensional NeRF and enforcing smoothness and realism of transitions via Score Distillation Sampling.

Abstract: We tackle the problem of text-driven 3D generation from a geometry alignment perspective. Given a set of text prompts, we aim to generate a collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain a set of objects aligned with each other. In order to achieve the alignment of the corresponding parts of the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of our method.

Published as a conference paper at ICLR 2025
A3D: DOES DIFFUSION DREAM ABOUT 3D ALIGN- MENT?
Savva Ignatyev"!
Nina Konovalova
Daniil Selikhanovych'
Oleg Voynov!.2
Nikolay Patakin²
Ilya Olkov! Dmitry Semushkin²
Alexey Artemov3
Anton Konushin
Alexander Filippov Peter Wonka" Evgeny Burnaey1.2
Skoltech, Russia
2 PAIRI, Russia *Meiida AI, Israel
AI Foundation and Algorithm Lab, Russia 'KAUST, Saudi Arabia
*Savva Ignatyev and Nina Konovalova contributed equally
Corresponding author: Savva Ignatyev (e-mail: savva.ignatyev @oootech..
of milhipp
20 steea
oma serul
oviannn Noyt:
hests strplt
paRk caletrl
mare:
Stacttae Hyondrraioo
lylel makk
lo coll
I0D Pamker
SOURE mwch
jol wit lightdbrr
Figure Our method A3D enables conditioning text-to-3D generation process on a set of text prompts to jointly generate a set of 3D objects with shared structure (top). This enables user to make "hybrids" combined different parts from multiple aligned objects (middle), or perform text-driven structure-preservinng preserving transformation of input 3D model (bottom).
ABSTRACT
We tackle the problem of text-driven 3D generation from geometry alignment perspective. Given a set text prompts, we aim to generate collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain set of objects aligned with each other. In order to achieve the alignment of the corresponding parts the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of method. voyleg.github.oo//

---

Published as a conference paper at ICLR 2025
Exising text-driven 3D givertionn
Existing text-driven 3D edring
Our aligned 3D generation
Viseal quality
quillt: ppear geoncry deail
Visul quality Pronge algrment
Peorptt aligmmert
Strcctrd commisionry:
"pooristhas desil
Smattul consiseocy
plazabiltt
rriimnsting (beoahl, qraltrpahl, ortnnded))
Prongt aligmmntt
Stretira unveiarrey
Inoendsrnt dootral properionn
Dutdkigg FED
TaElig noxile
Figure 2: Collections of objects generated with existing text-to-3D methods lack structural consisteney (left, (Shi et al. 2024)). Shapes obtained with existing text-driven 3D editing methods lack text-to- asset alignment and visual quality (middle, (Chen al. (2024a)) ). In contrast, our method enables the generation of structurally coherent, text-aligned assets with high visual quality (right).
INTRODUCTION
Creating high-quality 3D assets is a time- and labor-intensive process, even experienced 3D artists commonly break down into manageable steps. Prior to shaping an asset, an artist might conceptualize its design structure capture geometric proportions and spatial relationships of its semantically meaningful parts. Then, a series of detailed 3D design instances (c.g.. high niihhreesolution geometry and textures) consistent with established structure can be produced. Recent 3D generation research (Poole et al. 2023 Qiu et al. 2024 Shi et al. 2024) promises to significantly reduce the effort related to manual production of such high- nigh-resolution textured shapes, replacing it by automated Al-based step controlled with natural language. A text-to-3D generation pipeline could potentially be utilized to produce a collection of structurally aligned 3D objects, consisting of common set of semantic parts and sharing their structure, e.g., the pose or the arrangement of semantic parts.
However, existing 3D generation approaches synthesize objects independently and fail to maintain structural aligmment across them (Figure 2 left). One may attempt to enforce the alignment in a series of 3D objects by generating an initial one with a text-to-3D pipeline and obtaining the others with text-driven 3D editing methods (Haque et al., 2023 Chen et al., 2024a). Unfortunately, the latter struggle with the visual quality and sometimes fail to perform the necessary edits appropriately, resulting in low degree of the alignment with the text prompt (Figure 2 middle).
To address the limitations of existing approaches, we propose A3D, a method for jointly generating collections of structurally aligned objects from collection of respective text prompts. The idea of our method to embed a set of 3D objects and transitions between them into a shared latent space and enforce smoothness and plausibility of these transitions. We take inspiration from the transition trajectory regularization proposed for 2D GANs (Karras et al.) 2020, Sec. 3.2). We represent each set of 3D objects and the transitions between them with single Neural Radiance Field (NeRF) (Mildenhall et al. (020)) and train it with a text-to-image denoising diffusion model via Score Distillation Sampling (SDS) (Poole al.) 2023) to simultaneously correspond the set of text prompts and enforce the plausibility of the transitions between the objects.
Our method is naturally suited for several different scenarios that require control over the structure of the generated objects. (1) Generation of multiple structurally aligned 3D objects (Figure | top) enables artists choose an appropriate 3D asset among variety of generations, replace 3D objects within existing scenes, or transfer animations across distinct objects. (2) Combining parts of different objects into a "hybrid" (Figure [ middle) allows adjusting constituent elements without affecting the overall structure of the asset. (3) Structure- preserving transformations of a 3D object (Figure | bottom), user design simplified 3D shape with a particular pose and let the automatic generation process fill in complex geometric details and texture while preserving the structure.
Sets of objects generated using our method exhibit high degree of structural alignment and high visual quality scores, outperforming those obtained with state-of-the-art alternatives. Our method is easily adopted for the prructure-prseenvve text-driven 3D editing methods. Further, it is effective in combination with different text-to-3D generation frameworks. Overall, our work advances the state of the art in text-driven 3D generation and opens up new possibilities for applications requiring the generation of structurally aligned objects.
transformation, performing on par with specialized
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 TEXT-DRIVEN 3D ASSET GENERATION
Collecting large, high-quality, diverse 31D datasets poses significant challenges, 3D generation ap- proaches predominantly leverage 2D priors for training. DreamFusion (Poole et al., (2023) introduced Score Distillation Sampling (SDS) that enables training Neural Radiance Fields (NeRFs) Miilennhal] 2020) with the guidance of pre-trained 2D diffusion models. Subsequent research has refined this methodology to improve the quality and speed of 3D generation. Magic3D (Lin et al. 2023
ct al.
uses a coarse-to-fine optimization strategy to increase the speed and resolution. Fantasia3D CChen et al. 2023a) disentangles the geometry and texture training. Several works enhance realism, detail, and optimization speed by utilizing adversarial training (Chen et al.] (024d), 3D-view conditioned
diffusion models (Liu et al., 2023; Shi et al.) 2023a| 2024| Liu et al. 2024, Ye et al. 2024, Seo ct al. (2024), and Gaussian splatting- -ased models (Tang al. 2024b Yi et al. 2024 All these works focus on independent optimization for distinct prompts, resulting in generation of collections of objects that lack structural alignment, we show in Figure 2 and in ablation study. This
misalignment issue persists even amortized frameworks, where a single generative model is trained
to handle multiple prompts (Tang et al. 2024a Jun & Nichol, 2023: Hong al., 2024 Siddiqui
et al. 20244) Ma et al. 2024). Due to the mode-seeking nature of SDS (Poole et al. 2023). these
frameworks often produce misaligned objects with the structure sensitive to subtle variations in
the prompt, increasing inconsistency across generated outputs. Unlike the described methods that
optimize 3D objects independently or amortized models trained on large-scale datasets, our method
optimizes small set of objects jointly, allowing us to achieve structural consistency between objects.
2.2 TEXT-DRIVEN 3D ASSET EDITING
One straightforward way to produce a collection of aligned 3D objects is to generate an initial object using a text-to-3D pipeline and subsequently modify this object via text-driven 3D editing. Several methods have been proposed to manipulate Nese-based scene representations using text as guid- ance (Haque et al. 2023 Park et al.
2024 Bao et al. 2023 Zhuang et al. 2023). DreamBooth3D (Raj
jet al. 2023) and Magic3D (Lin et al.
2023 provide the capability to edit personalized objects while
leveraging the underlying 3D structure. FrcalDreamer (Li et al.) 2024), Progressive3D (Cheng
eet al., 2024), and Vox-E (Sella et al.] 2023) confine the effect of modifications to specific parts of the object, thus enhancing control of the editing process. Fantasia3D (Chen al., 2023a) and DreamMesh (Yang et al. 2024) focus on global transformations of one object into another. iteratively optimizing a 3D model to align with a text prompt via SDS. Iterative optimization with SDS does not guarantee preservation of the structure of the transformed object, so several techniques were proposed improve it. Coin3D (Dong et al., 2024) refines geometric primitives into high-quality assets by imposing deformation constraints through input masks. GrussianDreamer (Yi al. 2024) and LucidDreamer (Liang et al., 2024) show text-driven editing capabilities for Gaussian splats, which they initialize using a separate pipeline and fine-tune with the help of a diffusion model. [Haque eet al. (023)) and Palandra et al. (2024) use the SDS loss in combination with a pre-trained 2D image editing network InstructPix2Pix (Brooks et al., (023). MVEdit (Chen et al., (024a) goes one step further by avoiding SDS and proposes a special mechanism that coordinates 2D edits from different viewpoints. Although some of these methods allow obtaining sets of the aligned objects sequentially, the editing process is constrained by the configuration of the initially generated object. This limits the visual quality of the generated sets of objects, as we show in Figure 2 and in our experiments. In contrast, our method optimizes the whole transition trajectory between the objects, and produces both structurally consistent and high-quality results. Additionally, our method easily adapted for the task of structure- struttue-preserviin 3D editing, performing on par with specialized methods.
2.3 LATENT SPACE REGULARIZATION
To achieve the structural alignment between the generated objects, we embed these objects into a common latent space together with the transition trajectories between them. We draw inspiration from works on generative modeling of 2D images that show that alignment, disentanglement, and quality of the generated samples can be improved with regularization of trajectories between them. For example, Berrtello** et al. (2019) and [Sainburg et al. (2018) directly optimize the quality of the interpolated samples with adversarial training, and StyleGAN (Karras et al. 2020) explicitly
3

---

Published as a conference paper at ICLR 2025
regularizes the smoothness of the trajectories by calculating the perceptual path distance in the VGG feature space. Similarly, we employ diffusion model as a critic that encourages plausibility of the samples on the trajectories via SDS, leading to smooth transitions and aligned objects.
3 PRELIMINARIES
3.1 NEURAL RADIANCE FIELDS
Neural radiance field (NeRF) (Mildenhall al. 2020) a differentiable volume rendering approach that represents the scene as a radiance function parameterized with neural network. This network maps a 3D point l E TR3 and a view direction d E g into a volumetric density T E R+ and view-dependent emitted radiance E R at that spatial location. To render an image, NeRF queries 5D coordinates (μ, d) along camera rays and gathers the output colors and densities using volumetric rendering. The ray color C is calculated numerically through quadrature approximation:
C Ea,Tica
T. --
i ass
0; - exp(-Tillı H,+1ll),
(1)
where C₁ and T; are the radiance and density queried at the i'th position along the ray, and og and T] are the transmittance and accumulated transmittance.
Originally. NeRF is iteratively trained from set of posed images. At each iteration, a batch of camera rays is randomly sampled from the of all observed pixels and the photometric deviation between the colors C2 observed along the k'th ray and C2 rendered via Equation (1 is minimized:
Le = > C Cx112-
3.2 SCORE DISTILLATION SAMPLING
Score Distillation Sampling (SDS) (Poole et al. 2023 was proposed for fitting NeRF to text description of the 3D scene, without any input images, using pre-trained text-to-image diffusion model. The NeRF is iteratively guided towards consistency with the text prompt by using the diffusion model as critic for the rendered images. At each iteration, the image x is rendered for randomly sampled camera position. A random Gaussian noise c ./ I) is added to the image and the output of the denoising diffusion model & is obtained via é = E(y, t, a,x + o,c), where is the embedding of the text prompt. L ~ (4 1) is the diffusion timestep, and a and ar are weighting factors. The weights 0 of the NeRF network are then updated using the gradient of the SDS loss term:
VoLSDS = Er,, (w()) - E)0ox),
where (t) is another weighting factor.
Poole et al. (2023) use a NeRF-like network F that maps the 3D point μ into volumetric density and the diffuse RGB reflectance p R (albedo), i.e. (r. p) F (#:; 9). They obtain the emitted
radiance e for Equation (1) via shading with a random lighting:
c - p @ 1( n), n = -= / W
(4)
where E 3 is the radiance received by the scene at the point μ from the light sources, n is "surface normal", and is the element-wise product.
4 METHOD
Learning implicit 3D representations of the objects separately often produces non-aligned results (Figure 2). Our method uses a single NeRF-like network to represent set of aligned objects together with the transitions between them. For this, we introduce new input parameter u E R that represents a point latent space. We optimize the network, guiding it with a text-to-image diffusion model via SDS loss. The diffusion model is conditioned on weighted linear combination of individual text embeddings, where the weight coefficients correspond to the elements of u. At each iteration, we sample u randomly from the edges of the probability simplex. At the vertices of the simplex, the SDS loss guides the renders from the network towards consistency with the individual text prompts. At the edges of the simplex, the loss guides the renders towards image plausibility. This leads to plausible transitions between the objects in the respective pairs and, as a result, to the alignment between the objects. The overview of our method is shown in Figure 3
4

---

Published as a conference paper at ICLR 2025
"minotaur" "werewolf"
encode(prompt.) WIV1
y(u)
u
prompty
Diffusion U-Net
render
SDS loss
(1.0,...
(0.6, 0.4,...)
(0,1,...
NeRF F (1,, u)
Latent simplex
Figure 3: Overview of our method. The NeRF model, conditioned on the latent code u sampled from the edges of the latent simplex, produces render. The render is passed to the diffusion model, conditioned on the linear combination of the embeddings of the text prompts. Finally, the SDS loss is backpropagated to the NeRF model.
4.1 GENERATION OF MULTIPLE ALIGNED 3D OBJECTS
We adopt the SDS method for the joint generation of aligned 3D objects from set of N text prompts. To do this, we embed all the objects into common space of 3D reflectance fields represented with single neural field. We train our network as small generative model with the latent space built around the given set of text prompts. Specifically, we define the latent code u on the (N 1)-dimensional probability simplex [u RN #1 +
uN = L, u, 2 0} and assign the vertices of this simplex {u, = 1} the given textual prompts. We add this latent code an input parameter the neural field. We train it to represent the individual 3D objects at the respective vertices of the simplex and map the linear interpolations (edges) between the latent codes the vertices transitions between the objects. This allows us to regularize these transitions and achieve structural alignment across the objects.
Specifically, we iteratively train the network with the SDS loss (Equation 3 At each iteration, we sample the latent code u from the vertices and edges of the simplex. We render the image following Equations ① and (4), where the density and albedo now additionally depend on the latent code (T, = (μt, u; 0). At the vertices of the simplex, we condition the diffusion model on the text embeddings of the individual prompts {y,), ). At the edges, which represent transitions between the objects, we apply two kinds of regularization inspired by works on training GANs with mixed latent codes (Berthelot* et al. 2019: Karras al. 2020).
First, we encourage the network to produce plausible 3D objects for latent codes sampled at the transition trajectories. We use the text-to-image diffussin model critic to evaluate and improve the plausibility through SDS. For this, we obtain the text embeddings y for the diffusion model as the sum of the embeddings of the individual prompts weighted with the components of the latent code (u) W1Y1 + + uNy, N- For the edges of the latent simplex it corresponds to linearly interpolating between the pair of embeddings of the individual text prompts. In ablation study, show that similar effect to a lesser extent can be achieved by conditioning the diffusion model on some general prompt independent of the objects being generated.
Second, we encourage the transitions between the objects to be smooth. We avoid doing this directly give our model more flexibility and instead regularize the smoothness of the transitions implicitly. Specifically, we limit the depth of our neural field network, which limits the Lipschitz norm of the function parameterized by this network (Miyato et al., 2018), and enforce smoothness of rendered normal maps with corresponding loss (see Equation (5).
These regularization strategies encourage the network learn a mapping from the edges of the simplex to meaningful transitions between the individual 3D objects. In our ablation study, we show that this crucial for obtaining structurally aligned objects.
4.2 HYBRIDIZATION COMBINING THE ALIGNED 3D OBJECTS
The neural network trained with our method not only represents multiple aligned 3D objects but also enables smooth interpolation of the reflectance field between these objects at each point in 3D
5

---

Published as a conference paper at ICLR 2025
Anchor
Latent
Hybrid
Anchor
Latent
kangaroo
goplacr
Hybrid model
points
code
model
points
code
Figure 4: Our method allows us to blend different objects seamlessly, A proper alignment of multiple 3D models provides the ability to replace parts of one object with similar components of the other objects. We manually select spatial anchor points (left) and assign them to particular model. The latent code is linearly interpolated between anchors at every spatial location, resulting in smooth distribution over 3D space (second column). The resulting objects are shown on the right.
space zi independently. This allows for a natural and seamless fusion of objects, blending specific parts of individual generated objects into new forms, such as gopher with head of kangaroo shown in Figure To achieve this, the 3D space is partitioned into regions corresponding to specific objects and the reflectance field smoothly interpolated across the boundaries of these regions. This partitioning is defined by smooth spatial distribution of the latent code (4) as illustrated in the second column of Figure 4 The new hybrid model rendered following Equations ①T] and (4) with the reflectance field now depending the spatially varying latent code (T, p) F (1, (1))
4.3 STRUCTURE-PRESSRRVIN TRANSFORMATION OF 3D MODELS
Our method can be easily adapted for the transformation of a given source 3D model into a target 3D model described by text prompt while preserving the original structure, such as pose and proportions.
For this, first, we set up the neural network as described in Section 4.1 for two text prompts N 2. In this setup, the latent code is defined on a one-dimensional segment {ur E [o, 1]; U2 1 u1 Then, we initialize the network with the input 3D model across the whole latent space uniformly. This initialization can be done in different ways depending on the representation of the input model. In our experiments, we obtain the renderings of the input model for random set of viewpoints and fit the network these renderings photometrically by minimizing the loss function in Equation (2).
Afterwards, we select a text prompt describing the input model (chosen manually for simplicity in our experiments) and assign the endpoints of the latent segment #1 = 1 and U2 to this prompt and to the target prompt, respectively. Finally, train the network with SDS described in Section 4.
additionally keeping the constraint on the photometric consistency with the input model (Equation (2) the respective endpoint of the latent segment u1
5 EXPERIMENTS
We demonstrate the capabilities of our method in the three scenarios described above. We compare our method with alternatives quantitatively for the generation of pairs of aligned 3D objects and for the structure-prest preservin transformation of 3D models. We discuss the results of the hybridization of aligned objects generated with our method from the qualitative perspective. We implement of our method based on MVDream text-to-3D generation model (Shi et al. 2024), that uses an efficient version of NeRF Instant-NGP (Müller et al. 2022) to represent the 3D scene. In Appendix A we show that our method also effective in combination with different model RichDreamer (Qiu et al. 2024) that represents 3D objects using DMTet (Shen et al., 2021). We show additional applications of our method Appendix E We refer the reader the complete set of animated results of our experiments on the project page for a more complete picture. In Appendix B.4 we provide the computational costs and hardware details.
Metrics. We quantify three aspects of the generated pairs of aligned objects and the results of the structure-preservinn transformation. The first one is the degree of alignment between the correspond- ing structural parts of the objects in generated pair, or of source 3D model and its transformed version. Measuring such alignment directly would require explicit detection of corresponding struc- tural parts for arbitrary pair of objects, which is hard task by itself, even if the objects have similar structure. Recently, Tang et al. (2023) have proposed a method for finding corresponding points in pairs of images of arbitrary similar objects by matching features extracted from pretrained

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison for the generation of multiple aligned 3D objects.
GPTEva13D, % of comparisons where method preferred
Text-asset Text-geometry
Texture details
Geometry details
Overall
DIFT distance
CLIP
alignment
alignment
plausibility
quality
% of object size
vs. MVEdit
94
89
76
91
92
89
MVEdit
27.1
5.5
vs. Dreamer
77
56
77
LucidDreamer
26.4
11.3
GaussianEditer
99
90
76
97
GaussianEditor
22.8
2.4
A3D (Ours)
27.7
6.1
2D diffusion models. Based on this method, called DIFT, we define DIFT distance that we use to measure the structural alignment. To compute this distance for a pair of objects, we render them from the same viewpoint. We densely sample points on one of the renders and find the corresponding points on the other one with DIFT. For an ideally aligned pair of objects, sample and its corresponding point have identical image coordinates. So, we define the DIFT distance as the average distance between these coordinates across all samples. We normalize it by the size of the objects in image space, for better interpretability. We report the value averaged across multiple viewpoints around the objects and for the points sampled for each of the objects in the pair.
Second, we measure the semantic coherence between a generated object and the respective text prompt. We measure it following the methodology of GPTEvaL3D (Mao et al. 2023). that was shown to align with human perception well. Specifically, we ask a Large Multimodal Model GPT- 4o (OpenAl 2024) to compare the 3D objects generated with two methods for the same text prompt and choose the one that is more consistent with the prompt, based on Text-Asset Alignment and Text-Geometry Alignment. We compare our method against each alternative and report the percentage of comparisons in which our method is preferred. Additionally, we measure the coherence between the generated object and the prompt using CLIP similarity (Jain al. 2022), which is defined as cosine similarity between the CLIP (Radford et al., 202T) embeddings of a render of the object and the respective text prompt.
Finally, we evaluate the visual quality of the generated objects and the quality of their surface. For this, we compare the objects generated with two methods using GPTEvaI3D based on 3D Plausibility, Texture Details, Geometry Details, and Overall quality.
5.1 GENERATION OF MULTIPLE ALIGNED 3D OBJECTS
We evaluate our method in the generation of sets of structurally aligned objects on 15 pairs of prompts describing pairs of objects with similar morphology but different geometry and appearance, such as car and a carriage. We include various categories of objects, namely different kinds of animals, humanoids, plants, vehicles, furniture, and buildings, see the list of prompts in Table 5
As no existing method targets the generation of aligned 3D objects, we adopt for comparison several methods of text-driven generation and editing of 3D models. To obtain a pair of aligned objects with such a method,
generate one of the objects from scratch and transform it into the other one. We compare with MVEdit (Chen et al.] 2024a), LucidDreames (Liang et al.] 2024], and GaussianEditor (Chen et al. 2024c). MVEdit takes mesh as an input and generates multiple views of the edited object with InstructPix2Pix (Brooks et al. 2023) diffusion network. Then, the mesh is optimized photometrically
be consistent with the edited views, iteratively reducing the level of diffusion noise. LucidDreamer uses Gaussian Splatting and its own version of SDS algorithm inspired by the works on 2D image editing. GaussianEditor also uses Gaussian Splatting in combination with methods of 2D generative guidance, in particular Instruct-Pix2Pix, and additionally develops an anchor loss to control the flexibility of Gaussians. It only performs text-driven editing but not generation,
generate the initial 3D objects for this method using MVDream (Shi et al. 2024). In our ablation study, we also compare with pairs of objects generated using MVDream independently.
We show the quantitative comparison in Table I and the qualitative comparison in Figure 5 Our method generates pairs of objects aligned with the text prompts and with high visual and geometric quality. It outperforms all the other methods on all evaluation criteria of GPTEval3D and w.r.t. CLIP similarity. Compared to our method, MVEdit produces less detailed objects. LrcidDreamer produces noisier geometry and often struggles with multi-view inconsistency. CanssianFditor often struggles
7

---

Published as a conference paper at ICLR 2025
MVEdit PP
LacidDreamer Pr P2
MVDream p1 GaustianEditor
A3D (Ours) PI
Text prompt for the left object pi: carriage.
Text prompt for the right object p2: car.
Text prompt for the left object Pi: lego animal.
Text prompt for the right object Pat animal.
Figure 5: Pairs of objects generated with existing methods and method. The top rows show the results for one pair of prompts written below, the bottom two rows show the results for another pair of prompts. For each object, show color rendering and rendering of the geometry below it.
to obtain an object corresponding to the prompt. This may be due to Instruct-Pix2PM5 (used by this method) generating inconsistent guidance from different views for synthesized objects, as was trained on real-world data.
Pairs of objects generated with our method have a high degree of structural alignment, which is confirmed by low DIFT distance, less than one tenth of the size of object. W.r.t. this metric our method is only slightly outperformed by MVEdit and GaussianEditor that, while obtaining the second object in a pair by transforming the first one, often fail to change the geometry of the initial object. This leads to poor alignment with the text prompt and low quality of the generated objects overall (see the values of GPTEva13D). Moreover, the other methods, which transform a pair into the other one, often generate a variant of the initial object with the geometric structure unsuitable for the other prompt, in the carriage-car pair produced by MVEdit. This fundamentally limits the quality of sets of objects generated with this sequential approach. In contrast, our method jointly optimizes the of objects so that they simultaneously share the structure and correspond to their respective text prompts well.
object in
5.2 HYBRIDIZATION: COMBINING THE ALIGNED 3D OBJECTS
We show examples of the hybrid objects combining parts of aligned objects produced by our method, and illustrate the process getting these hybrids Figure In these experiments, for better visibility we intentionally choose the hyperparameters of our method to increase the visual difference between the generated objects. To choose which part of each object we want to use, assign several anchor points to each object and manually place these points in the common 3D space of the objects. We define the spatial distribution of the latent code u (HL) at the location μ (described in Section (4.2 via linear interpolation between the latent codes corresponding to the objects associated with the two closest anchors.
The examples of hybrids demonstrate that our method generates aligned 3D objects that can be seamlessly blended in different configurations. The coherent appearance the hybrid models demonstrates high degree of structural alignment across the generated objects. Remarkably, our method allows us to easily transition between the parts of the objects with different geometries, e.g., the necks of the gopher and kangaroo, which have different diameters, waists of the seahorse and mermaid, which have fins and hands nearby. This in contrast to methods that represent 3D objects with mesh (e.g., MVEdit), which have to be locally adjusted first to be stitched together.
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison for structure-preeeevving transformation.
GPT of comparisons where methed is preferred
Text-assct Text-gommetry
Texture details
Geemetry details
Overall
DIFT distance
CLIP
alignment
alignment
plausibility
quality
% object size
MVEdit
76
66
50
80
84
83
MVFdit
27.9
3.7
vs. GausiaaFdiitor
94
94
83
97
97
100
GaussianF-dito
24.6
1.8
A3D (Ours)
Source
MVEdit
LucidDreammrr
GaussianEditor
A3D (Ours)
Text prompt: beautiful princess sitting throne
Diverged
Diverged
Text prompt: skeleton of
Figure 6: Objects generated with existing methods and our method from an initial 3D model on the left. Each row shows the results obtained for the text prompt below. For each object, we show color rendering and rendering of the geometry.
5.3 STRUCTURE-PRESSERVNO TRANSFORMATION OF 3D MODELS
We evaluate the capability of our method to transform an initial 3D model while preserving its structure on 26 text prompts. For each prompt find coarse initial model with the desired structure on the web, or use the SMPL parametric human body model (Loper et al., 2023) in a desired pose. In princess
this way, obtain, for example, a skeleton from 3D model of a simple model of a sitting woman, see the list of text prompts in Table We compare with the same text-driven 3D editing methods in the generation of pairs of objects.
throne from
We show the quantitative comparison in Table and the qualitative comparison in Figure 6 Lu- cidDreamer diverged for half of the scenes, generates objects aligned with the text prompts and with high visual and geometric quality, while preserving the geometric structure of the initial 3D model in terms of pose and proportions. It generally produces results on par with state-of ofie-oo-an art specialized text-driven 3D editing methods, which confirmed by the metrics. Our method consistently outperforms MVEdit w.r.t. the asset quality and alignment with the prompt, by producing more detailed results. Unlike MVEdit, which is restricted superficial deformations object requested by the prompt, e.g., adding the throne and crown for the princess, or shrinking the cat down to its skeleton. This also explains a slightly higher DIFT distance for our method, since these additional parts do not have the corresponding parts in the initial 3D model. LucidDreamer produces the objects with inconsistent and distorted geometry. It rarely preserves the pose and overall structure of the initial 3D model and often struggles with the Janus problem, producing the objects with multiple faces, limbs, etc. On the other hand, it generates more detailed visual appearance compared with our method.
only compare with it qualitatively. Our method
surface, our method is able to add or remove significant parts the
ABLATION
We compare our method with two branches of baselines for generating pairs of objects. We refer to these haselines (A), (B), (C), (E), (F), and to complete method (D).

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative ablation study.
DIFT distance percentage of object size
(A) MVDraam, independdnnlly generated objects
30.1
MVDream mmlltipl objects one network
(3), 3-layer MLP
14.3
(C) A3D, transition plausibility with empty prompt
14.2 (E) AID. 2-layer MLP
13.5
(D) A3D, transition plausibility with blended prongt (Ours)
6.1 (D) A3D, 1-layer (Ouns
In the first branch, we study the effects of embedding a set of objects into single neural field and the importance of regularizing plausibility transition between them. We start with the basic version of the text-to-3D framework MVDream (Shi et al. 2024) that our method is based on (A). MV generates pairs of objects independently from one another. We modify MVDream to parameterize two objects with single neural field without regularizing the transition between them (B). We also implement a version of our method that regularizes the transitions using a diffusion model conditioned on an empty text prompt (C), instead of a blending of the input prompts in our full method (D).
In the second branch of comparisons we study the importance of the smoothness of the transitions. We achieve this by following the reasoning in (Miyato et al. (018)) and limiting the depth of our neural network. Specifically, in our complete method (D), we parameterize sets of objects with a multilayer perceptron (MLP) with one hidden layer on top of feature hash grid. We evaluate two alternative designs that use MLPs with two and three hidden layers (E. F).
We show the quantitative comparison in Table the qualitative comparison in Figure g and provide more details in Appendix D.I Our results fully support the motivation behind the components of our method. MVDream (A) produces pairs of objects with different poses and proportions, which is confirmed by a high DIFT distance, corresponding to the nearly one third of the size of an object. Version (B) improves alignment across the objects but still does not lead to similar poses and proportions. These results show that independent generation of the objects with a single model, does not guarantee good structural alignment, which can be explained by the mode-seeking nature of SDS (Poole et al., (023). Using an empty prompt to enforce plausibility of the transitions (C) consistently improves the alignment between the corresponding structural parts of the objects, both qualitatively and quantitatively. This shows that the key property of our method is achieved through regularization of plausibility of the transitions and not through interpolation between the latent codes of individual objects. The interpolation that we use in our complete method (D) additionally makes the objects similar to each other (while sacrificing their realism and making them more stylized if they are naturally notably different from each other) and further improves the structural alignment across the objects. Further tuning of the plausibility loss weight allows one to control the degree of aligmment, as we discuss in Appendix D.2 Our experiments with increasing the depth of the network (D-F) show that enforcing the smoothness of the transition between the objects is essential for the proper alignment.
7 CONCLUSIONS
We present A3D, the first method designed to generate a collection of objects structurally aligned with each other. This is achieved by encouraging the transitions between the objects, jointly embedded into a shared latent space, to be smooth and meaningful, which is demonstrated to be an essential property for proper alignment. We show that, when applied to the generation of the structurally aligned objects our method outperforms the editing- competitors in terms of the asset quality and text-object alignment, while keeping the geometric structural alignment on the state-of-art level. When applied to the 3D editing task, our method provides the results on par with recent methods specialized in this problem. Our method allows to compose novel objects seamlessly combining the parts from the different aligned objects in the generated collection. Our approach is limited to generating static aligned objects, and can not be applied pose-changing tasks. In future work, we plan to extend our framework to changing poses and 4D video generation by experimenting with different regularization techniques for the transitions between objects.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors acknowledge the use of the Skoltech supercomputer Zhores (Zacharov al. 2019) to obtain the results presented in this paper. The research was partially supported by the funding of the Skoltech Applied Al center.
ETHICS STATEMENT
Our method is built on MVDream model, so inherits all the problematic biases and limitations that this model may have. For example, MVDream fine-tuned the open-source Stable Diffusion 2.1 model (Rombach Esser) on the Objaverse (Deitke et al.) 2023) and LAION (Schuhmann al., 2022) datasets. The LAION-400M subset of the full LAION-SB was found to contain unwanted images (Birhane et al., 2021), including inappropriate and abusive depictions. Our method may have the potential to displace creative workers through automation and increase accessibility for the creative and gaming industries. There is the risk that our method could be used to produce fake content.
REPRODUCIBILITY STATEMENT
We build our method based the official MVDream implementation using threestudio frame- work (Shi et al. [202cc). We discuss the details of implementation our method in Appendix [B. the details of experiments with the methods that we compare with in Appendix B.3 and describe the evaluation details in Appendix
REFERENCES
Chong Bao, Yinda Zhang, Bangbang Yang. Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10919-209299 2023.
David Berthelot Colin Raffel* Aurko Roy, and Ian Goodfellow. Understanding and improving in- terpolation autoencoders an adversarial regularizer. In International Conference on Learning Representations, 2019. /tt.tttt////peeeeviiiii
nwt. wor.oooruridddemm fos
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.019633 2021.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 18392-18402, 2023.
Hansheng Chen, Ruoxi Shi, Yulin Liu, Bokui Shen, Jiayuan Gu, Gordon Wetzstein, Hao Su, and Leonidas Guibas. Generic 3d diffusion adapter using controlled multi-view editing, 2024a.
Hansheng Chen, Ruoxi Shi, Yulin Liu, Bokui Shen, Jiayuan Gu, Gordon Wetzstein, Hao Su, and Leonidas Guibas. Mvedit code. https: //ts::///iii http//gttttt thub. com Laain Drteeni./MMVEe /MVEdi 2024b.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appear- ance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1244-22256. 2023a.
Y. Chen, Z. Chen, C. Zhang, Wang, X. Yang. Y. Wang, Z. Cai, L. Yang, H. Liu, and G. Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21476-21485, Los Alamitos, CA, USA, jun 2024c. IEEE Computer Society. doi: 10. 1109/CVPS52733..044000000
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhon- gang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussian editor experiment with bear. eddt__nn__bee
com http:///wwwaaeeeeaaiaaaaaatt//aa 2023b.

---

Published as a conference paper at ICLR 2025
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai. Lei Yang. Huaping Liu, and Guosheng Lin. Gaussian editor code.
com/
Suaacyw/Gaussiiee anEiito 2023c.
Yiwen Chen, Chi Zhang, Xiaofeng Yang, Zhongang Cai, Gang Yu, Lei Yang, and Guosheng Lin. It3d: Improved text-to-3d generation with explicit view synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1237-1244, 2024d.
Xinhua Cheng, Tianyu Yang, Jianan Wang. Yu Li, Lei Zhang. Jian Zhang, and Li Yuan. Progressive3d: Progressively local editing for text-to-3d content creation with complex semantic prompts. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview net/ (ttp:uummdd0002222220000
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13142-131553 June 2023.
Wenqi Dong. Bangbang Yang, Lin Ma, Xiao Liu, Liyuan Cui, Hujun Bao, Yuewen Ma, and Zhaopeng Cui. Coin3d: Controllable and interactive 3d assets generation with proxy-guided conditioning. In ACM SIGGRAPH 2024 Conference Papers. SIGGRAPH '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400705250, doi: 10.1145/341551935544255
URL http:tp:::/o...oo////////
3657425
Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint 3tti::2003.2223444 2024.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- eext-guided object generation with dream fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 867-876, 2022.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02246, 2023.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In 2020 IEEE/CVF Conference Computer Vision doi: 10.1 10.1090CVP4222002200.0008
and Pattern Recognition (CVPR), pp. 8107-8116, 2020.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. Gaussian splatting code. https:///ithuue wtt. http:///aa//gaa..o////ooooo/o//a htt.://dooooiiiiaaaassssaaa----- spw.tiiaaasplattiinngg 2023.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.n0 02640,0333333 2023.
Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bingbing Ni. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. Proceedings of the AAA! Conference on Artificial Intelligence, 38(4):3279-328) Mar. 2024, doi: 10. 10.1669/aaai.v330 v38.4. 13. URL /ittts:://oj. aaai. org/ wnw.nndex....o AAA /att///AAAaaattaaaaaaaaaaa httpp//eeeeeeee/////e//
Y. Liang, X. Yang. J. Lin, H. Li, X. Xu, and Y. Chen. Luciddreamer: Towards ligh-fidetity text-to-3d generation via interval score matching. In 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 6517-6526, Los Alamitos, CA, USA, jun 2024. IEEE Computer Soci- ety. doi: 10. 10.10099CVP55573..200444 006233. URL wtt.hhtpp::///t wee.oopptersocccc..............e org erg/10. 10. 1ttp/00.119////PR22222 20..333..22240000..
Yixun Liang, Xin Yang. Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer code. https: / thub. com/EnViii Research/fucidrremm. 2023.
12

---

Published as a conference paper at ICLR 2025
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang. Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 300-09, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image 3d object. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 0298-9309, October 2023.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. images from a single-view image. In The Twelfth International Conference on Learning Representations, 2024. URL
Syncdreamer: Generating
Beet/forum?iid-M33y222..
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
Lykon. Dreamshaper 8 code. httppss//hagggnnneee....//////er
2023.
Zhiyuan Ma, Yuxiang Wei, Yabin Zhang, Xiangyu Zhu, Zhen Lei, and Lei Zhang. Scaledreamerr Scal- able text-to-3d synthesis with asynchronous score distillation. arXiv preprint arXiv:2407.020400 2024.
Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. Gpteval: A survey on assessments of chatgpt and gpt-4. arXiv preprint arXiv:2308. 12488, 2023.
Luca Medeiros. Language segment-anythhnn code. https://github. wacoo//uua-aaae medeiros/
San.g-segmest anathiingnnttitiig.g..... 2024.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and
Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 201 8.
URL
/er/foormm? (RgziTT
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41)): 1-15, 2022.
OpenAL. Hello got-4o. 2024-05-26.
2024. Accessed:
Francesco Palandra, Andrea Sanchietti, Daniele Baieri, and Emanuele Rodolà. Gsedit: Efficient text-guided editing of 3d objects via gaussian splatting. arXiv preprint arXiv:2403.05154, 2024.
JangHo Park, Gihyun Kwon, and Jong Chul Ye. ED-neRF: Efficient text-guided editing of 3d scene with latent space neRF, In The Twelfth International Conference on Learning Representations. 2024.
net wtt.:/eeteeidaaaaa0e 3DVDm
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion:
Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023. URL
htttpp://openreet
net. ht///oormm/// Nyy
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9914-9925, June 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Ben Mildenhall, Nataniel Ruiz, Shiran Zada, Kfir Aberman, Michael Rubenstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani.
Dreambooth3d: Subject-driven text-to-3d generation. ICCV, 2023.
Robin Rombach and Patrick Esser. Stable diffusion v2-1. https://huggingttee co/
Robin Rombach, Andreas Blattmann, Domínik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Tim Sainburg, Marvin Thielk, Brad Theilman, Benjamin Migliori, and Timothy Gentner. Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions. arXiv preprint arXiv: 1807.06650, 2018.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks net for:m iid M3Y7 M3Y74vmsMcY
Track, 2022. URL httt:ttt//ppenreeeeee
Etai Sella, Gal Fiehelman, Peter Hedman, and Hadar Averbuch-Elor. Vox-e: Text-guided voxel editing of 3d objects. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 430-440, October 2023.
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Hyeonsu Kim, Jaehoon Ko, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. In The Twelfth International Conference on Learning Representations, 2024. URL http:::/
net forum? id=Ubxx iqUuo2
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for nigh-resolution 3d shape synthesis. In Advances Neural Information Processing Systems (NeurIPS), 2021.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310. 15/10, 2023a.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++ v1.2 code. https::/ httpsss//hhgggiggggghhce co/ oudo-ai/ Zerol23plus-vi.
2023b.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream code. 2023c.
https: /ttp:::///hhhbt http///yeeeaaaeeee/eeeaaaaaaaaaa
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations, 2024. htttps://
net http:///www.etteeeoooooooooraoaa
Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Garreau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, et al. Meta 3d assetgen: Text-to-mesh generation with high-quality geometry, texture, and pbr materials. arXiv preprint 2000:2407.. 2024.
Jiaxiang Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024a.
Jaaxiag Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaus- sian splatting for efficient 3d content creation. In The Tweifth International Conference on Learning Representations, 2024b. URL TRRLhttps.: :
http:ss://ppreeeeeeeeeeee// /ope/ revi ew net wtr.et//ffruueee http:///fooormmmmmoolllxxxmmmagm 6=UyNXMNNN33
14

---

Published as a conference paper at ICLR 2025
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Pemgg Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Zhengyi Wang, Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. CRM: Single image to 3d textured mesh with convolutional reconstruction model. In European Conference on Computer Vision, pp- 57-74. Springer, 2025.
Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Zuxuan Wu, Yu-Gang Jiang, and Tao Mei. Dreammesh: Jointly manipulating and texturing triangle meshes for text-to-3d generation. In ECCV, 2024.
Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent-1-to-3: Consistent image 3d view synthesis via geometry- diffusion models. In 2024 International Conference on 3D Vision (3DV), pp. 664-674, 2024. doi: 10.1 1201190DD2253.2222.0000
Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer. Fast generation from text 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6796-6807, June 2024.
Igor Zacharov, Rinat Arslanov, Maksim Gunin, Daniil Stefonishin, Andrey Bykov, Sergey Pavlov, Oleg Panarin, Anton Maliutin, Sergey Rykovanov, and Maxim Fedorov. "zhores" "zhores"-petaalopp" supercomputer for data-driven modeling, machine learning and artificial intelligence installed in skolkovo institute of science and technology. Open Engineering, 9(1::112-520, 2019.
Jingyu Zhuang, Chen Wang, Liang Lin, Lingjie Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-10, 2023.
15

---

Published as a conference paper at ICLR 2025
Table 4: Quantitative evaluation of the RichDreamer- implementation of our method for the generation of multiple aligned 3D objects.
Text-asset aligument
Text- cett-geometry aliggment
plausibilitt
Texture
details
Geometry detaills
Overall
quality
CLIP
% of object size
DIFT distance
vs. Lrcamer
$1
59
66
46
50
53
LucidDreamer
26.4
11,3
GaussianEdhhur
66
71
71
65
72
GoussianEdior
22.8
GP)))al31), % of comparisons where RichDreamer- based our method preferred
A3DaMMVDrram (Ours, main)
27.7
A3D EDD-RicDreamer (Ouns)
28.8
5.8
mormaid
and
skeleton
gopher kangaroo
Pairs of objects generated using the implementation of method based Rich- Dreamer (Qiu et al. 2024). The respective pairs of prompts are shown below. For each object, we show a color rendering and rendering of the geometry below it.
Figure/ 7:
A GENERATION OF MULTIPLE ALIGNED 3D OBJECTS WITH RICHDREAMER
To evaluate the generalizability of our approach, we performed additional experiments on the genera- tion of pairs of aligned objects using an implementation of our method based on the RichDreamer model (Quu et al. 2024). RichDreamer employs a hybrid DMTet (Shen al. 202T) representation for 3D content creation. It additionally integrates special normal-depth diffusion model alongside the Score Distillation Sampling (SDS) method to refine the geometry. This approach allows for sharper edges, enhanced surface continuity, and more realistic object appearance. In addition to improving geometry, RichDreamer also employs albedo diffusion process for texture learning. The optimization process is split into three stages. First, the coarse geometry is optimized to establish the basic shape of the 3D object. Then, this geometry is refined, improving the fidelity and detail of the object surface. Finally, the texture on the surface is generated.
In Figure 7 we show examples of the pairs of objects generated with this implementation of our method. In Table 4 show the quantitative comparison of this implementation with the other methods. Our results with RichDreamer demonstrate the ability of our method to preserve the quality when switching the backbone. This version of method produces the 3D objects with higher degree of structural alignment compared with the main version based on the MVDream model. The reason for this may be that enforcing normal smoothness casier with an SDF-based rendering backbone compared with a backbone based on radiance fields. On the other hand, the version of our method based on RichDreamer produces the results with a lower visual and geometric quality, w.r.t. GPTEva13D. One reason for this may be that RichDreamer sometimes struggles with generation of fine structures.
B IMPLEMENTATION DETAILS
B. IMPLEMENTATION OF OUR METHOD BASED ON MVDREAM
We use the same NeRF architecture and the majority of hyperparameters for training MVDream (hi et al. 2024). The neural density field is parameterized with an MLP network with one hidden layer, built top of hierarchical feature hash grid with the dimension of 32. The grid has 16 levels starting from the resolution of 8 with features per level. For SDS, we use the multi-view diffusion model from MVDream, and decrease the level of noise gradually during training.
16

---

Published as a conference paper at ICLR 2025
To implement the transitions, we introduce an additional latent vector that is concatenated with the coordinate embedding obtained from the hash grid and passed to the NeRF network. At each training iteration, sample the latent vector randomly from the latent simplex (described in Section 4.1). With the probability - p. we sample the latent vector from the vertices of the simplex, Le., optimize single object, and with the probability p. sample the latent vector from the edges of the simplex, i.e., optimize a transition between two objects. We obtain our main results with p 0.5. When the latent vector is sampled from an edge of the simplex, we additionally sample a scalar parameter t from the uniform distribution t ~ (0, 1) and obtain the latent vector u for the current training iteration via linear interpolation between the latent codes of the individual objects in the pair with this parameter I.
We employ two types of regularization on the normal maps: the orientation penalty described in (Poole et al. 2023) and the normal smoothness loss. Given the direction of the normal Ni at the pixel with the indices i, J. the smoothness loss is defined as
H H-1-W11 Σ =1
1
Napl + [N++1 - Nash |
(S)
H 1)(W -
where / and W are the dimensions of the normal map. We gradually increase the weight of the orientation penalty from 100 to 1000. We set the weight of the normal smoothness loss to the value of 10.
B.2 IMPLEMENTATION OF OUR METHOD BASED RICHDREAMER
We mostly use the default configuration of RichDreamer, including the details of the architecture and the optimization process. This involves utilizing an MLP with one hidden layer and 64 neurons for prediction of the SDF and another MLP with the same structure for prediction of the albedo. Similarly to the implementation of our method based on MVDream, we employ the latent code concatenated with the encoded points from hash grid as the input to these networks. We initialize the geometry representation using a uniform sphere with a radius of one, and utilize SDS with Stable Diffusion 2.1 and depth-normal diffusion model improve the accuracy of depth predictions. For smoother interpolation between prompts, we incorporate normal consistency loss, and after experimentation, found that setting the loss coefficient between 3 and yields better results than the original configuration. To generate textures, we use a material system based on a diffuse and point-light setup without background. For generating albedo maps, we guide the prediction with an additional diffusion model.
B.3 DETAILS OF TESTING THE OTHER METHODS
MVEdit. We used the official implementation of MVEdit (Chen et al. 2024b) with the default values of all hyperparameters except the denoising strength for text-guided 3D-to-3D pipeline. We changed this value from the default 0.7 to 0.8 to increase the scale the changes that the method makes the input 3D mesh. In experiments, this lead higher quality of the results produced by MVEdit.
We obtain pairs of aligned objects with MVEdit in three steps, the first two of which follow the text-to-3D generation pipeline of MVEdit.
We generate the images the first object the pair using the Stable Diffusion 1.5 model (Lykon 2023 Rombach et al. 2022) conditioned on the respective text prompt.
2. We generate the mesh of the first object from these images using the Zerol23++ model (Shi et al. 2023b a). This step includes extraction of object masks for the generated images, that we obtain using the Language Segment-Anythe 2023).
model (Medeiros) 2024 Kirillov et al.
3. We obtain the mesh of the second object in the pair by following the text-guided 3D-to-3D pipeline of MVEdit initialized with the previously generated mesh of the first object.
For the structure structure-preservinn preserving transformation of 3D models, we follow the text-guided 3D-to-3D pipeline of MVEdit in the straightforward way.
17

---

Published as a conference paper at ICLR 2025
LeameDr We used the official implementation of LucidDreamer (Lingg al. 2023) with the default values all hyperparameters.
We obtain pairs of aligned objects with LecmmDr in three steps, the first two of which follow the text-to-3D generation pipeline LucidDreamer.
1. We generate a coarse point cloud of the first object in the pair using the Shape-E model conditioned on the respective text prompt.
2. We obtain the Gaussian Splatting representation of the first object by initializing it using the point cloud from the previous step and optimizing it using SDS conditioned on the same text prompt.
3. We obtain the Gaussian Splatting representation of the second object in the pair by initializing it with the previously generated Gaussian splats of the first object and optimizing using SDS conditioned the text prompt for the second object.
For the suncttre-preservvin transformation of 3D models, we extract the point cloud from the source mesh and use this point cloud for initialization of the Gaussian Splatting in the generation pipeline of LucidDreamer.
GaussianEditor. We used the official implementation of GaussianEditor (Chen et al. 2023c) with the same values of all hyperparameters that the authors use for their experiment with the bear scene (Chen et al. 2023b).
GaussianEdiior takes as input Gaussian Splatting representation of the scene and performs the text-driven editing of this scene. For the generation of pairs of aligned objects, we obtain the input Gaussian Splatting for the first object in the pair generated using MVDream. For the structure- preserving transformation 3D models, we obtain the input Gaussian Splatting for the source mesh. In both cases, we obtain the second object in the pair or the transformed object with the following steps.
1. We obtain the input Gaussian Splatting from 120 renders of the initial object, by following the original implementation of Gaussian Splatting (Kerbl et al. 2023) with the default parameters and 30k training iterations. We render the initial object from 120 camera positions evenly located around and use the known camera parameters during optimization of the Gaussian Splatting.
2. We obtain the second object in the pair by running GaussianEditor with the prompt "Turn the prompt_/ into a prompt_2" where the prompt_l and prompt describe the initial object and the second object in the pair respectively. For structure-preserving transformation use the prompt "Turn it into target promppp..
We note that Gaussian Splatting does not provide explicit representation of the 3D surface, derive the renderings of the surface (e.g., shown on Figure 5) from the renderings of depth maps.
B.4 COMPUTATIONAL COST
We run all experiments on a single Nvidia A100 GPU. To generate single object, MVDream, which we use as the baseline of our method, requires 10k iterations, which corresponds approximately 45 minutes. To generate a pair of objects, our method typically requires 20k iterations, which corresponds to 1.5 hours. The two main steps of our adaptation of M MVEdit, namely text-driven 3D generation to obtain one of the objects in the pair and text-driven editing obtain the other object, require 40 minutes in total. To generate pair of objects, LucidDreamer typically requires 2 hours. With GaussianEditor, we generate an initial object in the pair using MVDream, which requires 45 minutes, and then edit the first object into the second one, which requires 15 minutes, the total time required to generate a pair of objects is hour. Overall, the running time of our method is comparable with the alternatives.
We have experimented with the generation of up to 5 aligned objects at a time using our method. We decided not to rely knowledge sharing and used a simple linear heuristic for scaling the number of iterations. We add 10k optimization iterations (45 minutes) per object, so that the generation of objects requires 50k iterations, which corresponds to 3 hours and 45 minutes. Informally, have
18

---

Published as a conference paper at ICLR 2025
noticed that sublinear scaling also produces the results of a high quality, it would also be possible to use fewer iterations.
C EVALUATION DETAILS
For evaluation, we place the results of all methods into the same coordinate space. We manually rotate the results of MVEdit for better consistency with the other methods. We render the results from 120 camera positions evenly located around the object, consistent with the threestudio format (hhi et al. 2023c).
The results for editing- based competitors split into two groups. The first group, which corresponds to the generative part of their pipeline produces "source" 3D objects. The second group consists of the objects that obtained by feeding the objects from the first group to the corresponding 31) editing pipeline. The object and its corresponding transformation could be obtained by taking some object prompt from the first group and taking its complementary prompt from the pair from the second group.
DIFT distance. Given pair of images LA, Lg and corresponding masks MA. MM we denote the DIFT (Tang et al. 2023) mappings from the first image to the second F' and from the second to the first as Fg. We build two 2D point clouds pa pD by filtering regular 2D grids of points with
masks MA, MB. We define the DIFT distance as following: SDIFT = 2N EN -
OP
FB PPP where OAA and dp are the diameters of the 2D pointclouds Pa and P We average the distance across the 120 viewpoints around the object.
OpB
GPTEva13D. We follow the procedure proposed by (Mao et al.) 2023) precisely, with one change. The version the GPT model used the original study is no longer available through OpenAI API, utilize newer version GPT-4o. We compare each pair of methods based on 90 pairwise comparisons. For each comparison, we randomly sample a pair of objects produced by the two methods for the same text prompt and compose the request to the model. Each request consists of pair of grids renderings the compared objects and a textual description of the questions to the model. Interestingly, the left result in the majority of comparisons regardless of the quality.
observed that the compressed version of the model, GPT-4o-mini, prefers
CLIP.
We use ViT-L/14 version of the CLIP model. We calculate the CLIP similarity between the RGB render of the object and the respective text prompt for each of the 120 viewpoints around the object and report the average value.
D ADDITIONAL RESULTS
In 5 show the quantitative comparison of methods for the generation of multiple aligned 3D objects per each pair of objects. In Table we show the quantitative comparison of methods for prrectue-preservvng transformation per each pair of initial model and text prompt. We note that LucidDreamer diverged for half of the scenes.
D.1 EXTENDED DISCUSSION ABLATION RESULTS
In Tables and 8 we show the results of the quantitative ablation study per each pair of objects, and in Table 9 we show the comparison using GPTEval3D. In Figure we show qualitative comparison for pair of objects, and refer the reader the complete set of animated results of our experiments on the project page. We study our two main regularizations: encouraging the network to (1) learn plausible transitions between the objects, and (2) learn smooth transitions.
To demonstrate the effects of progressively decreasing the strength of smoothness regularization, we compare method with 1-layer MLP (D), with the versions with 2-layer MLP (E), and 3-layer MLP (F). Weakening the regularization (increasing the number of layers) leads to a lower degree of alignment, confirmed by the DIFT distance, without any significant improvement of the visual and
19

---

Published as a conference paper at ICLR 2025
Table 5: Quantitative comparison for the generation of multiple aligned 3D objects per each pair of objects.
Similanity
J, % object
Prompt 1
Prompt 2
MVEdit
Lucidirrumerr
GaissianFditor
A3D (Ours)
MVEdi:
LucidIrreamer
GoursianEditor
A3D (Ours)
animaal
crab animal
29.2
24.8
19.9
28.0
5.0
10.7
1.9
5.9
bicycle
motorcycle
4.5
bird unimal
dinosaur animal
24.0
25.0
21.5
26.8
6.4
15.7
1.6
camtiage
21.8
23.9
22.2
29.1
3.0
10.5
1.6
4.4
dwarf
minotaur
27.2
24.4
17.0
26.7
6.0
9.3
gopher
kangaroo animal
horse onimal
horse skeleton
26.7
24.4
25.9
29.1
2.1
4.2
1.9
7.2
animal
unimal
25.9
25,4
26.0
24.3
5.1
16.7
5.5
magnolia
space marine
ww2 soldier
28.4
24.6
19.1
25.3
3.2
13.7
0.7
mermaid
scahorse
30.4
27.6
20.9
29.9
10.4
14.3
1.6
7.6
standing
robot
26.8
11.7
atakehune ship
modem yacht
27.1
27.8
9.7
8.7
gothic cathedral
hindu temple
26.6
31.3
23.8
26.6
4.9
12.2
1.8
3.5
chair
gothic throne,
26.6
28.9
25.2
26.5
3.2
10.1
1.7
Table 6: Quantitative comparison for cructtre-preservvvn transformation per each pair of initial model and text prompt. We explain how we use the prompt describing the input model with our method in Section 4.3
Smilalyy
J. ofjject
Prorppt descrhin the model
MVTId
LaxidTTmaae Conconidder A30)Om))))
MIVivl
Taaedibromzr Geseinldiie (Duno
aub warticr
greck hoplie
27.8
29.0
27.0
27.4
277
17.2
25.9
uxocado chair
cbai
27.7
26.4
32.5
62
3.8
63
clonts atting
eltixx
32.1
29A6
28.7
31.6
22
13.7
dnga
parme
188
bearded devel with ste
2522
21.5
233
305
3.4
15.1
188
54
ditting
siting
sutm planct with
plabe stna
24.5
192
26.0
4.1
11.6
goont
wusting jrane 1-bbin
23.7
11.
huiding both hands
22.5
263
256
ione Brone
amroga seedkn chair
31.9
34.9
27.5
5.6
1.8
15.5
juli lightuber
fensale jodi with lightnder
30,4
26.2
22.5
26.7
3.8
14.1
4.0
Trmaal jadi with lightoher
212
21.4
213
787
3.2
highly detailed
26.4
feesale marble staue
31.8
30.4
27.2
233
space warhammne
Litk
beme stintal
29.0
26.7
292
6.5
15.5
benutifal prnsass attine
feruse siting
25.2
21.1
IR6
25.4
12.9
rbbu
246
2533
24.7
25.5
3.4
14.9
12
98
standing
wenrina black telccat with rnd
and t-bitt
23.2
29.4
24.6
4.4
213
20

---

Published as a conference paper at ICLR 2025
Table 7: Quantitative ablation study comparing method (D) with the baselines per each pair of objects using DIFT distance. See descriptions of the baselines in Section
DIFT distance L. % of object size
Prompt
Prompt
(A) MVDream (8) MVDrcam-tuult
(C) Empty prompt (D) Ours (E) 2-layer MLP (F) 3-layer MLP
ant animaal
crab animal
33.8
34.6
26.9
5.9
15.8
21.2
bicycle
metoreycle
17.9
7.6
5.1
5.5
7.9
9.1
bird animal
dinosaur animal
28.3
21.9
13.6
15.1
13.9
carriage
33.0
29.7
23.7
4.4
18.6
12.4
gopher unimal
kangaroo unimal
37.7
38.2
27.3
4.3
6.7
8.0
animal
lego unimal
25.8
10.8
4.8
5.5
3.6
2.8
magnolia
space marine
ww2 soldier
21.7
10.0
13.0
7,5
9.1
17.8
mermaid
scahorse
34.1
27.4
24.0
man standing
robot standing
33.8
7.5
5.4
5.8
23.6
25.8
atakebune ship
modemn yacht
39.5
15.3
16.6
28.3
26.8
gothic cathedral
hindu temple
35.7
8.0
6.1
3.5
8.0
12,7
chair
gothic throme,
34.4
27.2
29.7
2.6
25.0
26.9
Average
30.1
18.7
14.2
13.5
14.3
Table 8: Quantitative ablation study comparing our method (D) with the baselines per each pair of objects using CLIP similarity. See descriptions of the baselines in Section 6
CLIP Similarity
Prompt
Prompt
(A) MVDream
(B) MVDream-tmult
(C) Empty prompt
(D) Ours
(E) 2-layer MI.P (F) 3-layer MI.
ant anioaal
animal
29.0
29.8
28.5
bicycle
motorcycle
27.3
26.0
26.6
25.7
26.5
27.0
bird animal
carriage
30.5
29.5
29.0
29.1
29.1
30.7
29.4
gopher animal
kangaroo unimal
30.4
29.8
30.2
28.7
29.5
29.9
animal
horse skeleton
28.9
29.7
30.5
30.3
animal
lego animal
26.4
25.0
26.4
24.3
24.0
23.3
magnolia
sakura
31.7
28.1
24.9
30.6
28.9
29.0
space marine
ww2 soldier
27.0
26.5
28.7
25.3
26.6
29.3
mermaid
seahorse
29.9
29.3
28.9
29.9
29.0
28.8
man standing
robot standing
29.3
29.9
30.8
29.9
29.0
29.7
stakebune ship
modern yacht
28.5
27.7
28.8
28.9
28.4
27.9
gothic cathodral
hindu temple
28.6
29.0
29.7
26.6
28.6
31.4
chair
gothic throne, royal
27.7
27.5
27.5
26.5
28.1
28.4
Average
28.5
28.1
28.4
27.7
28.0
28.5
Table 9: Quantitative ablation study comparing our method (D) with the baselines using GPTEva13D. See descriptions of the baselines in Section
GPTEva13D. % of comparisons where our method preferred
Text-asset alignment
Text-gooment aligument
3D
Texture details
Geometry details
Overall
plausibility
quality
(A) MVDream, independently generated objects
60.0
59.0
53.2
59.0
50.0
59.0
(B) MVDreum multiple objects in one network
38.0
37.1
38.7
38.7
35.5
33.9
(C) A3D, transition plausibility with empty prompt
47.0
44.9
65.6
54.4
68.8
68.8
(E) A3D, 2-layer MLP
60.2
$2.1
36.8
54.3
42.6
42.6
(F) A3D, 3-layer ML.P
59.9
$1.4
50.0
$3.6
50.0
51.5
21

---

Published as a conference paper at ICLR 2025
(A) MVDream (B) MVDream- mult. (C) Empty prompt
(D) Ours
2-layer MLP
3-layer MLP
and animal
Prompt:
silhouettss crab animal
Overlay of Prompt:
Figure 8: The first two rows show pairs of objects generated with our method (D) and the baselines (see the main text for their description). The last row shows an overlay of the silhouettes of the objects, demonstrating the alignment of their structural parts.
geometric quality, confirmed by GPTEva13D and CLIP. We relate the preservation of the quality to the following. The Instant-NGP 3D representation, which we use for the main implementation of our method, mostly stores the neural field in the feature hash grid, while the MLP only plays an auxiliary role.
To demonstrate the effects of progressively removing the plausibility regularization, compare our full method with the regularization conditioned on a blend of the text prompts (D), with the regularization conditioned on an empty text prompt (C), and without the plausibility regularization (B). Quantitatively, decreasing the plausibility of transitions also decreases the degree of alignment between the objects, confirmed by the DIFT distance, but may slightly increase the visual and geometric quality of the results and their semantic coherence with the text prompts, measured with GPTEva13D and CLIP. Qualitatively, when we relax the restriction on the plausibility of transitions, the objects become less strictly aligned with each other and obtain more characteristic properties corresponding to the text prompts, especially they are naturally different. For example, the ant and crab get generated with different numbers of legs, and the crab obtains pair of claws; the car in the car-carriage pair changes from vintage to modern; the proportions of the gopher and kangaroo become more naturalistic.
D.2 VARYING DEGREE OF ALIGNMENT
We achieve the alignment between the objects through regularization of transitions between them. Specifically, we encourage the network to learn plausible and smooth transitions. By varying the strength of these regularizations, we can control the degree of alignment between the objects and choose between more strict or more loose alignment.
The strength of the plausibility regularization is defined by the probability p of sampling the latent code from the edges of the latent simplex instead of its vertices (see Appendix for details), In Figure 9 we compare the results of our full method (D) with the results obtained with a de- creased strength of the regularization (G), and without the regularization (B). When we decrease the plausibility of transitions, the objects become less strictly aligned with each other and obtain more characteristic properties corresponding to the text prompts. For example, the ant and crab get generated with different numbers of legs, and the crab obtains a pair of claws, while the proportions of the gopher and kangaroo become more naturalistic.
We demonstrate the effects of progressively decreasing the strength of smoothness regularization in our ablation study (Section 6 and Appendix D.I). We compare the results of our method with l-layer MLP (D), 2-layer MLP (E), and 3-layer MLP (F) in Figure 8 and on the project page. The variants with more layers produce more loosely aligned objects.
On the project page, we show examples of transitions between the generated objects. The transitions generally smooth, gradually transforming one object into another. The plausibility of the interme- diate results is higher for pairs of objects with a higher degree of alignment. The implementation of method based on RichDreamer produces smoother transitions compared to the implementation
22

---

Published as a conference paper at ICLR 2025
Ours
(G) Wenker reg.
No reg.
Ours
Weaker reg.
(B) No reg.
Object
Object
Silhouettes
crab animal
gopher kangaroo animal
Figure 9: The first two rows show pairs objects generated with our method in the default con- figuration (D), with decreased strength of the plausibility regularization (G), and without this regularization (B). The last row shows an overlay of the silhouettes the objects, demonstrating the alignment of their structural parts. Each three columns show the results for pair of prompts written below.
based MVDream, which we relate the implicit surface prior in DMTet that encourages the network to learn smooth geometry.
E OTHER EXPERIMENTS
Our method can generate instances of the same object with different details. In Figure 10 we show examples of objects with adjusted accessories.
Our method can produce the results with some degree of diversity for fixed set of text prompts, as we show the top row Figure [T] The diversity of the results produced by our method is mostly defined by the frameworks that we use for implementation: MVDream and RichDreamer. These frameworks use Score Distillation Sampling. Optimization with the Score Distillation Sampling employs high values of the Classifier-Free Guidance scale, which leads to mode-seeking behavior and lack diversity. Additionally, the diffusion models used in these frameworks are tuned on Objaverse dataset, which further decreases the diversity of the results produced by these models. One way to obtain different instances of the same set objects with our method is to describe the desired differences in the text prompts,
show at the bottom row of Figure 11
Our method is robust to variations the text prompts. In the first two rows of Figure 12 we show the results for pairs of text prompts with the same meaning but with different phrasing. While not identical, the generated pairs exhibit a high degree of alignment between the objects and correspond to the prompts well. In the last of Figure 12 we show the result for pair of prompts describing two different objects with the same attributes but using different phrasing, These objects are also aligned well and have high quality. Overall, A3D does require too much prompt engineering but one can expect some diversity in the results for different formulations of the text prompts.
We show a possible approach of applying our method an image-to to-se-tt-3DD model by performing the structure-preservive transformation of 3D object generated with this model. We show some results obtained in this way Figure 13 We generate the initial 3D models using the fast image-to-3] pipeline of CRM [Wang et al. 2025) and then transform these models using method as described Section 4.3 We use the faster implementation our method based on the RichDremmer framework. Our method preserves the structure of the initial object generated from the image well.
23

---

Published as a conference paper at ICLR 2025
brown monkey animall
with rod hat
animall waaring rod how
woman in black aneakers woman is red hoots
Figure 10: Pairs of instances of the same object with adjusted details generated with our method.
ADD (Ouns), instance
A3D (Dur), instance
AID (Our), inatnace
AID (Ouns), instance
nunning animal with bow ninning animal with bow
snimal en homee skeleton
A3D (Cless),
AND (Ours,
(Ouns)
standing
standing robot standing
standing and Tabiit mbbo standing
The top row shows two examples of slightly different instances of pairs of objects generated with our method for the fixed pairs of text prompts, written below. For some text prompts, the baseline text-to-3D framework MVDream tends to produce the exact same object every time (bottom row, left). For the respective pairs of prompts, method is also limited (bottom row, middle). One way to obtain different instances of the same set of objects with our method in this case is to describe the desired differences in the text prompts (bottom row, right).
Figure 11:
diversity of the results produced by our
black standing majestically with shiny coor
hurse skeloton
sheek blsck. and proud
horee skeleton
grean-haired stands waseing bright clothes
young age, black L-birt and jenns, short hair
standing grozn hair and bright clotbes
young man with short werring a black Tashirt and Jeans
D
jumping racoum weuting black pirtto
tiper that jumps block pirate hat
Figure 12: Each of the top shows two pairs of objects generated with method for pairs of text prompts with a similar meaning but a different phrasing. The last row shows the result for pair of prompts describing two different objects with the same attributes but using different phrasing.
24

---

Published as a conference paper at ICLR 2025
Turget prompti skeleton
Target pronpt rohoc
Figure 13: Examples of application of our method to image-to-3D pipeline. Each triplet of images shows an input image, the 3D model generated with the image-to-33 pipeline, and the result of structure- structure-preserving transformation of this model using method with the target prompt written below. The input image of the black cat is designed by macrovector Freepik.
25


Reviews:
[{"review_id": "0l0mM039R1", "reviewer": "Reviewer_R6mh", "summary": "This paper tries to tackle the problem of text-driven 3D generation from a geometry alignment perspective. To achieve the alignment of the corresponding parts of the generated objects, this paper tries to optimize the continuous transitions in a shared latent space. Visualization results demonstrate the effectiveness.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of optimization in a shared latent space for smooth transitions is reasonable.\n\n2. This writing style is easy to follow.", "weaknesses": "1. Fairness issues. This work is built on MVDream, which highly relies on the pre-trained knowledge from SD2.1. This raises the question: compared with other approaches, does the effectiveness of A3D come from the method itself or the pre-trained knowledge of the SD model? Relevant experiments are needed to demonstrate it. Besides, the authors are encouraged to compare different SD versions, like SD1.5 and SDXL.\n2. In Tab.2, MVEdit outperforms A3D on CLIP score and DIFT distance. The authors claim that A3D can add or remove significant parts. However, this indirectly indicates that pre-trained SD models may cause unfair knowledge leakage.\n3. What is the training cost compared with baseline methods, like MVEdit, LucidDreamer, and GaussianEditor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVmOO7RMwP", "reviewer": "Reviewer_2STX", "summary": "This paper tackles an interesting problem of hybridization and editing regarding geometry-aligned text-to-3D generation, in which they leverage 3D shape and transitional embedding between different prompts to ensure continuous transition within the same 3D representation. By embedding these objects into a common latent space, they achieve both seamless hybridization of different shapes as well as high-quality 3D alignment between the generated scene and template 3D shape.", "questions": "See the weakness section above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to follow.\n- The motivation the paper is sound, tackling the task of 3D shape editing through addressing hybridization and alignment of difference 3D scenes corresponding to the given shape, which I believe is an innovative and novel solution to the task at hand.\n- The technical solution this paper offers to the problem is simple and effective: training a single NeRF model to handle multiple scene from different prompts during the optimization phase, and then allowing for arbitrary translation between the different scenes with interpolation of latent vectors is a elegant approach to achieve 3D alignment and controllable 3D editing of the scene.", "weaknesses": "- This paper uses NeRF for seamless representation of difference scenes, and I understand design choice was to enable seamless transition between different optimized scenes within the representation space of NeRF. However, more explicit 3D representations such as Instant-NGP or 3DGS offer their own advantages, such as speed and controllability, and I believe are a viable candidates for comparison. Can this method be applied to such different forms to 3D representation? If so, how do their performance different from the current model that uses NeRF?\n\n- How much 'deformation' from the given template shape does this method allow for? Figure 4 shows the generated shapes display variations in geometry from the given template - is there any mechanism within the paper that specifically allows for this \"loose\" alignment (SD operating in the latent space, perhaps)? How does 3D geometry look in the \"middle states\" of transitioning the entire network from one object to another? \n\nI am very much willing to raise the score if the mentioned questions are addressed during the discussion phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqynD5cvAq", "reviewer": "Reviewer_b2FB", "summary": "This paper proposes A3D, a new method to generate structurally-aligned 3D objects from a set of text prompts. A3D can first generate multiple aligned 3D objects, then make hybrids combined of different parts from the generated objects. Another advantage is that A3D can achieve smooth structure-preserving transformation given a coarse mesh.", "questions": "See the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall idea is interesting. The authors observe that the previous 3D editing method fails to generate structurally-aligned high-quality 3D objects (e.g., either lack of structure alignment or detailed texture). A3D provides an effective way to generate aligned 3D objects by introducing additional latent code conditions.\n2. The results in Figure 3 demonstrate that A3D indeed learns a smooth transition between different latent codes and can be decoupled with 3D positions by setting anchor points.\n3. The authors provide sufficient experiments and comparisons with previous works and A3D has better performance in generating aligned objects.\n4. The presentation is clear and easy to follow. Some visualizations are useful for understanding the core idea of learning smooth transitions.", "weaknesses": "1. I noticed that some generated textures from A3D are still poor. Is this caused by the limitation of SDS loss or the framework of A3D?\n2. I am wondering whether A3D is sensitive to text prompts. It would be better if the author could evaluate this point. is A3D robust to various text prompts?\n3. In section 4.1, A3D uses some regularization strategies to supervise the network, including limiting the depth of the neural network and enforcing the smoothness of the rendered results. which will cause performance degeneration compare with raw SDS-based optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jDtrRpgFm", "reviewer": "Reviewer_nGyS", "summary": "The paper proposes a method that enables the joint generation of a collection of 3D objects with semantically corresponding parts aligned across them.  A3D addresses 3D editing and consistent generation by embedding the 3D objects and the transitions between them into a shared latent space and enforcing the smoothness and plausibility of these transitions. This allows for several practical scenarios that benefit from alignment between the generated objects, including 3D editing, object hybridization, and structure-preserving transformations.", "questions": "1. Can it be generalized to other diffusion models like Stable Diffusion-based SDS?\n\n2. Is it capable of performing on Gaussian splatting-based paradigms?\n\n3. Can this method generate diverse sets of 3D results given one single set of text prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Proposed A3D can jointly generate a collection of 3D objects aligned in structure from a set of text prompts. This method can be applied to various scenarios, such as 3D editing, object mixing, and structure-preserving transformation.\n2. Compared to existing methods that generate 3D objects independently, A3D can generate a collection of 3D objects that are structurally consistent, text-aligned, and of high visual quality.\n3. A3D can be used in conjunction with different text-to-3D generation frameworks to improve the generation quality while maintaining the structure.\n4. The results are interesting.", "weaknesses": "1. Lacking a diagram that can illustrate the process of the transition and latent coding process. Only the text description on this part might be not clear enough.\n\n2. The generated results are confined to the formed shape in the latent space, which might result in unnatural and unpleasant results. e.g., in Fig. 6, the crab and ant share the same latent shape but the generated assets are not reasonable enough for both(Too many legs for the ant and no claws for the crab).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IR6DjjuUVf", "reviewer": "Reviewer_DfS8", "summary": "This paper focuses on generating a set of objects with similar structures using one text-to-3D generation model. To this aim, it introduces a  latent code to embed these desired objects into a common space. Besides training the text-to-3D generation model with their individual texts, this work enforces two kinds of regularization on the transitions between the objects. Experiments on multiple aligned 3D objects generation, combining parts of different objects into a ''hybrid'' and preserving transformations of a 3D object verify that the proposed method can generate diverse objects while maintaining structural consistency.", "questions": "1. I am curious if the the proposed method can be applied to image-to-3D generation methods. Since one image can be converted into a text embedding, the proposed method should support the image-to-3D setting.\n2. What is the training time overhead for the proposed method?  Can you compare the training time with other methods and discuss how the training time scales with the number of objects being aligned?\n3. For the ablation study, Table 3 only compares the DIFT distance metric. Can you also compare other metrics like Table 1 and Table 2?\n4. I wonder if the proposed method can edit some details. For example, \"a happy running man\", \"a sad running man\", \"a happy running minotaur\" and \"a sad running minotaur\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper introduces an interesting problem, how to make one text-to-3D generation model flexibly produce different objects while maintaining structural consistency?\n2. The proposed method proposes to embed these objects into a common latent space, allowing them to share similar structures.  \n3. Experiments show the potential of the proposed method on different tasks, including generation of multiple objects, hybridization and structure-preserving transformation.", "weaknesses": "1. The paper is not well-written. It does not provide an overview to introduce the method architecture. It is better to provide a high-level flowchart of the proposed method, and then introduce the components of the proposed method by referring them to the presented flowchart. In the Method section, many details are given in the appendix.\n2. The experiments lack the ablation studies for the two kinds of regularization on the transitions between objects. Can you remove the two regularizations, respectively and conduct corresponding ablation experiments to verify their effectiveness?\n3. Can the proposed method be applied to image-to-3D generation models? Can you discuss the potential challenges or modifications needed to adapt their method to image-to-3D models, and how this might impact the structural consistency of the generated objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)