Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation
Authors: Slava Elizarov
Ciara Rowles
Simon Donné
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-3D; Geometry Image; Non-watertight mesh; Data efficiency
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a fast and data-efficient Text-to-3D model using geometry images as the surface representation.

Abstract: Generating high-quality 3D objects from textual descriptions remains a challenging problem due to high computational costs, the scarcity of 3D data, and the complexity of 3D representations. We introduce Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D model that utilizes geometry images to efficiently represent 3D shapes using 2D images, thereby avoiding the need for complex 3D-aware architectures. By integrating a Collaborative Control mechanism, we exploit the rich 2D priors of existing Text-to-Image models, such as Stable Diffusion, to achieve strong generalization despite limited 3D training data. This allows us to use only high-quality training data while retaining compatibility with guidance techniques such as IPAdapter. GIMDiffusion enables the generation of 3D assets at speeds comparable to current Text-to-Image models, without being restricted to manifold meshes during either training or inference. We simultaneously generate a UV unwrapping for the objects, consisting of semantically meaningful parts as well as internal structures, enhancing both usability and versatility.

Published as a conference paper at ICLR 2025
GEOMETRY IMAGE DIFFUSION: FAST AND DATA-EFFICIEN TEXT-TO-3D WITH IMAGE-BASED SURFACE REPRESENTATION
Slava Elizarov, Ciara Rowles, Simon Donné Unity Technologies
ABSTRACT
Generating high-quality 3D objects from textual descriptions remains a challeng- ing problem due to high computational costs, the scarcity of 3D data, and the com- plexity of 3D representations. We introduce Geometry Image Diffusion (GIMD- iffusion), novel Text-to-3D model that utilizes geometry images to efficiently represent 3D shapes using 2D images, thereby avoiding the need for complex 3D- exploit aware architectures. By integrating Collaborative Control mechanism, the rich 2D priors of existing Text-to-Iman models, such as Stable Diffusion, to achieve strong generalization despite limited 3D training data. This allows us use only high-quality training data while retaining compatibility with guidance techniques such as IPAdapter. GIMDiffusion enables the generation of 3D assets at speeds comparable to current Text-to- Text-to-Image models, without being restricted to manifold meshes during either training or inference. We simultaneously generate UV unwrapping for the objects, consisting of semantically meaningful parts well as internal structures, enhancing both usability and versatility.
INTRODUCTION
Automatic 3D object generation offers significant benefits across video game production, cinema, manufacturing, and architecture. Despite notable progress in this area, particularly with Text-to- 3D diffuuson models (Boss et al.] [2024] Siddiqui et al. 2024 Wang et al. 2023), generating high- quality 3D objects remains a challenging task due to computational costs, data scarcity, and the complexity of typical 3D representations. For one, it is crucial that the generated objects be re-lit within graphics pipelines, necessitating the use of physically-bar -based rendering (PBR) materials, for which little data scale exists. Furthermore, graphics pipelines predominantly use meshes their primary 3D representation: processing these at scale is notoriously difficult due to their irregular graph structure. Most techniques instead generate an intermediate representation, which increases the burden of training data pre-processing and generated object post-processing.
We propose diffusing joint albedo textures and geometry images (Gu al. 2002). 2D representation of 3D surfaces, using Collaborative Control scheme (Vainer et al.) 2024). This enables 3D object generation from text prompts, as shown in fig. [ The image-basee based representation allows us to repurpose existing image-based architectures, while the Collaborative Control scheme enables us to leverage pre-trained Text-to-Image Image models, considerably reducing the required training data and costs. Geometry images, and more specifically multi-chart geometry images (Sander et al.) (2003), offer two great advantages over other shape representations: they impose no constraints on the topology of the generated object and inherently partition it into semantically meaningful parts, allowing for easier manipulation and editing.
We believe that GIMDiffusion opens up promising new research direction in Text-to-3D tD generation, providing a practical and efficient approach that can inspire future advancements in the field. In summary, the advantages of GIMDiffusion include:
Image-based: By leveraging existing 2D image-based models instead of developing new 3D) architectures, we simplify both model design and training.
Fast Generation: We generate well-defined 3D meshes in under 10 seconds per object, which could be further enhanced using distillation techniques.

---

Published as a conference paper at ICLR 2025
palm a
A smealker
metal sirield
A satchel bag
A delicious burger with lettooe, tomato, and cheese
mustmmm
Bly
Aeather ket with spiiess
Fantasy wespon. buttle seythe
wooden chest with" patterns celtic knows
A scooter bike
Highly urnale gold intaid
Figure 1: The objects are generated entirely using our method, including the structure, texture, and UV map layout, all created completely from scratch. For each object, we show the generated albedo texture, the textured mesh, the untextured mesh, and the respective text prompt.
Generalization:
Through collaborative control, we reuse pre-trained Text-to-Image priors, allowing for strong generalization beyond our limited 3D training data.
Separable Parts: GIMDiffusion creates assets that consist of distinct, aamantically-meaningie separable parts, facilitating easier manipulation and editing.
Albedo Textures: The 3D assets generated by GIMDiffusion do not have baked-in lighting effects, making them directly suitable for graphics pipelines.
UV unwrapping: Our method jointly generates UV unwrapping for the objects, learned from the training dataset. To the best of our knowledge, this is the first Text-to-3D method capable of achieving this.
Direct geometry: Our 3D assets do not require the application of iso-surface extraction algorithms, which reduces potential artifacts and simplifies the overall workflow.
RELATED WORK
2.1 TEXT-TO-IMACE GENERATION
Diffusion models (Soh--Dickstein et al. 2015 Song et al. 2020) and flow matching (Lipman et al., 2022). alongside the rise of versatile, general- purpose architectures such as transformers (Vaswani et al. 2017). have brought considerable progress in generative modeling. In particular, text- conditioned image generation was revolutionized by approaches based on Latent Diffusion (Rom- bach et al.) 2021) and its further extensions (Podell et al., [223]] Pernias et al., 2024 Esser et al. 2024). Foundational models like Stable Diffusion, trained on extensive internet-scale datasets (such as LAION-5B (Schuhmann et al. 2022 ), are capable of generating complex scenes from text prompts while exhibiting an implicit understanding of scene geometry. Due to the high cost training such
purrpoe-
models, they often repurposed for other tasks or modalities (Zhang et al. 20033; Hu al.) 2023 Ke et al.; 2023). Our proposed GIMDiffusion a prime example of this, as adapts the base model specifically output albedo textures.
2

---

Published as a conference paper at ICLR 2025
2.2 CONDITIONING DIFFUSION MODELS
Control mechanisms modify pre-trained foundational models, enabling them to accept additional conditions. Existing pixel-aligned control techniques fall into two categories: fine-tuning the base model with modified input and output spaces (Duan et al. 2023 Ke et 2023)) or a separate model that alters the base model's 's internal states (Zhang et al. 2023] Zavadski et al.] (023). The latter approaches, such as ControlNet (hhng 20.3). have gained wide adoption due their ability to preserve the original model's performance while adding conditions such as human poses or depth images. AnimateAnyom (Hu et al. 2023) leverages similar architecture to inject the base model's
hidden states into new branch that aligns with the base model's 's output.
In our method, we need to both control the base model (which will output UV-space albedo textures) and extract sigmificant features from it (to generate the geometry image modality). Collaborative Control (Vainer et al.] 2024) achieves exactly this by introducing bidirectional communication between both models, originally designed for Bext-to-PBR-Tesee generation.
2.3 TEXT-TO-3D CENERATION
We identify two main approaches to Text-to-3D generation: optimization-basee and feed-forward methods. Optimization-bati sampling (SDS) (Poole et al. [022 Wang et al. 2022 2023) to iteratively optimize a 3D scene, represented e.g. by NeRF (Mildenhall et al. 2020) or Gaussian Splats (Kerbl et al. 2023). These methods can produce content of high perceptual quality, but at the cost of impractically long generation times (Lorraine et al., 2023 Xie et al.. (024). The key advantage of this approach is its ability to utilize the rich 2D prior, allowing for 3D object generation without the need for expensive 3D data.
methods adapt 2D diffusion models to 3D by applying score distillation
However, the lack of camera conditioning leads to discrepancies among different viewing angles (Janus effect (Poole al., 2022) ) and projection artifacts. To mitigate these issues, 3D-aware
architectures and retraining on restrictive 3D datasets are often used, which can weaken the 2D prior (Shi et al.) 2023b Liu et al. 2023 Höllein al. 2024| [Zheng & Vedaldi 2023 Shi et al.) 2023a Kant et al., 2024). Additionally, the original SDS formulation can lead to issues such as saturated
colors, oversmooth geometry, and limited diversity (Wang al. 2023 Zhu et al. 2023 Katzir et al.,
2023 Alldieck et al. 2024. Liang et al., 2023; Wu et al. 2024).
Feed-forward methods directly generate 3D shapes without the need for iterative refinement. These methods employ complex, specialized architectures and typically require training on expensive 3D data from seratch. While seminal works like Point-E (Nichol et al. 2022) and its follow-ups (Huang al. 2024 Zeng et al. 2022) demonstrate impressive generalization and diversity, the inherent lack of connectivity information limits the expressiveness of point clouds. Instead, many current methods rely on other proxy representations, such as neural implicits (Xie al. 2021 Mildenhall et al. 2020 Malladi et al. 1995] Jun & Nichol 2023) Zheng et al., [022 Chen & Zhang
2018; Mescheder et al.
2018 Yariv et al.
2023) or triplanes (Hong et al.) 2023; Chan al.
2021 Tochilkin et al. 2024
Boss et al. 2024 Bensadoun et al. 2024), to represent the objects.
Both groups of methods require pre- and post-processing to transform between the mesh domain and the proxy representation, e.g. through marching cubes (Lorensen & Cline 1987) or tetrahedra (Doi & Koide 1991). This process is costly and lossy, introducing issues such as quantization or grid- like artifacts, and leading to information loss, the loss of part segmentation and internal structures. Moreover, generated objects often require UV unwrapping, a complex post-processing step, for use in artist workflows. In contrast, our model jointly generates the UV map along with the object shape, requiring only a simple triangulation step.
2.4 GEOMETRY IMAGES
have been largely overlooked in deep learning (Sinha et al. 2016). XDGAN (Alhaija et al. 2022) is pioneering effort that utilizes GIMs as the representation of choice for a StyleGAN based architecture (Karras et al., 2018). However, due to the architcctural constraints, the training data must be perfectly aligned with template atlas, which limits its applicability to real-world data. Furthermore, the pre- pre-processing algorithm provided in the paper is restricted to shapes of genus zero Section shows how to handle arbitrary shapes.
Geometry images (GIMs) Gu et al. 2002 Sander et al. 2003)
3

---

Published as a conference paper at ICLR 2025
Fiy aganit
1331
Nlbedo
SD
Goomctyy
Trinng
hausindy Ounstau
Sell-ateection
Grsowsttention
Cross communicciion layer
Figure 2: The method overview. Two separate diffusion models generate respectively albedo textures and geometry images. The former is frozen pre-trained model, while the latter is an architectural clone trained from scratch. Models interact via the Collaborative Control Scheme. To obtain a mesh, generated geometry image is triangulated with the algorithm described in section 3.3.3
Recent concurrent work (Yan al. 2024) has advocated for low-resolution 64 64 geometry images as a 3D representation for clas-cood diffusion models and highlights its efficiency on small-scale dataset of 8000 objects (Collins et 2021), albeit with limited generalization. In contrast, GIMDiffusion addresses general Text-to-3D. rather than training model from scratch, we leverage a pre-trained Text-to- imaem diffusion model (using a Collaborative Control scheme trained on Objaverse (Deitke et al. ) to retain generalization and diversity in the shapes, their appearance, and the UV atlas layout.
METHOD
3.1
GEOMETRY IMAGES
Geometry images (Gu et al. 2002) encode 3D surfaces in image format. This is achieved by
R3, where denotes the 3D surface, and [o, 1j represents the UV coordinates in the unit square, typically sampled uniform grid of the desired resolution. The function Φ also known as a surface parametrization (Floater & Hormann 2005). The choice is erucial and is usually designed to minimize spatial distortion (e.g. conformal mapping).
mapping function o: [0, 1J2
+ 1
Unlike 3D meshes, which require explicit data structures to maintain connectivity, geometry images implicitly connect each pixel its neighbors. Gu et al. (2002) tackled surface mapping by cutting the input surface and warping it onto a disc, but this approach is limited to manifold objects and introduces significant distortions for high-genus shapes. Multi-Chart Geometry Images (Sander et al.) 2003) address this limitation by mapping surfaces piecewise onto multiple charts of arbitrary shape, each homeomorphic disc. This approach adds flexibility (removing the manifold constraint) and reduces distortion (providing greater geometric fidelity). However, Multi-Chan Geometry Images lack connectivity information between charts and are susceptible sampling artifacts, which can lead to visible cracks along boundaries, as discussed in section 4.5 Additionally, the algorithm proposed to construct restricted well-behaved manifold meshes. The optimal surface parametrization
remains active area of research (Sawhney & Cranc 2017 Srinivasan al. 2023).
Most available 3D meshes have textures applied through UV maps. We observe that these handcrafted maps can be used to construct desirable multi-chart Φ. Beyond providing low-distortion mappings, these UV maps strategically place chart boundaries in locations optimal for animation and downstream processing tasks. Furthermore, charts in handerafted UV maps often carry semantic meanings, which propagate to the output of our model. This evident in fig. 3 where the hands, face, and various parts of the gunslinger's appearance separated in the UV atlas.
4

---

Published as a conference paper at ICLR 2025
(b) Extructed geometry image
Input mesh
Extmattdd albedo texture
(d) Reconstructed mesh
Figure 3: (a) Ground-truth geometry, (b) geometry image and (c) albedo texture from our data pre-
processing, and (d) the reconstruction using dedicated VAE. We note the highly separable nature of the ground truth object, which is split into small components. The only visible artifact after decoding the missing connection between charts of the geometry image, discussed in sections 3.3 and 4.5
As the density of the generated geometry's triangulation is limited by the resolution of the underlying geometry image, we follow Rombach et al. (202T) and use VAE to increase the effective resolution of our model. To address the irregularities geometry images and better match their distribution particularly the need to accurately reconstruct the discontinuities at the boundaries of the charts we add channel to represent the multi-chart mask and modify the loss function accordingly. Otherwise, we follow the VAE training procedure from StableDiffusion 1.5 (Rombach et al. 20.Tp, leaving out only the GAN and LPIPS losses (see details in appendix A). As shown fig. the reconstruction using this VAE reconstructs everything well, except for the missing connections between the charts.
3.2 COLLABORATIVE CONTROL
To leverage the prior knowledge encoded in existing 2D Text-to-Imen Image models, we use the Collabora- tive Control approach (Vainer al. 2024). As shown in fig. 2 this approach comprises two parallel networks: a pre-trained RGB model and a new model for the geometry image. The former respon- sible for generating UV-space albedo textures, while the latter generates the geometry images. These two models are connected by a simple linear cross-network communication layer, which allows them share information and collaborate generating pixel-aligned outputs across these different modal- ities. Crucially, this also enables the geometry model to influence the frozen model, guiding it to generate UV-space textures that would otherwise lie at the fringes of its training distribution. The frozen base model also drastically reduces the amount of data required to train the joint model while retaining generalizability, diversity, and quality (Vainer et al. 2024).
3.3
DATA HANDLING
3.3.1 DATASET
We train our model on the Objaverse dataset (Deitke al. 2022). We curate this dataset to include only objects with both high-quality structures and semantically meaningful UV maps by filtering out 3D scans and low-poly models. The final dataset contains approximately 100,000 objects. Each data entry is accompanied by captions provided by Cap3D (Luo et al. 2023) and Hong et al. (2024). During training, we randomly sample these captions and apply random rotations 90, 180, or 270 degrees to the extracted texture atlases. We now discuss how to transform these meshes into geometry images and back: the entire pre-psicg was performed on consumer-gren grade PC hardware (AMI) Ryzen 9 7950X, GeForce RTX 3090, 64 GB RAM) and took approximately 20 hours.
3.3.2 GEOMETRY IMAGE CREATION
To create Multi-Chart Geometry Images for 3D meshes, their existing UV maps. UV mapping defined the mapping p: V - [0, 1]2 where V is the set of vertices in a aD mesh, and p maps each
5

---

Published as a conference paper at ICLR 2025
vertex to UV coordinates. Note that this mapping is not injective (multiple vertices can be mapped to the same UV coordinates), nor is it a simple function of the vertex positions in R2, as modern mesh formats allow multiple vertices with distinct UV coordinates at the same 3D location. These issues mean that is not invertible, which would be a simple way to create a geometry image function. However, we argue that p is locally invertible and propose constructing a Multi-Chart Geometry Image based on the individually invertible areas of the available UV mapping. As mentioned before, the charts in a UV atlas tend to be semantically meaningful, so we aim to preserve them.
We begin by identifying the connected components of the mesh, which provides an initial separation into charts. Within each component, we identify two situations where p is not invertible: duplicated vertices with distinct UV coordinates, and a "crease", i... a line where the UV coordinates change direction (similar to a "mirror" boundary condition). The former is straightforward to detect, as we can find duplicated vertex positions with different UV coordinates. The second case is identified by creating a heatmap of the UV-space access pattern and detecting local minima, which indicate an indexing pattern that "doubles back" on itself. We then further split the individual charts along any detected creases. In line with the desirable mapping properties discussed in (Sinha et al., 2016), we adjust the geometry image mapping approximate an equal-area projection by rescaling each 2D chart with respect to the area of the corresponding surface.
In cases where only partial UV mapping is available, we use XAtlas (Young) 2022 the missing regions. However, since the XAtlas parametrization is of lower quality and lacks semantic properties, exclude meshes where less than 80% of the surface area has been unwrapped manually. This simple heuristic allows us to construct multi-chart geometry images for nearly all training examples. However, this method does not account for all possible degenerate cases of p. Therefore, we verify that the constructed is injective and skip any training meshes where this assumption is violated.
UV-unwrap
In Objaverse (Deitke et al., 2022), the objects are not aligned in canonical way. Although most shapes are oriented with the Y-axis pointing upward, there is no fixed front view, leading to rotational ambiguity between the X- and Z-axes. To resolve this ambiguity and confine it to a single coordinate,
we leverage cylindrical coordinates (r, 0 Φ) so that the ambiguity is restricted to the azimuth angle A.
3.3.3 MESH EXTRACTION
As mentioned in section 3.1 geometry images implicitly encode connectivity information of the mesh by treating neighboring pixels as con- nected in 3D space, i.e. quad mesh. However, to convert them into more widely supported tri- angular mesh, we need specify exactly how to triangulate the quads. For this, we closely follow the algorithm from Sander et al. (2003). For any 2 x 2 block of pixels in the GIM, we create up to two triangles depending on how many of the pixels describe part of the object. If necessary, the quad is split along its shorter diagonal. As shown in fig. 4 and in line with our goal of area- trea-preserving mapping (Sinha et al.) 2016), the result- ing triangulation is nearly uniform over the surface, which may or may not be desirable for specific applications. With our model's working resolution of 768 x 768, our GIMs can encode meshes with up to 589, 824 vertices. We consider the generation of arbitrary topologies or mesh generation with polygon constraints a promising area of future work for GIMDiffusion.
Figure 4: The resulting triangulation of our gener- ated objects is nearly uniform across the surface, due to the area-preserving nature of the geometry images in our training dataset.
3.4 TRAINING
For the frozen base model, we used a zero-terminal-SNR (Lin et al. 2024) StableDiifusson v2.1 (Rombach et al. (02I) as the base text-to-Image model, which remains frozen and generates albedo textures. The geometry model is an architectural clone with the cross-attention layers omitted and is trained from scratch, along with the cross-nect network communication layers. Initially, we trained the model at 256 x 256 resolution for 250,000 steps with a batch size of 384, and then at the final output resolution of 768 x 768 for a total of 100,000 steps with a batch size of 64. All stages of training were conducted with learning rate of 35 on 8 A 100 GPUs.
version fine-tuned from

---

Published as a conference paper at ICLR 2025
An organic alien gun is style of H.R. Giger
Wooden chest with Van Gogh's 's Starry Night
A steampunk airplane
An avocado-shaped chair
A steampunk gun
Figure GIMDiffusion generalizes well beyond the training data, successfully following more outlandish prompts.
4 RESULTS
the results of our method on set of text prompts, generating objects in manner that aligns with common queries, such as those in gaming workflow. The objects are well-defined and can relit from any direction, as the albedo textures are free from lighting- related artifacts. The generated UV layouts closely resem- ble those created by artists, offering editable assets.
Albedo
Textued
In fig. 6 we highlight the method's ability to generate meaningful variations when the seed prompt is per- turbed. GIMDiffusion produces significantly different UV layouts with even slight variations in the seed prompt, enabling diverse object creation. Leveraging the rich nat- ural image prior, our model generalizes well beyond the initial 3D dataset (see appendix E for the layout novelty study). Further examples illustrating this
Prompt: A fly agaric mushroom
Figure 6: Sample diversity of GIMDiffu- sion for different random seeds.
generalization are shown fig.
4.1 SEPARABILITY AND INTERNAL STRUCTURE
key advantage of our method its ability to generate objects divided into distinct semantic (or nearly semantic) parts, as shown in fig. / making the generated objects more suitable for editing and animation. This capabil- ity arises from the multi-chart representation design and the semantic information embedded in the training data through handcrafted UV maps, which loosely correspond to the semantic components of objects. This approach also allows users to easily correct imperfections, such as misaligned parts or extraneous geometry, and even com- bine different parts from multiple generations. Addition- ally, our method generates internal structures, such as the filament inside light bulb or the interior of a fish tank, because geometry images represent the entire object holistically, not just its visible surfaces.
Albedo texture
Exploded view
Figure 7: An exploded view of the gen- erated "Fly agaric mushroom" shows se- mantically meaningful chart separation.
4.2 IPADAPTER COMPATIBILITY
Major efforts have been made to achieve style control in diffusion models (Ye et al. (023). and our method is com- patible with these techniques. Since we leverage frozen base model to generate the albedo textures, we can ap- ply a pre-trained IPAdapter and produce stylized output meshes, as shown in fig. 8 Despite the significant mis- match between the application domains (natural images versus albedo atlases), the style guidance remains success- ful. However, we find that this approach starts to break when the text prompt deviates too much structurally from
IPAdapter input
<Noo prompt>>
kingsize bed
Figure 8: We guide the reverse pro- cess by applying a pre-trained IPAdapter to the frozen base model (Ye al. (023).
7

---

Published as a conference paper at ICLR 2025
the content of the conditioning image. We attribute this to the fact that IPAdapter aims leverage every aspect of the conditioning image; but e.g. IPAdapterlnstruct (Rowles et al. 2024) offers a selec- tive extraction of just appearance style without entangling structural information.
4.3
EVALUATION
An antique gold A pair of polka- A cactus with pocker watch dotted sneakers pink flowers
An old, frayed straw hat
worn-on
leather
briefease
MVDream
Greaser
LGM
CRM
SF3D
Ours
Figure 9: The qualitative evaluation on prompts from Bench (He al. 2023). Note that our model produces well-defined shapes and detailed appearances due to high- nigh-resolltion of underlying geometry images. The only noticeable defects are visible cracks and minor misplacements small parts, which we aim to address in future work (see section 4.5)
We selected several recent works with avail- able source code for comparison, including both
A wore out leather An old, frayed hot
optimization-based and feed-forward methods.
Specifically, we chose GaussianDreamer (Yi
al. 2023), DreamGaussian (Tang al. [203))
MVDream (Shi et al.) 2023b). LGM (Tang et al.
2024). CRM (Wang et al. 2024) and SF3D (Boss)
2024). We adapted state-of-the-arr Image- al.
briefease
to-3D models LGM, CRM and SF3D for Text-to-
3D by integrating the same Text-to-Image model
used in our approach (see more details on eval- uation in appendix C
CRM
SF3D
Qurs
We treated the 3D mesh as the final outpu[] and included post-processing time in the total gener- ation time, using default settings for all methods. For Image-to-3D models, we also accounted for
Figure 10: Our method generates layouts of higher quality than parametrization algorithms used in other methods.
GGaussianDrena Dreamer outputs Gaussian splats, which converted using the method provided by LGM for evaluation
8

---

Published as a conference paper at ICLR 2025
Table 1: Evaluations
Method
Time
UV-unwrapping
Relightable
Separable
7 T
4
4 (
Greasian
minutes
No
No
No
26.7
MVDream
DreanGiussian
minutes
XAtlas
No
No
19.8
LGM
1.8 minutes
XAtlas
No
No
29.9
0.068
0.081
0.060
CRM
sce
XAIlss
No
39.6
0.012
0.017
0.013
SF3D
3.5 see
Cube projection
Yes
No
48.2
0.112
0.117
0.122
10
Generative
Yes
0.038
0.042
0.035
the 3-second generation time required by our base model to create single image. All evaluations
were performed on single A A100 GPU.
We used T Bench (He et al. 2023) for automatic evaluation, measuring both generation quality and prompt alignment of the resulting meshes. BeBnh includes three prompt sets: Single Object, Single Object with Surroundings, and Multiple Objects. Since our model specifically designed for single-object generation, we focused evaluation on that track. We used the code provided in the Be repository for comparisons. Overall, we evaluated 100 prompts (see appendix D). Visual comparisons provided in fig.
Bench evaluates only multi-view renderings, it does not assess the underlying mesh quality. To address this, we followed[Shen et al. (2023) and calculated metrics such as the percentage of triangle aspect ratios > 4 (AR>4( %) )), radius ratios > 4 (RR>4( )),,, and minimum angles < 10 (MA<10( % )), using Py Vista (Sullivan & Kaszynski 2019) for all feed-forward methods. While these intrinsic metrics cannot guarantee overall quality, they provide reasonable proxy for some applications.
As shown in table T our method achieves competitive results compared to state-of-the-ar models, while providing superior editability via separable parts and producing nearly artistic UV maps. Unlike other approaches that rely on vertex colors or pre-defined fixed unwrapping, our method natively generates U UV-space textures. This results in more logical chart separation, better area utilization, and sharper details compared to other methods, as demonstrated fig. TO
4.4 ABLATION STUDY
In this section, we validate our method by ablating key design choices and evaluating their impact. Specifically, we ablated three aspects of our approach: the absence of cross-attention layers in the geometry image branch, the use of the Collaborative Control mechanism, and the
Ours
with croo--at alL.
cylindrical coordinate transform. To ensure consistency, all experiments used the same batch size, number steps, and hardware.
First, we reintroduced cross-attention layers in the geom- etry image branch, making it a full architectural clone of the frozen branch. As shown in the second row of fig. II] while the model still follows the prompts, its generaliza- tion ability significantly deteriorates, especially with out- of-distribution prompts like "a steampunk gun", which is consistent with the observations in Vainer et al. (2024).
wil Ciallab. comoo
Next, we removed the Collaborative Control mechanism and fine-tuned the base model to generate both albedo textures and geometry images, concatenating their latents. This resulted in a significant drop in both shape and tex- ture quality, as seen in the third row of fig. II
steampunk gim
red fire hydrant
A metal toaster
Figure 11: The first shows full method's performance, with subsequent rows removing key design elements.
Finally, we ablated cylindrical coordinates by fine- tuning the model as before but omitting the coordinate transformation. Without this step, the model struggled to
generate even basic shapes.

---

Published as a conference paper at ICLR 2025
4.5 LIMITATIONS
Our current method has certain limitations. The most common issue is the appearance of visible cracks in the generated meshes. These artifacts result from two key factors: Multi-Char Geometry Images lack inter-chart connectivity information, and irregular chart boundaries are prone to undersampling (see fig. [2), as analyzed by (Snnder et al. (2003). The original paper proposed zippering algorithm to close the cracks between adjacent charts, but this approach assumes the watertightness of the underlying mesh, which is not generally true for real- world meshes. In future work, we aim to generalize and adapt the zippering algorithm. Sampling-related modifications, such as those proposed by Gauthier & Poulin (aainn((000))) (2009); Yan et al. (024). could further improve the results.
The blue area repre- sents a chart, undersampling at insufficient resolutions can af- fect chart boundaries. Affected pixels are highlighted in red.
Figure 12:
Additionally, we believe this issue is exacerbated by the VAE's latent compression. Areas smaller than x 8 pixels fall below the VAE's latent resolution, leading to visual artifacts such as noisy edges and misaligned charts. We hypothesize that integrating wavelets into the VAE, as suggested by Sadat et al. (2024), could mitigate these issues.
Another limitation stems from the discrepancy between the natural image prior and the domain of our model. Although individual charts in a geometry image can be arbitrarily rotated and still represent the same 3D object, the frozen base model is not rotationally equivariant (Weiler al. 2021). Human faces serve as a good example - while typically upright in natural images, they can appear in random orientations on texture maps, leading varying quality for such prompts (see examples in appendix B). In future work, we plan to investigate ways to establish canonical orientations for these charts, ensuring better alignment with the diffusion prior.
Additionally, we sometimes observe that the model duplicates parts of the object, which can result in visual artifacts in the generated meshes due z-fighting. However, the segmented nature of the generated objects makes these artifacts easy to resolve manually.
DISCUSSION AND FUTURE WORK
In this work, we present Geometry Image Diffusion (GIMDiffusion)) a novel Text-to-3D generation paradigm that utilizes geometry images its core 3D representation in combination with powerful natural image priors in the form of pre-trained diffusion models. Our results show that GIMDiffusion can generate relightable 3D assets with high-quality UV maps efficiently Image methods generate normal images, while avoiding the need for complex, custom 3D-aware architectures. We believe that our research lays the groundwork for new direction in Text-to-3D generation.
existing Text-to-
Future improvements include eliminating visible cracks, achieving better alignment with 2D priors, and enhancing the precision of generations. Incorporating topology prediction and conditioning on specific polygon budgets could provide greater control over the generated 3D objects, making them more practical for gaming and other graphics pipelines. Furthermore, the inherent capability of GIMDiffusion to model both geometry and parameterization makes it particularly well-suited to addressing the longstanding challenge of UV mapping creation for existing geometries. Equally promising is the potential of GIMDiffusion in related fields such as interactive editing, animation and Text-to-Vided generation.
ACKNOWLEDGMENTS
We would like to thank Shimon Vainer from Unity Technologies, Dr. Lev Melnikovsky from the Weizmann Institute and Alexander Demidko for their valuable feedback and insightful discussions. Konstantin Kutsy has been invaluable in helping us operate training infrastructure.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.087744 2023.
Hassan Abu Alhaija, Alara Dirik, Andr'e Knorig, Sanja Fidler, and Maria Shugrina. Xdgan: Multi- modal 3d shape generation in 2d space. In British Machine Vision Conference, 2022. URL
wwtp/oorggsoo/..e.. https://api. www.sttieeennteeeeeee. ottp://ccccccca..ooooooororpooaa 2552733885
Thiemo Alldieck, Nikos Kolotouros, and Cristian Sminchisescu. Score distillation sampling with learned manifold corrective. ArXiv, abs/2401.0522 .05293, 2024.
URL https://api
semant wwtpsccholaa..rg/oooooooooooo ID: :66900922901
Raphael Bensadoun, Tom Monnier, Yanir Kleiman, Filippos Kokkinos, Yawar Siddiqui, Mahen- dra Kariya, Omri Harosh, Roman Shapovalov, Benjamin Graham, Emilien Garreau, Animesh Karnewar, Ang Cao, Idan Azuri, lurüi Makarov, Bric-Tuan Le, Antoine Toisoul, David Novomy, Oran Gafni, Natalia Neverova, and Andrea Vedaldi. Meta 3d gen, 2024. htth:tt:: //axxiv org //55/bb//50 2407. 02599
Mark Boss, Zixuan Huang, Aaryaman Vasishta, and Varun Jampani. Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement. 2024. URL https
//api. coamanntcsccolar.
Eric Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio
Gallo, Leonidas J. Guibas, Jonathan Tremblay, S. Khamis, Tero Karras, and Gordon Wetzstein.
Efficient geometry-aware 3d generative adversarial networks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16102-16112, 2021. URL https
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5932-5941, 2018. URL
https: www.steettttte icscholarr orgg/Crpuus :4457478
Jasmine Collins, Shubham Goel, Achleshwar Luthra, Leon I.. Xu, Kenan Deng, Xi Zhang, T. F. Y. Vicente, Himanshu Arora, T. L.. Dideriksen, Matthieu Guillaumin, and Jitendra Malik, Abo: Dataset and benchmarks for real-world 3d object understanding. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2t.p...1999.... 21094-21104, 2021. URL https
7/api. semant wwa.nnaaicccchhaaa.o wttp:/ooo/ooocce.3c3333333
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Lud- wig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog-
nition (CVPR), pp. 13142-13153, 2022. URL https::/ Corpus 2448585558
semanticscholes
Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE Transactions on Information and Systems. 74:214-224, 1991. URL
https://api. www.seeeannncchcceeeeee
60496605
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Yiqun Duan, Xianda Guo, and Zhengbiao Zhu. Diffusiondepthh Diffusion denoising approach for monocular depth estimation. ArXiv, abs/2303.05021. 2023. URL https://api
Bemanticscholar. wttp/ooooo//.../5/0101
Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transform- ers for high-resolution image synthesis. ArXiv, https/24030330066 2024. URL https::/app semant cscho ar. atr.orr/CCoruuust11226744

---

Published as a conference paper at ICLR 2025
Michael S Floater and Kai Hormann. Surface parameterization: a tutorial and survey. Advances in multiresolution for geometric modelling, pp. 157-186, 2005.
Mathieu Gauthier and Pierre Poulin. Preserving sharp edges in geometry images. In Graphics Interface, 2009. URL /api. semant wwwpaaanniiiccccoooooooooo htt.:hoo......oooooooooraaa00000
Xianfeng Gu, Steven J. Gortler, and Hugues Hoppe.
Geometry images. Proceedings of the 29th
annual conference on Computer graphics and interactive techniques, 2002. URL http::t/ps:/ppp
semanticscholase htt.:shh........ooooooooopaiaaaa
Yuze Hc, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. bench: Benchmarking current progress in text-to-3d generation. arXiv preprint arXiv:3310.22977.2 2023.
Lukas Höllein, Aljavz Bovzivc, Norman Muller, David Novotny, Hung- Yu Tseng, Christian Richardt, Michael Zollhofer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text- to-image models. ArXiv, abs/2403.01 2024. httpss//::/ap semant icsccholsal
org/CorpusID: 268998662222
Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang. Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. ArXiv, 0bs/22403002233 2024. URL httphhhtttsappt htttps://api.semattiiii.. semant icscholar
org/Corpus ID: 266227886666
Yicong Hong, Kai Zhang, Jiuxiang Gu. Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. ArXiv, abs/2311 1..4430, 2023. URL httpp:://// semant iesnnnlacccccla...... wwtp:eilaarrrrrooouusssseeee
265050698
Liucheng Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Con- sistent and controllable image-to-vider synthesis for character animation. ArXiv, abs/2311.171111 2023. URL http::ttt:///tttm http://.pppssmaa....mmmmmm swwant www.ettcscchhhhcceecc www.aar....rraooeooeeeeee wrw.ooooo//oooorrooooooo www.ccoooo.....oooooooo...aaaaa.
Zixuan Huang, Justin Johnson, Shoubhik Dehnath, James M. Rehg. and Chao- Wu. Pointinfinity:
Resolution-invariane point diffusion models. 2024. https:////ap ies.
org/CozpusID:2ee :685892269688999
Heewoo Jun and Alex Nichol.
Shap-e: Generating conditional 3d implicit functions.
ArXiv, abs/2305.02463, 2023. URL.
258480331
Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, S. Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin. Spad Spatially aware multiview diffusers. ArXiv, abs/2402.00235,, 2024. URL httthtttt:///pttt
httpttts:: www.pp.....at.a.mmmmammmmmmm semant icschot
Pot/CorpusID:267..0
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4405,4405, 2018. URL.httss://aag www.ppp...ssa.m..mm.aa...p.em..e swwanti www.ciccchhaa.....ooo www.aaar.rr//oooooouoooo 54482423
Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. ArXiv, abs/2310. 17590, 2023. URL, htt.htttp://api...
264490431
Bing Wen Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion- image generators for monocular depth estimation.
ArXiv, A20/2312.022155. 2023. URI. httpt:tt////ett http:/ppp..eaatiiiecccemo sewanticscholl.........c ar. org T 265609019
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42:1 14, 2023. URL httpttttss...... //api. semant htw.aattccccoooorrrro/oo/oooo oew.hhooaar..oogoooooooosa//323
12

---

Published as a conference paper at ICLR 2025
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. ArXiv, abs/2311. 1284, 2023. URL BtRphttss:20 api semant www.mmaannsccchaar.......maa htt.:///.........o.oooo.aaaaaaa.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5404-541 1100-, 2024.
Yaron Lipman, Ricky T Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. October 2022.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3 Zero-shot one image to 3d object. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9264-9275, 2023. BRLLhttps:/ httpss:/api...... /api..pppp.p/p semanticscholat
org/Corpus ID: 257631 738
William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. Proceedings of the 14th ammual conference on Computer graphics and interactive tech- niques, 1987. URL AtRtsstss: http:/ppp...mmanncccccccc.oeoaea http://oo..aatoooooooooriaa.aaa.
Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 17900- 17910, 2023. URL hrttpsts://app www.ste wwwcchclaaaaeo..coocco wwwp:/arrr.rr//oouuussoo ID 259145352
semant
Tiange Luo, Chris Rockwell, Honglak Lec, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.0722 077797 2023.
Ravi Malladi, James A. Sethian, and Baba C. Vemuri. Shape modeling with front propagation: level set approach. IEEE Trans. Pattern Anal. Mach. Intell., 17:158-175, 1995. URL https:
stpp...senattcccccool..... org/ Cttp://rrppu:DD::i:e 95001100
Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4455-4465, 2018. URL https
7/api. iepp..seeanticcchooolo
otgg/oorpusD::544555..
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf. Communications of the ACM, 65:99 106, 2020. URL http::tts://pppp
semanticson semanticscholer wtw.scooolaa.oog/ooorsssssa.o :t1.:oppps10::2115559797 2211:25911
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. ArXiv, abs/2212.08851,, 2022. URL
htttps://pp..stt semant cschol ar wwt.:orgg/ooooo TD :0:22552211
Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In International Confer- ence on Learning Representations, 2024. URL https: //t::///ap //ppi..pppp semant. wwsscholar.oo org/
Corpus ID:
Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high higo-resolution image synthesis. ArXiv, abs/2307 .01552, 2023. URL. htttps://app..... semant cschol ar. wwt./ooro//ooooooo
259341735
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ArXiv, abs/2209. 14988,, 2022. URL https: semanticschoiar. org/
CorpusTD:220 252::
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. BRLhts/:://pp
semant wwcholcchhoocoooooccooooocooccce ar www.choll..rroooooooooooooo Corp/srrrpsss ID: :4533::222888
Ciara Rowles, Shimon Vainer, Dante De Nigris, Slava Elizarov, Konstantin Kutsy, and Simon Donné, Ipadapter-instruet: Resolving ambiguity in image-based conditioning using instruct prompts, 2024.
URLhttss://axxx
wtg. abs/2408. 03209
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Litevae: Lightweight and efficient variational autoencoders for latent diffusion models. arXiv preprint
MrXiv:2405.14477, 2024.
Pedro V. Sander, Zoë Wood, Steven Gortler, John M. Snyder, and Hugues Hoppe. Multi- chart geometry images. In Eurographics Symposium Geometry Processing, 2003. URL
http://///ppi.ssmmtttcccccccii orw.eaannnncccccooo.oooooooooree : :899489
Rohan Sawhney and Keenan Crane. Boundary first flattening, ACM Transactions on Graphics (TOG), 37:1 - 14, 2017. URL https::/app. wtt.://atccccccoooooooooooo//rra
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jit- sev. Laion-5b: An open large-scale dataset for training next generation image-text models. ArXiv, wwtp/210.004/pppp 2022. URL httth::///e/ sew.niiccccccccc..........m wwt.ssschhhaaaaaoooooreeeee.e.ee 252917726
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeic, Sanja Fidler, Nicholas Sharp. and Jun Gao. Flexible isosurface extraction for gradient- based mesh optimization. ACM Trans. Graph., 42(4):37-1, 2023,
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero (rrol22++::: single image consistent multi-view diffusion base model. ArXiv, abs/2310. 2023a. Utthttps:://
semanticschol orq/CorpusID:
266436552
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and X. Yang. Mvdream: Multi- view diffusion for 3d generation. ArXiv, 16512, 2023b. URL https::/app
semant cccchlllllllcoooocccccccccccccca ar. wwwp/orpsssuuuoorooeeoooo 261 2221333233
Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Garreau, Oran Gafni, Natalia V. Neverova, Andrea Vedaldi, Roman Shapovalov, and David Novotny, Meta 3d assetgen: Text-to- mo-m generation with high-quality geometry, texture, and phr materials. ArXiv, abs/2407.02245,, 2024. URI. httph:ttt://aat semant cscho ar www. Corg/Sr 270878069
Ayan Sinha, Jing Bai, and Karthik Ramani. Deep learning 3d shape surfaces using geom- etry images. In European Conference on Computer Vision, 2016. URL https::/app
semanticschol.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. March 2015.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. score-based generative modeling through stochastic differential equations. ArXiv, Abs/2011.13455 2020. URL httpt:t//p/pp semant cscho wachhooorr..ooo www.oo/r/o/ooooloo 221209335
Pratul P. Srinivasan, Stephan J. Garbin, Dor Verbin, Jonathan T. Barron, and Ben Mildenhall. Nuvo: Neural uv mapping for unruly 3d representations. ArXiv, abs/2312.05283,, 2023. URL https://apii semanticscho ar. at..org//oorpuss11:2611
14

---

Published as a conference paper at ICLR 2025
C. Bane Sullivan and Alexander Kaszynski. PyVista: 3d plotting and mesh analysis through streamlined interface for the visualization toolkit (VTK), Journal of Open Source Software, 4(37): 1450, may 2019. doi: 10.21105/joss.04400. URL 01450
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 2023.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. ArXiv, abs/2402.05054, 2024.
URL https: Semant cschhlac. org/ http//Crpuusli ID:
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. XrXiv, abs/2403.02151, 2024. WRI.https://eet semant iesemnntiscchi. ccscho a
org/Corpus ID: 4824
Shimon Vainer, Mark Boss, Mathias Parger, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Nicolas Perony, and Simon Donn's e. Collaborative control for geometry pbr image generation. ArXiv, abs/2402.05555 009919, 2024. URL http:::://appp..aaaaasaee.eeeeee 267547446
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017. lktts://///p
www.app..seemammm semant http:///.aaaaaaaaaaoooaoooooooar
Haochen Wang, Xiaodan Du Jiahao Li, Raymond A. Ych, and Gregory Shakhnarovich. Score jacobian chaiming: Lifting pretrained 2d diffusion models for 3d generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12619-12629, 2022. URL https: www.eeeiincnoo.....oeeee wescholar......cccccoccccoccca.e org/ Corpus ID: 254125253
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan LI, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processsing Systems, volume 36, pp. 8406-8441. Curran Asso- ciates, Inc., 2023. htL wttp::tppp./ppppeeeei........opp 2023/file/lai 222222222770031133355500-020eee2
Zhengyi Wang, Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403. 240.024403. 2024.
Maurice Weiler, Patrick Forr'e, Erik P. Verlinde, and Max Welling, Coordinate independent con- volutional networks isometry and gauge equivariant convolutions on riemannian manifolds. ArXiv, abs/2106.060000 2021.
http:/sshhhhaaaaeeoorooeeeeeeeee org/CorpusiD:
235417099
Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang. Consistent3d: Towards consis- tent high-fidelity text-to-3d generation with deterministic sampling prior. ArXiv, abs/2401.00 1000000, 2024. httpa///eeee/pppaaaaaaaaaaaaaaaa wwwpaaattttccccoooooooooooroooe
Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, and Xiaohui Zeng. Latte3d: Large-scale amortized text-to-enhanced3d synthesis. ArXiv, abs/2403. 15385, 2024. URL https: http::::://pp....aaaattaeeaaaeee www.nticsccooo....... www.://aaaarrgooousosss.a
268667414
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 41, 2021. htLLhtts::
semant www.moiicccooaaaa..........
org/CorpusID: 244478494
Jaazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
Xingguang Yan, Han-Hung Lee, Ziyu Wan, and Angel X. Chang. An object is worth 64x64 pixels: Generating 3d object via image diffusion. 2024. URL setttps://api.smmatiiccc
org/CorpusID:2700 719920
Lior Yariv, Omri Puny, Natalia V. Neverova, Oran Gafni, and Yaron Lipman. Mosaic-sdf for 3d generative models. ArXiv, abs/2312.09222,, 2023. httLhttts:/sstt semant icscho ar
Stg/CorzpusDD22622000
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672. 2023.
Taoran Yi, Jiemin Fang. Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:2310.08529, 2023.
Jonathan Young. Xatlas: Mesh parameterization / uv unwrapping library.
https://githu/
com/ htc///pccy//// katlaa 2022.
Denis Zavadski, Johann-Friedrich Feiden, and Carsten Rother. Controlnet-xs: Designing an efficient
and effective architecture for controlling text-to-image diffusion models. ArXiv, abs/2312.06573, 2023. /ttp///pps//ppppeeeaatttttttteea wtw.:eescsmaatcccccccccso.. wwtpollaa..rr/g/ooooooooooooo ott.aaaata..ataaaaaaaooooooo.aaa 266pl662:266666999
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. ArXiv, abs/2210.06978, 2022.
URL httlhttts:/aapp semant eschol a org ott.::oo.......ooooooooott////11
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3813-
3824, 2023. URL htttps:/p// semant wwwchlar..r....ooeo.coo.. orw.ooruusuorooooooooooooou rorcoolar.orr///rrro ID: 256827727
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Chuanxia Zheng and Andrea Vedaldi. Free3d: Consistent novel view synthesis without 3d repre- sentation. ArXiv, abs/2312.04551, 2023. URL hhttps://aapp sew.ntccsccooooa.oooooooo
CorpusID:2660522
Xin Zheng, Yang Liu, Peng-Shuai Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based style- gan for 3d shape generation. Computer Graphics Forum, 41, 2022. URL https://app
semanticschol.s org//oorpa111:250000000
Junzhe Zhu, Peiye Zhuang, and Oluwasammi Koyejo. Hifa: High-fidelit text-to-3d generation with advanced diffusion guidance. In International Conference Learning Representations, 2023.
URL http::tp///// //tttps://pii. sew.nm/ecchhllla...cooooppsssspp
16

---

Published as a conference paper at ICLR 2025
A VAE DESIGN AND TRAINING DETAILS
SD VAE reconstruction
GIM VAE reconstruction
Figure 13: Comparison of geometry reconstruction between the SD VAE and our GIM VAE. The SD VAE reconstruction could not be meaningfully triangulated, point cloud visualized instead of mesh
The Stable Diffusion Variational Autoencoder (SD VAE) is trained on natural images and achieves high reconstruction quality in that domain. However, it does not transfer well the domain of geometry images.
The first issue that the reconstructed images contain considerable amount of noise (see fig. 13) While these perturbations may be imperceptible in the image domain. they cause significant problems the 3D domain, such as the loss of fine geometrie details. We hypothesize that this behavior arises from the LPIPS (Zhang et al., 2018) and GAN losses used during training, as GANs are known for introducing high-frequency noise into images.
The second issue is the color range bias of the SD VAE, which favors the most common colors in natural images. When applied points in 3D space, this bias can cause undesirable warping and other artifacts.
Lastly, the original VAE struggles to accu- rately reconstruct discontinuities at chart bound- aries, which is critical for triangulation (see sec- tion 3.3.3) Convolutional layers tend to interpo- late across boundaries, leading to false geometry due to the creation of smooth transitions instead of sharp ones.
Table 2: VAE evaluations
Method
Latent dim
PSNR ?
SD VAE
96 x 96 x 4
31.87 49.02
GIM VAE
96 x 96 8
To address these issues, we made the following design choices:
Disentangling chart boundaries from geometry: We introduced additional input and output channels for multi-chart mask. This mask be thresholded to model sharp boundaries, avoiding interpolation artifacts.
Removing GAN and perceptual losses: These were removed to prevent noise from being introduced during training.
Increasing latent dimension: We expanded the latent dimension to 8 to better capture geometric details.
17

---

Published as a conference paper at ICLR 2025
We trained our VAE following the procedure and codebase of Stable Diffusion (Rombach et al. 2021]. The model was trained on 8 A100 GPUs with a batch size of 128 for 100k steps. The PSNR results reported in table 2
B FAILURE CASES
Charts in the Geometry Image layout can be randomly rotated without affecting the resulting 3D shape, as this rotation alters only the parameterization. However, GIMDiffusion relies on Stable Diffusion 2.1 prior, which is not rotationally equivariant and exhibits strong orientational biases for certain features. For example, it struggles to reliably generate an upside-down face. Consequently, the quality of generated outputs may degrade for certain prompts. With the prompt "human head", for instance, the quality can vary depending on the seed, particularly when the chart containing the face oriented unfavorably.
Fortunately, GIMDiffusion's fast inference speed enables efficient seed exploration to achieve the desired quality. In future work, aim introduce canonical orientations for charts mitigate this issue.
Prompt: human head
Figure 14: Generation results for human head for three different seeds.
C EVALUATION DETAILS
Be HHe et al. 2023) evaluates both quality and aligmment in text-to-3D generation. Quality is assessed through standardized multi-view rendering pipeline provided by the authors. This process generates 2D images of the 3D object from various angles and uses scoring models (Radford et al.) 2021 Xu et al., 2024) to evaluate overall visual consistency and fidelity. To spot issues such as view inconsistency (e.g., the Janus problem), a regional convolution technique is applied. Alignment measures the semantic consistency between the generated 3D object and the input text prompts using a multi-view captioning process coupled with GPT-4 (Achiam et al. 2023) evaluation. The final score computed as the average of the quality and alignment metrics.
To compare our method with state-of-the-at art image-to-3D approaches, we generated a set of images based on the Bench evaluation prompts (see appendix D) using the same base model method (Stable Diffusion 2. For compatibility with image-to-3D methods, we appended the phrase "single object, uniform background" each prompt. For each prompt, three images were generated, and the
18

---

Published as a conference paper at ICLR 2025
best aligning with the text and depicting the object from natural angles manually selected. Figure [ showcases several examples.
A bright red fire hydrant
A ceramic teapot with floral patterns
leather briefcase
A crumpled silver aluminum soda
A cherry red vintage lipstick tube
plush velvet armchair
An old, frayed straw hat
A sleek stainless steel teapot
Figure 15: Examples of images used to condition Image-to-3D models during evaluation.
D EVALUATION PROMPTS
cactus with pink flowers
31. A partly broken shell of a tortoise
2. A rainbow-colored umbrella
32. A long woolen scarf, striped red and black
An antique wooden rocking horse
33. A tattered old explorer's 's map
4. A golden retriever with blue bowtie
34. A well-used black iron frying pan
5. An ivory candlestick holder pair of polka-dotted sneakers steaming mug hot chocolate with whipped cream
35. A crumpled silver aluminum soda can
36. A thick, green-spined book with yellowed pages
37. A shimmering emerald pendant necklace
8. A bright red fire hydrant
38. A well-worn straw sun hat
gleaming silver saxophone
39. tarnished silver letter opener
10. A leather-bound book with gold details
40. An antique glass perfume bottle
11. vibrant sunflower with green leaves
41. polished mahogany grand piano
12. A castle-shaped sandcastle
42. A dented brass trumpet
13. neon green skateboard with black wheels
43. A pristine white wedding gown
14. A pirate flag with skull and crossbones
44. A chipped porcelain teacup
15. plush teddy bear with a satin bow
45. A rustic wrought-iron candle holder
16. ripe watermelon sliced in half
46. A vibrant, handmade patchwork quilt
17. sparkling diamond ring in a velvet box
47. A plush velvet armchair
18. vintage porcelain doll with frilly dress
48. A sleek, black top hat
19. chameleon perched on a tree branch
49. A paint-splattered easel
20. tarnished brass pocket watch
50. A bent steel crowbar
21. A ceramic teapot with floral patterns
51. A crisp paper airplane
22. antique ruby-studded brooch
52. worn-out rubber tire swing
23. A simple burgundy colored feather quill
53. An intricately-carve wooden chess set
24. vintage iron-cast typewriter
54. bright red kite with a frayed tail
25. A shiny emerald green beetle
55. A smooth, round opal stone
26. crystal glass paperweight with abstract de- sign
56. A rusty, vintage metal key
57. A delicate, handmade lace doily
27. velvet cushion stitched with golden threads
58. A sturdy mahogany walking cane
28. small porcelain white rabbit figurine
59. A sparkling crystal chandelier
29. left-handed electric guitar painted black
60. A worn-out red flannel shirt
30. bright blue plastic swimming goggles
61. A cracked porcelain doll's face
19

---

Published as a conference paper at ICLR 2025
62. A dusty classic typewriter
82. A scuffed up soccer ball
63. glossy grand black piano
83. A pair of worm-nn blue jeans
64. A faux-fur leopard print hat
84. A well-loved stuffed teddy bear
65. futuristic, sleek electric car model
85. A chipped, white coffee mug
66. cherry red vintage lipstick tube
86. A bright, yellow rubber duck
67. A cobweb-covered old wooden chest
87. A sleek stainless steel teapot
68. gold glittery carnival mask
88. A water-streaked glass window pane
69. A tattered world map with stained edges
89. An intricate ceramic vase with peonies painted on
70. shiny red apple
71. A worn-out leather briefcase
90. A fuzzy pink flamingo lawn ornament
72. An antique gold pocket watch
91. blooming potted orchid with purple flowers
73. A sleek, slim smartphone
92. An old bronze ship's wheel
74. wet, vibrant beach ball
93. A sparkling diamond tiara
75. A rusty, abandoned bicycle
94. A vintage plaid woolen blanket
76. A fluffy, orange cat
95. A pair of shiny black patent leather shoes
77. Crisp, folded origami paper
96. An elegant feather-quill ink pen
78. A shiny, new electric guitar
97. A fragrant pine Christmas wreath
79. weather-beaten wooden bat
98. A silver mirror with ornate detailing
80. A delicate crystal champagne flute
99. A green enameled watering can
81. An old, frayed straw hat
100. classic leatherette radio with dials
E LAYOUT NOVELTY
iN
Render
Texture
Layout mask
Layout mask
Texture
Render
Generated objects
Ncarest neighbors
Figure 16: Each row displays generated object alongside its nearest neighbor by layout mask from the dataset. The nearest UV layouts differ significantly from the corresponding generated ones, often representing entirely different object types.
To demonstrate our model's ability to generate novel UV layouts not present in the training data, we visualize dataset entries that are closest to the generated samples in terms of layout similarity. For this purpose, we constructed an index of binary multi-chart masks extracted from all geometry images in our dataset using the Faiss library (Douze et al. 2024). We then queried this index to find the nearest neighbors for the generated samples.
During the training phase of our model, the input data were augmented with random rotations by angles that are multiples 90 degrees. Accordingly, for each query image, we computed the nearest neighbors across four rotations and selected the neighbor with the lowest Euclidean distance.
The visualizations in fig. reveal that there are non-tin con-trivial differences between the generated layouts and those present in the training data, highlighting the model's capability for generating novel layouts.
20


Reviews:
[{"review_id": "lKbzYJto4Q", "reviewer": "Reviewer_Vjsj", "summary": "This paper studies how to generate high quality mesh of objects with GIM.\n\nThe 3D objects are parameterized into 2D chart geometry image. This representation is compact where the prior of 2D SD model can be exploited and maintains good geometry and semantic properties. The generation happens vis modifying standard SD with cross attention between different attributes of the chart.", "questions": "Please see the first two questions in the weakness: the chart quality and diversity; and the artifacts on the boundary.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The reviewer loves the elegant idea very much. I personally believe this is the correct way in a long-term to generate usable (industry level) and high-quality assets for objects. Because most generative model for objects now depends on large synthetic dataset, with known geometry.\n- The system is simple and effective.\n- Also, the semantically meaningful parameterization has much more space in the future for editing and manipulation.", "weaknesses": "- My main concern is maybe the diffusion overfit the proposed UV chart distribution. But we know that there is uncertainty of how an object can be parameterized into a chart. Is there any measurement of how good the chart is in the proposed preprocessing pipeline? e.g. it maintains the surface area or local differential properties? Can the proposed method generate different chart pattern for the exact same object? And how diverse is the chart?\n- There are some small artifacts in the boundary of the mesh, which may due to the boundary of the chart. I believe this can be fixed in future works, but an analysis of this artifacts may provide more intuition of the future work.\n- Maybe one or two applications like relighting .etc will future demonstrate the strength of generating object via GIM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4QEm2sv4B", "reviewer": "Reviewer_nxjJ", "summary": "The paper introduces Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D generation model that uses geometry images to represent 3D shapes in a 2D format. This approach leverages the priors of pre-trained image diffusion models, allowing efficient 3D object generation without complex 3D-aware architectures. By retaining pre-trained weights, GIMDiffusion is compatible with tools like IP-Adapter, enabling flexible style control and broad applicability. The use of multi-chart geometry images further enhances the model’s ability to represent complex shapes, offering a scalable and versatile solution for text-driven 3D generation.", "questions": "1. How does the variability in multi-chart geometry image layouts impact GIMDiffusion’s training stability and scalability?\n2. The quality of shape edges appears to be suboptimal in the generated results. Is this a known limitation of the geometry image representation, and are there potential solutions to address this?\n3. The authors freeze the albedo generation branch based on the pre-trained SD model, but SD’s performance in UV space is often limited. Could the authors provide experimental evidence to show that this decision does not compromise the quality of generated RGB textures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Geometry images provide a low-complexity 3D representation, allowing GIMDiffusion to fully leverage pre-trained 2D diffusion models for 3D generation.\n2. The model preserves the weights of the pre-trained image diffusion model, making it compatible with plugins like IP-Adapter for style transfer and customization.", "weaknesses": "1. My primary concern with multi-chart geometry images is the variable layout configurations that a single 3D shape can produce. That means a single 3D shape may correspond to many different geometry images, potentially complicating the training process and impacting consistency across different shapes. Is any technique used to standardize the conversion from mesh to geometry image.\n2. Generated results in the paper often show lower quality around shape edges, possibly due to the geometry image representation. This limitation affects the clarity and sharpness of complex geometries.\n3. The model freezes the pre-trained Stable Diffusion (SD) model for generating albedo textures. However, SD has known limitations in UV space, and freezing this branch could compromise RGB quality. Providing experiments to validate this decision would help substantiate the model’s robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyctPvOY5A", "reviewer": "Reviewer_zGeZ", "summary": "1. GIMDiffusion uses geometry images to represent the surface of 3D shapes in a 2D format. This method, unlike traditional 3D meshes, can utilize the extensive 2D image generation capabilities of diffusion models like Stable Diffusion. This approach results in faster generation times, lower data requirements, and the elimination of the need for complex 3D-specific model architectures, which are often cumbersome and data-hungry.\n\n2. The model introduces a Collaborative Control mechanism to integrate 2D Text-to-Image models with geometry images. Through this mechanism, the geometry and texture generation processes are interconnected, allowing each to influence the other. This setup ensures that the textures generated align well with the 3D shape, facilitating realistic and coherent representations even when there is limited 3D data available.\n\n3. Efficiency and Performance: GIMDiffusion achieves faster-than-average generation times, producing complex 3D assets in approximately 10 seconds. Moreover, the model avoids intermediate 3D processing stages like iso-surface extraction, further streamlining the pipeline.", "questions": "1. Could you include comparisons with more recent Text-to-3D approaches, such as MVDream, CRM, DreamGaussian, and other fast, high-quality methods based on Gaussian Splatting? Including comparisons to more recent approaches would offer readers a clearer understanding of where GIMDiffusion stands relative to current advancements.\n\n2. How does GIMDiffusion perform with diverse object types, especially those with intricate internal structures or shapes? Could you provide more detail on potential methods to mitigate visible seams or artifacts between geometry images? It would be helpful if you could discuss or experiment with approaches for reducing these seams, such as stitching algorithms or blending techniques that could improve texture continuity, especially for high-fidelity applications.\n\n3. The visual quality of this method is not obviously better than some other recent methods, which raises concerns about its competitive edge. While GIMDiffusion offers efficiency and compatibility with 2D architectures, it would benefit from additional improvements or evaluations to enhance the clarity, realism, and continuity of generated textures and geometry.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Innovative 2D-Based Approach for 3D Generation: By using geometry images, GIMDiffusion avoids the need for specialized 3D architectures, simplifying the overall system and reducing the computational resources required.\n2. The paper introduces a Collaborative Control mechanism, which allows two parallel diffusion models to communicate effectively. This cross-model communication lets the geometry generation model inform and guide the 2D texture model, a unique architectural contribution that enhances model efficiency and adaptability.\n3. Rapid and Resource-Efficient Processing: The generation time is highly competitive, making this model a viable choice for real-time applications and scalable deployment.", "weaknesses": "1. In figure 9, the paper observes artifacts, such as visible seams between geometry images, which could compromise visual quality, especially in high-precision applications requiring seamless textures and continuous surfaces.\n\n2. While the paper highlights the efficiency and practicality of geometry images, it doesn’t fully address potential limitations of this 2D representation for complex or high-genus shapes. Multi-chart geometry images mitigate some challenges, but further analysis is needed on cases where this approach may fail, such as in scenarios with high geometric distortion or disconnected regions.\n\n3. Limited Comparison with Recent Methods: The paper's evaluation lacks comprehensive comparisons with the latest advancements, instead focusing on older benchmarks like DreamFusion, ProlificDreamer, and Shap-E. Expanding comparisons to include recent approaches, such as MVDream, CRM, DreamGaussian, and Text-to-3D using Gaussian Splatting, would clarify where GIMDiffusion specifically excels or encounters limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P65niT4Oxc", "reviewer": "Reviewer_tGfn", "summary": "This paper presents a method to generate geometry images with diffusion models, which represents meshes with texture maps with images. Specifically, the authors finetuned stablediffusion to generate the albeda images and use collaborative control to generate the geometry images, which can then be unfolded to 3D meshes. The authors evaluated their method on T^3 bench and shows that it can generally well beyond training data and also have better geometry.", "questions": "I noticed that the albedo branch used the pretrained VAE, buth GIM is trained from scratch. Have the authors considered also using the pretrained model for GIM? What is the reason of not using the pretrained model? Is there any comparison for that?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using pretrained diffusion models for native 3D generation is interesting and novel\n- The proposed method is straightforward and easy to understand. The proposed components are also well validated in the ablation study.\n- The generated 3D meshes have semantic information and can be decomposed into several parts, make it potential for downstream editing tasks. It also directly generates UV maps and is directly relightable.", "weaknesses": "- Although the generated meshes are quite sharp and have details, there are obvious discontinuous artifacts, which can be observed in the video. The reason might be that during reconstructing the ground truth geometry images, these parts are already disconnected. Also, in Fig. 7, the boundaries of the mesh also have noticeable artifacts. What is the reason behind these artifacts? Are these artifacts solvable by improving the current pipeline? Is it possible to apply some post-processing techniques to deal with that?\n- The details of the evaluation is not that clear: how many prompts are used during evaluation and what are they? The authors may provide the number of prompts and some examples of that.\n- I would like to see more comparison with two-stage mesh methods, e.g. first multi-view and then reconstruct mesh, especially CRM, which works faster and have better quality than LGM to my knowledge. It can also produces UV maps. Besides, the authors should also show more examples of comparison in the paper. It would be great if the authors could provide the same comparisons similar to what have been shown in Table 1 for CR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)