Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation

## Abstract

Abstract Generating high-quality 3D objects from textual descriptions remains a challenging problem due to computational cost, the scarcity of 3D data, and complex 3D representations.
We introduce Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D model that utilizes geometry images to efficiently represent 3D shapes using 2D images, thereby avoiding the need for complex 3D-aware architectures.
By integrating a Collaborative Control mechanism, we exploit the rich 2D priors of existing Text-to-Image models such as Stable Diffusion.
This enables strong generalization even with limited 3D training data (allowing us to use only high-quality training data) as well as retaining compatibility with guidance techniques such as IPAdapter. In short, GIMDiffusion enables the generation of 3D assets at speeds comparable to current Text-to-Image models.
The generated objects consist of semantically meaningful, separate parts and include internal structures, enhancing both usability and versatility.

## 1 Introduction

Automatic 3D object generation promises significant benefits across video game production, cinema, manufacturing, and architecture.
Despite notable progress in this area, particularly with Text-to-3D diffusion models [, , ], generating high-quality 3D objects remains a challenging task due to computational costs, data scarcity, and the complexity of typical 3D representations. Furthermore, it is crucial that the generated objects can be re-lit within graphics pipelines, necessitating the use of physically-based rendering (PBR) materials.
Although graphics pipelines predominantly use meshes as their primary 3D representation, processing these at scale is notoriously difficult. Most techniques instead generate an intermediate representation, which adds additional burden to training data pre-processing and generated object post-processing.

We propose diffusing geometry images [], a 2D representation of 3D surfaces, with a Collaborative Control scheme [].
This enables 3D object generation from text prompts, as shown in Fig. 1.
The geometry image representation allows us to repurpose existing image-based architectures, while the Collaborative Control scheme enables us to leverage pre-trained Text-to-Image models, considerably reducing the required training data and cost.
Geometry images, and more specifically multi-chart geometry images, offer two great advantages over other representations: they do not impose constraints on the topology of the generated object, and they naturally separate the generated object into semantically meaningful parts, making the resulting objects easier to manipulate and edit.
We believe that GIMDiffusion opens up a promising new research direction in Text-to-3D generation, providing a practical and efficient approach that can inspire future advancements in the field.

In summary, the advantages of GIMDiffusion include:

- •
Image-based: By leveraging existing 2D image-based models instead of developing new 3D architectures, we simplify both model design and training.
- •
Fast Generation: We generate well-defined 3D meshes in under 10 seconds per object, which could be further enhanced using distillation techniques.
- •
Generalization: Through collaborative control, we reuse pre-trained Text-to-Image priors, allowing us to generalize well beyond our limited 3D training data.
- •
Separate Parts: GIMDiffusion creates assets that consist of distinct, semantically-meaningful, separable parts, facilitating easier manipulation and editing of individual components.
- •
Albedo Textures: The 3D assets generated by GIMDiffusion do not have baked-in lighting effects, making them suitable for various environments.
- •
Straightforward Post-processing: Our 3D assets do not require the application of isosurface extraction algorithms or UV unwrapping, which reduces potential artifacts and simplifies the overall workflow.

## 2 Related work

### 2.1 Text-to-Image Generation

Diffusion models [, ] and flow matching [], alongside the rise of versatile, general-purpose architectures such as transformers [], have brought considerable progress in generative modeling.
In particular, text-conditioned image generation was revolutionized by approaches based on Latent Diffusion [] and its further extensions [, , ].
Foundational models like Stable Diffusion, trained on extensive internet-scale datasets (such as LAION-5B []), are capable of generating complex scenes from text prompts while exhibiting an implicit understanding of scene geometry.
Due to the high cost of training such models, they are often repurposed for other tasks or modalities [, , ]: our proposed GIMDiffusion is a prime example of this, restricting the base model to output albedo textures specifically.

### 2.2 Conditioning Diffusion Models

Control mechanisms modify pre-trained foundational models, enabling them to accept additional conditions to guide the generation process.
Existing pixel-aligned control techniques fall into two categories: fine-tuning the base model with modified input and output spaces [, ], or a separate model that alters the base model’s internal states [, ].
The latter approaches, such as ControlNet [], have gained wide adoption due to their ability to preserve the original model’s performance while adding conditions such as human poses or depth images.
AnimateAnyone [] leverages a similar architecture to inject the base model’s hidden states into a new branch that aligns to the base model’s output.

In our application, we need to both control the base model (which will output UV-space albedo maps) and extract significant features from it (to generate the geometry image modality).
Collaborative Control [] achieves exactly this by introducing bidirectional communication between both models, original for Text-to-PBR-Texture generation.
The ability to integrate completely new modalities into existing models, without retraining and with a reduced risk of catastrophic forgetting, is invaluable to our application.

### 2.3 Text-to-3D generation

We identify two main approaches to Text-to-3D generation: optimization-based and feed-forward methods.
*Optimization-based* methods rely on pre-trained 2D image diffusion models to generate 3D assets [, ] and can produce content of high perceptual quality, at the cost of impractically long generation times [, ].
These methods adapt existing 2D diffusion models to 3D by applying score distillation sampling [, ] to iteratively optimize a 3D representation [, ]. The key advantage of this approach is its ability to utilize the rich 2D prior, allowing for 3D object generation without the need for expensive 3D data.

However, the lack of camera conditioning leads to discrepancies among different viewing angles (Janus effect []) and projection artifacts, which require 3D-aware architectures and retraining on restrictive 3D datasets to mitigate these issues [, , , , , ], while weakening the 2D prior.
Additionally, the original formulation can lead to issues such as saturated colors, oversmooth geometry, and limited diversity [, , , , , ].

*Feed-forward* methods directly generate 3D shapes without the need for iterative refinement.
However, we must consider compatibility with graphics pipelines, which predominantly use textured meshes as their primary representation — an inherently challenging modality for learning-based models.
While seminal works like Point-E [] and its follow-ups [, ] demonstrate impressive generalization and diversity, the inherent lack of connectivity information limits the expressiveness of point clouds. Instead, many current methods rely on proxy representations, such as neural implicits [, , , , , , ] or neural triplanes [, , , , ], to represent the objects.

Because of this, these methods require pre- and post-processing to transform between the mesh domain and the proxy representation, *e.g*. through marching cubes [] or tetrahedra []. This process is costly and far from lossless, introducing issues such as quantization or grid-like artifacts, and resulting in the loss of information, including part segmentation and internal structures. While direct mesh generation techniques exist [], these (as well as the other feed-forward method) must be trained from scratch on 3D data, which is scarce and expensive to gather.
By remaining in the image domain, GIMDiffusion leverages Collaborative Control to ease this data burden [].

### 2.4 Geometry Images

Geometry images (GIM) [, ] have been largely overlooked in deep learning [].
XDGAN [] is a pioneering effort that utilizes GIMs as the representation of choice for a StyleGAN-based architecture []. However, due to the architectural constraints, the training data must be perfectly aligned with a template atlas, which limits its applicability to real-world data. Furthermore, the pre-processing algorithm provided in the paper is overly restricted to shapes of genus zero — Section 3.3.2 shows how to handle arbitrary shapes.

Recent concurrent work [] has advocated for low-resolution $64\times 64$ geometry images as a 3D representation for class-conditioned diffusion models and highlights its efficiency on a small-scale dataset of $8000$ objects [], albeit with limited generalization.
In contrast, GIMDiffusion tackles general Text-to-3D: rather than training a model from scratch, we leverage a pre-trained Text-to-Image diffusion model (using a Collaborative Control scheme trained on Objaverse []) to retain generalization and diversity for the shapes, their appearance, *and* the UV atlas layout.

Figure: Figure 2: (a) Ground-truth geometry, (b) geometry image and (c) albedo texture from our data pre-processing, and (d) the reconstruction using our dedicated VAE. We note the highly separable nature of the ground truth object, which is split into small components. The only visible artifact after decoding is the missing connection between the various charts of the geometry image, as discussed in Secs. 3.3 and 4.3.
Refer to caption: extracted/5835733/figures/gim_reconstruction/images/gunslinger_geo.png

## 3 Method

### 3.1 Geometry Images

Geometry images [] represent a 3D surface in image format as a function $\phi:[0,1]^{2}\rightarrow S\in\mathbb{R}^{3}$, where $S$ is the object surface and $\left[0,1\right]^{2}$ represents the UV coordinates in the unit square, typically sampled on a uniform grid of the desired resolution.
The choice of the function $\phi$ is critical and is usually designed to minimize spatial distortion (using conformal mapping) for a uniform covering of the geometry.

Unlike a traditional mesh, which requires explicit data structures to maintain connectivity information to form faces, geometry images implicitly connect each pixel to its immediate neighbors.
While Gu *et al*. cut the input surface and warp it onto a disc [] (*i.e*. restricted to manifold objects, resulting in significant distortions for high-genus objects), Multi-Chart Geometry Images [] address this limitation by using atlas construction to map the surface piecewise onto several charts of arbitrary shape (each chart being homeomorphic to a disc). This approach adds flexibility (removing the manifold constraint), and reduces distortion (providing greater geometric fidelity), though the optimal construction of such multi-chart GIMs remains an active area of research [, ].

Fortunately, most available 3D meshes have textures applied through UV maps. We observe that these handcrafted UV maps can be employed to construct a desirable multi-chart $\phi$.
Furthermore, charts in handcrafted UV maps often carry semantic meanings, which propagate to the output of our model. This is evident in Fig. 2, where the hands, face, and various part of the gunslinger’s appearance are clearly separated in the UV atlas.

As the density of the generated geometry’s triangulation is restricted by the resolution of the underlying geometry image, we follow [] and use a VAE to increase the effective resolution of our proposed model.
To address the irregularities in geometry images and better match their distribution — particularly the need to accurately reconstruct the discontinuities at the boundaries of the charts — we add a channel to represent the multi-chart mask and modify the loss function accordingly.
Otherwise, we follow the VAE training procedure from StableDiffusion1.5 [].
As shown in Fig. 2, the reconstruction using this VAE does not visually differ from the input mesh, except for the missing connections between the charts, which we do not currently handle.

### 3.2 Collaborative control

To maximally leverage the prior knowledge encoded in existing 2D Text-to-Image models, we use the Collaborative Control approach []. As shown in Fig. 3, this approach comprises two parallel networks: a pre-trained RGB model and a new model for the geometry image. The former is responsible for generating UV-space albedo textures, while the latter generates the geometry images. These two models are connected by a simple linear cross-network communication layer, which allows them to share information and collaborate in generating pixel-aligned outputs across these different modalities. Crucially, this also enables the geometry model to influence the frozen model, guiding it to generate UV-space textures that would otherwise lie at the fringes of its training distribution. The frozen base model also drastically reduces the amount of data required to train the joint model while retaining generalizability, diversity, and quality [].

Figure: Figure 3: The Collaborative Control Scheme [] applied in GIMDiffusion, where two separate diffusion models generate respectively albedo textures and geometry images. The former is a frozen pre-trained model, while the latter is an architectural clone trained from scratch.
Refer to caption: extracted/5835733/figures/collab_control/images/tex_hovercraft_19.png

### 3.3 Data Handling

#### 3.3.1 Dataset

We train our model on the Objaverse dataset [].
We curate this dataset to include only objects with both high-quality structures and semantically meaningful UV maps by filtering out 3D scans and low-poly models.
The final dataset contains approximately $100,000$ objects.
Each data entry is accompanied by captions provided by Cap3D [] and Hong *et al*. [].
During training, we randomly sample from these captions and apply random rotations of 90, 180, or 270 degrees to the extracted texture atlases.
We now discuss how to transform these meshes into geometry images and back: the entire preprocessing was performed on consumer-grade PC hardware (AMD Ryzen 9 7950X, GeForce RTX 3090, 64 GB RAM) and took approximately 20 hours.

#### 3.3.2 Geometry Image Creation

Figure: Figure 4: Seam detection in our multi-chart geometry image creation procedure to isolate *locally invertible* areas of the UV mapping. (Left) If two neighboring mesh regions correspond to two distinct charts in the UV map, the vertices on the boundary will be duplicated and have different UV coordinates. (Right) If the UV mapping loops back onto itself, there will be a local minimum in the UV access heatmap, and we place the seam along the line with the smallest UV-degree to effectively separate these regions.

To create multi-chart geometry images for the training shapes, we use their existing UV maps.
UV mapping is defined as the mapping $\rho:V\rightarrow\left[0,1\right]^{2}$, where $V$ is the set of vertices in a 3D mesh and $\rho$ maps each vertex to UV coordinates.
Note that this mapping is not injective (multiple vertices can be mapped to the same UV coordinates), nor is it a simple function of the vertex positions in $\mathbb{R}^{3}$, as modern mesh formats allow for multiple vertices with distinct UV coordinates at the same 3D location (to handle sharp texture transitions).
These issues mean that $\rho$ is not invertible, which would be a simple way to create a geometry image function.
However, we argue that $\rho$ is *locally invertible* and propose to construct a multi-chart geometry image based on the individually invertible areas of the available UV mapping.
As mentioned before, the islands in a UV atlas tend to be semantically meaningful, so that we aim to preserve them.

We begin by identifying the connected components of the mesh, which provides an initial separation into charts. Within each component, we identify two situations where $\rho$ is not invertible: duplicated vertices with distinct UV coordinates, and a “crease”, *i.e*. a line where the UV coordinates change direction (similar to a “mirror” boundary constraint). Both situations are visualized in Fig. 4. The former is straightforward to detect, as we can detect duplicated vertex positions. The latter is detected by creating a heatmap of the access pattern in UV space and detecting local minima. We then further split the individual charts along any detected creases.
Finally, in line with the desirable mapping properties discussed in [], we adjust the geometry image mapping to approximate an equal-area projection by rescaling each 2D chart with respect to the area of the corresponding surface.

In cases where only a partial UV mapping is available, we use Xatlas [] to UV-unwrap the missing regions. However, since the Xatlas parametrization is of lower quality and lacks semantic properties, we exclude meshes where less than $80\%$ of the surface area has been unwrapped manually. This simple heuristic allows us to construct multi-chart geometry images for nearly all training examples.
However, it does not account for all possible degenerate cases of $\rho$.
Therefore, we verify that the constructed $\rho$ is injective and skip any training meshes for which this assumption is violated.

#### 3.3.3 Coordinate transform

In Objaverse [], the objects are not aligned in a canonical way.
Although almost all shapes are oriented with the Y axis pointing upward, there is no fixed front view.
In practice, this leads to a rotational ambiguity between the X and Z axes.
To resolve this ambiguity and isolate it to a single coordinate, we leverage cylindrical coordinates $\left(r,\theta,\phi\right)$ so that the ambiguity is contained within the azimuth $\theta$.

Figure: Figure 5: The resulting triangulation of our generated objects is near-uniform over the surface, thanks to the area-preserving nature of the geometry images in our training dataset.
Refer to caption: extracted/5835733/figures/wireframe.png

#### 3.3.4 Mesh Extraction

As mentioned in Sec. 3.1, geometry images implicitly encode connectivity information of the mesh by treating neighboring pixels as connected in 3D space, *i.e*. a quad mesh.
However, to convert them into a more widely supported triangular mesh, we need to specify exactly how to triangulate the quads.
For this, we closely follow the algorithm from Sander*et al*. [].
For any $2\times 2$ block of pixels in the GIM, we create up to two triangles depending on how many of the pixels are valid: if necessary, the quad is split along its shorter diagonal.
As shown in Fig. 5, and in line with our goal of area-preserving mapping [], the resulting triangulation is nearly uniform over the surface, which may or may not be desirable for specific applications (as the working resolution of our model is $768\times 768$, our GIMs can encode meshes with up to $589,824$ vertices).
We consider the generation of arbitrary topologies or with a polygon constraint a promising area of future work for GIMDiffusion.

### 3.4 Training

For the frozen base model, we used a zero-terminal-SNR [] version fine-tuned [] from StableDiffusion v2.1 [] as the base Text-to-Image model, which is kept frozen and generates the albedo textures. The geometry model is an architectural clone that is trained from scratch, together with the cross-network communication layers. Initially, we train the model at $256\times 256$ resolution for $250,000$ steps with a batch size of $48$, and then at the final output resolution of $768\times 768$ for a total of $100,000$ steps with a batch size of $8$.
All stages of training were conducted with a learning rate of $3e^{-5}$ on 8 A100 GPUs.

## 4 Results

Figure 1 shows the results of our method on a set of text prompts, generating objects as they might reasonably be queried, for example, in a gaming workflow.
The objects are well-defined and can be relit from any direction, as the generated albedo textures do not contain any lighting-related artifacts. Furthermore, Fig. 6 illustrates that our proposed method is able to generate non-trivial variations for seed or prompt perturbations, which is crucial for the practical use of such a system.

A key observation we make with regard to the concurrent work of Yan *et al*. [] is that their method exhibits only very little variation in the UV layout of the generated objects within a given class. In contrast, GIMDiffusion shows significantly different UV layouts when either the seed or the prompt is varied slightly, which increases the practical value of generated multiple objects.
Finally, benefiting from the rich natural image prior, our model generalizes well beyond the initial 3D dataset.
Additional examples demonstrating this generalization are shown in Fig. 7.

Figure: Figure 6: Sample diversity of GIMDiffusion for minor changes to the prompt or different random seeds for the initial gaussian noise. It is clear that the generated variations differ significantly, not just in appearance and structure but also in the texture’s atlas layout. This is invaluable in practical applications, where users typically generate multiple options and pick the best one.
Refer to caption: extracted/5835733/figures/gim_diversity/mushroom_texture.png

Figure: Figure 7: GIMDiffusion, thanks to the powerful natural image prior of the base model in combination with the Collaborative Control scheme, generalizes well outside the “vanilla” nature of the Objaverse training data.
Refer to caption: extracted/5835733/figures/gim_generalization/giger_gun.png

Figure: Figure 8: We can stylistically guide the reverse process by applying a pre-trained IPAdapter to the frozen base model generation the albedo textures []. This is extremely valuable in applications where the assets need to match an existing “feel”.
Refer to caption: extracted/5835733/figures/ipadapter/chair_photo.png

Figure: Figure 9: The generated images retain the semantically meaningful separation of charts in the texture atlas. We illustrate this here by showing an exploded view of a generated “A fly agaric mushroom”. It is clear that the various parts of the mushroom were separated as a human might, and that the method is even able to model internal parts of the shape.
Refer to caption: extracted/5835733/figures/gim_diversity/mushroom_texture.png

### 4.1 IPAdapter Compatibility

In practical terms, guiding the style of generated objects is crucial for many applications.
Major efforts have been made to achieve style control in diffusion models [], and our method is compatible with these techniques, similar to the original Collaborative Control approach [].
As we are leveraging a frozen base model to generate the albedo textures, we can apply a pre-trained IPAdapter to that base model and produce stylized output meshes, as shown in Fig. 8.
Despite the significant mismatch in the application domain (natural images compared to albedo atlases), the style guidance remains successful.
We find that this approach starts to break down when the text prompt deviates too much (structurally) from the content of the conditioning image, which we attribute to the fact that IPAdapter aims to leverage every aspect of the conditioning image; but *e.g*. IPAdapterInstruct [] offers a selective extraction of just appearance style but without entangling structure.

### 4.2 Separability and Internal Structure

A key advantage of our method is that it generates objects divided into distinct semantic (or nearly semantic) parts as shown in Fig. 9, making the generated objects more suitable for editing and animation.
This capability arises from the multi-chart representation design and the semantic information embedded in the training data through handcrafted UV-maps, which loosely correspond to the semantic components of objects. This approach also allows users to easily correct imperfections, such as misaligned parts or extraneous geometry, and even combine different parts from multiple generations.
Additionally, our method generates internal structures, such as the filament inside a light bulb or the interior of a fish tank, because geometry images represent the entire object holistically, not just the visible surfaces.

### 4.3 Limitations

Our current method is not without limitations.
The most common issue is the appearance of visible cracks in the generated meshes.
While we do not currently stitch the charts’ seams together, which could improve the visual quality of the generated meshes, we believe that this problem is further exacerbated by the VAE’s latent compression.
Areas smaller than $8\times 8$ pixels are essentially below the VAE’s latent resolution, causing visual problems.

We also encounter considerable ambiguity in the geometry image mapping.
Although individual charts in a geometry image can be rotated arbitrarily and still represent the same 3D object, the frozen base model is not rotationally equivariant [].
Human faces serve as a good example — while typically upright in natural images, they can appear in random orientations on texture maps, which leads to varying (and overall lower) quality for such prompts.

Additionally, we find that the model sometimes duplicates parts of the object, which can lead to visual artifacts in the generated meshes due to z-fighting.
However, the separated nature of the generated objects makes these artifacts relatively easy to resolve manually.

## 5 Discussion and future work

In this work, we present Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D generation paradigm that utilizes geometry images as its core 3D representation in combination with powerful natural image priors in the form of pre-trained diffusion models.
Our results show that GIMDiffusion can generate relightable 3D assets as efficiently as existing Text-to-Image methods generate normal images, while avoiding the need for complex, custom 3D-aware architectures. We believe that our research lays the groundwork for a new direction in Text-to-3D generation.

Further quality improvements include addressing issues such as inter-chart alignment and eliminating visible cracks.
Additionally, incorporating topology prediction and conditioning on specific polygon budgets would enhance control over the generated 3D objects, making them more suitable for use in gaming and other graphics pipelines.
Equally promising is the potential of GIMDiffusion in related fields such as animation or text-to-video generation.

## 6 Acknowledgements

We would like to thank Shimon Vainer from Unity Technologies, Dr. Lev Melnikovsky from the Weizmann Institute and Alexander Demidko for their valuable feedback and insightful discussions. Konstantin Kutsy has been invaluable in helping us operate our training infrastructure.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see the first two questions in the weakness: the chart quality and diversity; and the artifacts on the boundary.
soundness: 3
strengths: - The reviewer loves the elegant idea very much. I personally believe this is the correct way in a long-term to generate usable (industry level) and high-quality assets for objects. Because most generative model for objects now depends on large synthetic dataset, with known geometry.
- The system is simple and effective.
- Also, the semantically meaningful parameterization has much more space in the future for editing and manipulation.
summary: This paper studies how to generate high quality mesh of objects with GIM.

The 3D objects are parameterized into 2D chart geometry image. This representation is compact where the prior of 2D SD model can be exploited and maintains good geometry and semantic properties. The generation happens vis modifying standard SD with cross attention between different attributes of the chart.
weaknesses: - My main concern is maybe the diffusion overfit the proposed UV chart distribution. But we know that there is uncertainty of how an object can be parameterized into a chart. Is there any measurement of how good the chart is in the proposed preprocessing pipeline? e.g. it maintains the surface area or local differential properties? Can the proposed method generate different chart pattern for the exact same object? And how diverse is the chart?
- There are some small artifacts in the boundary of the mesh, which may due to the boundary of the chart. I believe this can be fixed in future works, but an analysis of this artifacts may provide more intuition of the future work.
- Maybe one or two applications like relighting .etc will future demonstrate the strength of generating object via GIM.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does the variability in multi-chart geometry image layouts impact GIMDiffusion’s training stability and scalability?
2. The quality of shape edges appears to be suboptimal in the generated results. Is this a known limitation of the geometry image representation, and are there potential solutions to address this?
3. The authors freeze the albedo generation branch based on the pre-trained SD model, but SD’s performance in UV space is often limited. Could the authors provide experimental evidence to show that this decision does not compromise the quality of generated RGB textures?
soundness: 3
strengths: 1. Geometry images provide a low-complexity 3D representation, allowing GIMDiffusion to fully leverage pre-trained 2D diffusion models for 3D generation.
2. The model preserves the weights of the pre-trained image diffusion model, making it compatible with plugins like IP-Adapter for style transfer and customization.
summary: The paper introduces Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D generation model that uses geometry images to represent 3D shapes in a 2D format. This approach leverages the priors of pre-trained image diffusion models, allowing efficient 3D object generation without complex 3D-aware architectures. By retaining pre-trained weights, GIMDiffusion is compatible with tools like IP-Adapter, enabling flexible style control and broad applicability. The use of multi-chart geometry images further enhances the model’s ability to represent complex shapes, offering a scalable and versatile solution for text-driven 3D generation.
weaknesses: 1. My primary concern with multi-chart geometry images is the variable layout configurations that a single 3D shape can produce. That means a single 3D shape may correspond to many different geometry images, potentially complicating the training process and impacting consistency across different shapes. Is any technique used to standardize the conversion from mesh to geometry image.
2. Generated results in the paper often show lower quality around shape edges, possibly due to the geometry image representation. This limitation affects the clarity and sharpness of complex geometries.
3. The model freezes the pre-trained Stable Diffusion (SD) model for generating albedo textures. However, SD has known limitations in UV space, and freezing this branch could compromise RGB quality. Providing experiments to validate this decision would help substantiate the model’s robustness.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you include comparisons with more recent Text-to-3D approaches, such as MVDream, CRM, DreamGaussian, and other fast, high-quality methods based on Gaussian Splatting? Including comparisons to more recent approaches would offer readers a clearer understanding of where GIMDiffusion stands relative to current advancements.

2. How does GIMDiffusion perform with diverse object types, especially those with intricate internal structures or shapes? Could you provide more detail on potential methods to mitigate visible seams or artifacts between geometry images? It would be helpful if you could discuss or experiment with approaches for reducing these seams, such as stitching algorithms or blending techniques that could improve texture continuity, especially for high-fidelity applications.

3. The visual quality of this method is not obviously better than some other recent methods, which raises concerns about its competitive edge. While GIMDiffusion offers efficiency and compatibility with 2D architectures, it would benefit from additional improvements or evaluations to enhance the clarity, realism, and continuity of generated textures and geometry.
soundness: 3
strengths: 1. Innovative 2D-Based Approach for 3D Generation: By using geometry images, GIMDiffusion avoids the need for specialized 3D architectures, simplifying the overall system and reducing the computational resources required.
2. The paper introduces a Collaborative Control mechanism, which allows two parallel diffusion models to communicate effectively. This cross-model communication lets the geometry generation model inform and guide the 2D texture model, a unique architectural contribution that enhances model efficiency and adaptability.
3. Rapid and Resource-Efficient Processing: The generation time is highly competitive, making this model a viable choice for real-time applications and scalable deployment.
summary: 1. GIMDiffusion uses geometry images to represent the surface of 3D shapes in a 2D format. This method, unlike traditional 3D meshes, can utilize the extensive 2D image generation capabilities of diffusion models like Stable Diffusion. This approach results in faster generation times, lower data requirements, and the elimination of the need for complex 3D-specific model architectures, which are often cumbersome and data-hungry.

2. The model introduces a Collaborative Control mechanism to integrate 2D Text-to-Image models with geometry images. Through this mechanism, the geometry and texture generation processes are interconnected, allowing each to influence the other. This setup ensures that the textures generated align well with the 3D shape, facilitating realistic and coherent representations even when there is limited 3D data available.

3. Efficiency and Performance: GIMDiffusion achieves faster-than-average generation times, producing complex 3D assets in approximately 10 seconds. Moreover, the model avoids intermediate 3D processing stages like iso-surface extraction, further streamlining the pipeline.
weaknesses: 1. In figure 9, the paper observes artifacts, such as visible seams between geometry images, which could compromise visual quality, especially in high-precision applications requiring seamless textures and continuous surfaces.

2. While the paper highlights the efficiency and practicality of geometry images, it doesn’t fully address potential limitations of this 2D representation for complex or high-genus shapes. Multi-chart geometry images mitigate some challenges, but further analysis is needed on cases where this approach may fail, such as in scenarios with high geometric distortion or disconnected regions.

3. Limited Comparison with Recent Methods: The paper's evaluation lacks comprehensive comparisons with the latest advancements, instead focusing on older benchmarks like DreamFusion, ProlificDreamer, and Shap-E. Expanding comparisons to include recent approaches, such as MVDream, CRM, DreamGaussian, and Text-to-3D using Gaussian Splatting, would clarify where GIMDiffusion specifically excels or encounters limitations.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: I noticed that the albedo branch used the pretrained VAE, buth GIM is trained from scratch. Have the authors considered also using the pretrained model for GIM? What is the reason of not using the pretrained model? Is there any comparison for that?
soundness: 3
strengths: - The idea of using pretrained diffusion models for native 3D generation is interesting and novel
- The proposed method is straightforward and easy to understand. The proposed components are also well validated in the ablation study.
- The generated 3D meshes have semantic information and can be decomposed into several parts, make it potential for downstream editing tasks. It also directly generates UV maps and is directly relightable.
summary: This paper presents a method to generate geometry images with diffusion models, which represents meshes with texture maps with images. Specifically, the authors finetuned stablediffusion to generate the albeda images and use collaborative control to generate the geometry images, which can then be unfolded to 3D meshes. The authors evaluated their method on T^3 bench and shows that it can generally well beyond training data and also have better geometry.
weaknesses: - Although the generated meshes are quite sharp and have details, there are obvious discontinuous artifacts, which can be observed in the video. The reason might be that during reconstructing the ground truth geometry images, these parts are already disconnected. Also, in Fig. 7, the boundaries of the mesh also have noticeable artifacts. What is the reason behind these artifacts? Are these artifacts solvable by improving the current pipeline? Is it possible to apply some post-processing techniques to deal with that?
- The details of the evaluation is not that clear: how many prompts are used during evaluation and what are they? The authors may provide the number of prompts and some examples of that.
- I would like to see more comparison with two-stage mesh methods, e.g. first multi-view and then reconstruct mesh, especially CRM, which works faster and have better quality than LGM to my knowledge. It can also produces UV maps. Besides, the authors should also show more examples of comparison in the paper. It would be great if the authors could provide the same comparisons similar to what have been shown in Table 1 for CR.

Meta-review: This paper introduces Geometry Image Diffusion, a novel framework for Text-to-3D generation that leverages 2D diffusion-based techniques to represent 3D shapes through geometry images. By sidestepping the complexity of existing 3D-aware architectures, the method significantly enhances efficiency and generalization in Text-to-3D tasks. The integration of a Collaborative Control mechanism further allows the model to exploit rich 2D priors from established Text-to-Image models like Stable Diffusion, resulting in high-quality and semantically meaningful 3D asset generation.

All reviewers provided positive ratings for this work, with several highlighting its novelty and potential impact on Text-to-3D generation. The authors also effectively addressed concerns raised during the review process, further strengthening the consensus around the paper’s merit. Based on the feedback and discussions, the area chair finds no reason to overturn the consensus and recommends acceptance. However, there are still areas that need further attention. Reviewer Vjsj noted that the MGIM boundary can produce some artifacts, and Reviewer nxjJ remains concerned about the diversity of the representation. The authors are encouraged to address these issues in the final version to further strengthen the paper.

Decision: Accept (Poster)