GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Authors: Jun Gao
Tianchang Shen
Zian Wang
Wenzheng Chen
Kangxue Yin
Daiqing Li
Or Litany
Zan Gojcic
Sanja Fidler
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D GAN
mesh
texture
topology

Abstract: As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident. In our work, we aim to train performant 3D generative models that synthesize textured meshes which can be directly consumed by 3D rendering engines, thus immediately usable in downstream applications. Prior works on 3D generative modeling either lack geometric details, are limited in the mesh topology they can produce, typically do not support textures, or utilize neural renderers in the synthesis process, which makes their use in common 3D software non-trivial. In this work, we introduce GET3D, a Generative model that directly generates Explicit Textured 3D meshes with complex topology, rich geometric details, and high fidelity textures. We bridge recent success in the differentiable surface modeling, differentiable rendering as well as 2D Generative Adversarial Networks to train our model from 2D image collections. GET3D is able to generate high-quality 3D textured meshes, ranging from cars, chairs, animals, motorbikes and human characters to buildings, achieving significant improvements over previous methods.

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Jun Gao 12,,3
Tianchang Shen 1.2,3
Zian Wang 1.2.3
Wenzheng Chen 1,2,3
Kangxue Yin
Daiqing Li
Or Litany
Zan Gojeic'
Sanja F.2
NVIDIA'
University of Toronto
Vector Institute
(jung, Irahen, zians, weuzchen, kangxuey, daiqingl, olitany, agojeic, stidler)dvvddaa.co
Abstract
As several industries moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident. In our work, aim train performant 3D generative models that synthesize textured meshes that can be directly consumed by 3D rendering engines, thus immediately usable in down- stream applications. Prior works on 3D generative modeling either lack geometric details, are limited in the mesh topology they can produce, typically do support textures, or utilize neural renderers in the synthesis process, which makes their use in common 3D software non-trivial. a Generative model that directly generates Explicit Textured 3D meshes with complex topology, rich geometric details, and high fidelity textures. We bridge recent success in the differentiable surface modeling, differrntiablee rendering well as 2D Generative Adversarial Networks to train our model from 2D image collections. GET3D is able to generate high-quality 3D textured meshes, rang- ing from cars, chairs, animals, motorbikes and human characters to buildings, achieving significant improvements over previous methods. Our project page: https://nv-- tlabs. .ee..gitttiii github. 1o/GET3D
this work, we introduce GET3D,
Introduction
Diverse, high-quality 3D content is becoming increasingly important for several industries, including gaming, robotics, architecture, and social platforms. However, manual creation of 3D assets is very time-consuming and requires specific technical knowledge as well as artistic modeling skills. One of the main challenges thus scale while one can find 3D models on 3D marketplaces such as Turbosquid [4] Sketchfab [3], creating many 3D models to, say, populate game movie with crowd characters that all look different still takes significant amount of artist time.
To facilitate the content creation process and make it accessible to a variety of (novice) users, generative 3D networks that can produce high-quality and diverse 3D assets have recently become an active area of research [5, 13, 38, 41, 48, 59, 62, 55, 54, 60, 21]. However, to be practically useful for current real-world applications. 3D generative models should ideally fulfill the following requirements: (a) They should have the capacity to generate shapes with detailed geometry and arbitrary topology, (b) The output should be a textured mesh, which primary representation used by standard graphics software packages such as Blender [14] and Maya [1], and (c) We should be able to leverage 2D images for supervision, as they are more widely available than explicit 3D shapes.
Prior work on 3D generative modeling has focused on subsets of the above requirements, but no method date fulfills all of them (Tab. l). For example, methods that generate 3D point clouds [5,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Method
Application
Representation
Supervision
Textured
Arbitrary topology
OceNet [38]
3D generation
Implicit
3D
PointFlow [59]
3D generation
Point cloud
3D
X
Texture3D [48]
3D generation
Mosh
2D
V
StyleNerf [23]
3Dwawe NV
Neural field
2D
K
EG3D [8]
3D-aware
Neural field
2D
PIGAN
3D-aware NV
Neural field
2D
GRAF [52]
Dawware NV
Neural field
2D
Ours
3D generation
Mesh
2D
Table 1: Comparison with prior works. (NV: Novel view synthesis.)
59, 62] typically do not produce textures and have to be converted to a mesh in post-processing, Methods generating voxels often lack geometric details and do not produce texture [57, 18, 25, 35]. Generative models based on neural fields [38, 13] focus on extracting geometry but disregard texture. Most of these also require explicit 3D supervision. Finally, methods that directly output textured 3D meshes [49, 48] typically require pre-defined shape templates and cannot generate shapes with complex topology and variable genus.
Recently. rapid progress in neural volume rendering [40] and 2D Generative Adversarial Networks (GANs) [31, 32, 30, 27, 47] has led to the rise of 3D-aware image synthesis [7. 52, 8, 44, 46, 23]. However, this line work aims to synthesize multi-view consistent images using neural rendering in the synthesis process and does not guarantee that meaningful 3D shapes can be generated. While mesh can potentially be obtained from the underlying neural field representation using the marching cube algorithm [34], extracting the corresponding texture is non-trivial.
In this work, we introduce a novel approach that aims to tackle all the requirements a practically useful 3D generative model. Specifically, we propose GET3D, Generative model for 3D shapes that directly outputs Explicit Textured 3D meshes with high geometric and texture detail and arbitrary mesh topology. In the heart of our approach is generative process that utilizes differentiable explicit surface extraction method [55] and a differentiable rendering technique [42, 33J. The former enables us to directly optimize and output textured 3D meshes with arbitrary topology, while the latter allows us to train our model with 2D images, thus leveraging powerful and mature discriminators developed for 2D image synthesis. Since model directly generates meshes and uses highly efficient (differentiable) graphics renderer, we can easily scale up our model to train with image resolution high 1024 x 1024, allowing to learn high-quality geometric and texture details.
We demonstrate state-of-the-art performance for unconditional 3D shape generation on multiple categories with complex geometry from ShapeNet [9], Turbosquid [4] and Renderpeople [2], such as chairs, motorbikes, cars, human characters, and buildings. With explicit mesh as output representation, GET3D is also very flexible and can easily be adapted to other tasks, including: (a) learning to generate decomposed material and riew-dependent lighting effects using advanced diffcrentiable rendering [12], without supervision, (b) text-guided 3D shape generation using CLIP [51] embedding.
2 Related Work
We review recent advances in 3D generative models for geometry and appearance, as well as 3D-aware generative image synthesis.
3D Generative Models In recent years, 2D generative models have achieved photorealistic quality in high-resolution image synthesis [31, 32, 30, 47, 27, 17, 15]. This progress has also inspired research in 3D content generation. Early approaches aimed to directly extend the 2D CNN generators to 3D voxel grids [57, 18, 25, 35, 56], but the high memory footprint and computational complexity of 3D convolutions hinder the generation process at high resolution. As an alternative, other works have explored point cloud [5, 59, 62, 41], implicit [38, 13], or octree [28] representations. However, these works focus mainly on generating geometry and disregard appearance. Their output representations also need to be post-processed to make them compatible with standard graphics engines.
More similar to our work, Textured3DGAN [49, 48] and DIBR [11] generate textured 3D meshes, but they formulate the generation as a deformation of template mesh, which prevents them from generating complex topology or shapes with varying genus, which our method can do. PolyGen [43] and SuriGen [36] can produce meshes with arbitrary topology, but do not synthesize textures.

---

Figure 1: We export our generated shapes and visualize them in Blender. GET3D is able to generate diverse shapes with arbitrary topology, high quality geometry, and texture.
3D-Aware Generative Image Synthesis Inspired by the success of neural volume rendering [40] and implicit representations [38, 13), recent work started tackling the problem of 3D-aware image synthesis [7, 52, 44, 24, 23, 63, 8, 46, 53, 58]. However, neural volume rendering networks typically slow to query. leading to long training times [7, 52]. and generate images limited resolution. GIRAFFE [44] and StyleNerf [23] improve the training and rendering efficiency by performing neural rendering at a lower resolution and then upsampling the results with a 2D CNN. However, the performance gain comes at the cost of reduced multi-vize consistency. By utilizing dual discriminator, EG3D [8] can partially mitigate this problem. Nevertheless, extracting textured surface from methods that are based neural rendering is non-trivial endeavor. In contrast, GET3D directly outputs textured 3D meshes that can be readily used in standard graphics engines.
3
Method
We now present GET3D framework for synthesizing textured 3D shapes. Our generation process split into two parts: geometry branch, which differentiably outputs a surface mesh of arbitrary topology, and a texture branch that produces texture field that can be queried at the surface points to produce colors. The latter can be extended to other surface properties such as for example materials (Sec. 4.3.1). During training, efficient differentiable rasterizer is utilized to render the resulting textured mesh into 2D high-resolution images. The entire process differentiable, allowing for adversarial training from images (with masks indicating object of interest) by propagating the gradients from the 2D discriminator to both generator branches. Our model is illustrated in Fig. 2. In the following, we first introduce 3D generator in Sec 3.1, before proceeding the differentiable rendering and loss functions in Sec 3.2.
3.1 Generative Model of 3D Textured Meshes
We aim to learn a 3D generator M, E = G(z) to map a sample from a Gaussian distribution E AV I) to mesh M with texture E.
Since the same geometry can have different textures, and the same texture can be applied different geometries, we sample two random input vectors E I252 and E R 512 Following StyleGAN [31, 32, 30), we then use non-linear mapping networks fgo and fux to map z and 22 to intermediate latent vectors Wi (g) and W2 = (22) which are further used to produce styles that control the generation of 3D shapes and texture, respectively. We formally introduce the generator for geometry in Sec. 3.1.1 and the texture generator in Sec. 3.1.2.

---

Training
Inference
DMTR
Real PPila??
Geometry giseratoc
1)
Z
Realy Pailus?
reshape
(0.
20 Silsouecte
Mapping melwork
Teature getrenik
Testure mushs
Dilieenniable pundering
Diccritiisatoo lock
Figure 2: Overview of GET3D: We generate 3D SDF and texture field via two latent codes. We utilize DMTet [55] to extract a 3D surface mesh from the SDF, and query the texture field at surface points to get colors. We train with adversarial losses defined on 2D images. In particular, we use rasterization-bant bbased differentiable renderer [33] to obtain RGB images and silhouettes. We utilize two 2D discriminators, each on RGB image, and silhouette, respectively, to classify whether the inputs are real or fake. The whole model is end-to-end trainable. Note that additionally provide an improved version of our Generator in Appendix Sec. A.5 and Fig. C.
3.1.1 Geometry Generator
We design our geometry generator to incorporate DMTet [55], a recently proposed differentiable surface representation. DMTet represents geometry signed distance field (SDF) defined deformable tetrahedral grid [20, 22], from which the surface can be differentiably recovered through marching tetrahedra [16]. Deforming the grid by moving its vertices results in better utilization of its resolution. By adopting DMTet for surface extraction, we can produce explicit meshes with arbitrary topology and genus. We next provide brief summary of DMTet and refer the reader the original paper for further details.
Let (Vr, T") denote the full 3D space that the object lies in, where Vr the vertices in the tetrahedral grid T. Each tetrahedron Ti T defined using four vertices {VaL> Vo V Yo Yd, J. with & e (I...., K}. where K is the total number of tetrahedra, and Vi, VT, Vi, E R In addition to its 3D coordinates, each vertex V, contains the SDF value s; E IR and the deformation Av; E IR3 of the vertex from its initial canonical coordinate. This representation allows recovering the explicit mesh through differentiable marching tetrahedra [55]. where SDF values in continuous space are computed by barycentric interpolation of their value s; on the deformed vertices v/ V Av,
Network Architecture We map E G222 to SDF values and deformations at each vertex v; through series of conditional 3D convolutional and fully connected layers. Specifically, we first use 3D convolutional layers to generate feature volume conditioned on W1. We then query the feature at each vertex v; E VT using trilinear interpolation and feed into MLPs that outputs the SDF value and the deformation Av; In cases where modeling a high-resolution is required (e.g. motorbike with thin structures in the wheels), we further use volume subdivision following [55].
Differentiable Mesh Extraction After obtaining s; and Av; for all the vertices, we use the differentiable marching tetrahedra algorithm to extract the explicit mesh. Marching tetrahedra determines the surface topology within each tetrahedron based on the signs of 8;. In particular, a mesh face is extracted when sign(s)) * sign(s,), where i. denotes the indices of vertices in the edge of tetrahedron, and the vertices m.j of that face determined by a linear interpolation
as m, v/==-v/a: Note that the above equation only evaluated when 8; * Sy, thus it is 9,-8; differentiable, and the gradient from m,jj can be back- prok-prop into the SDF values s; and deformations Av. With this representation, the shapes with arbitrary topology can easily be generated by predicting different signs of s,.
3.1.2 Texture Generator
Directly generating a texture map consistent with the output mesh is not trivial, as the generated shape can have arbitrary genus and topology. We thus parameterize the texture texture field [45].

---

Specifically, we model the texture field with a function ft that maps the 3D location of surface point p E IR conditioned on the W2, to the RGB color с E R at that location. Since the texture field depends on geometry, we additionally condition this mapping on the geometry latent code W1, such that c = [P, W1 @ W2), where ( denotes concatenation.
Network Architecture We represent our texture field using a tri-plane representation, which is efficient and expressive in reconstructing 3D objects [50] and generating 3D-aware images [8] Specifically, we follow [8, 32] and use a conditional 2D convolutional neural network to map the latent code W1 ① W2 to three axis- axis-aligned orthogonal feature planes of size N N x (C 3), where N = 256 denotes the spatial resolution and C' = 32 the number of channels.
Given the feature planes, the feature vector f E TR:2 of surface point p can be recovered as
f' = E. (_T.((T)))) where (p)) is the projection of the point p to the feature plane e and p(-) denotes bilinear interpolation of the features. An additional fully connected layer is then used to map the aggregated feature vector f to the RGB color c. Note that, different from other works on 3D-aware image synthesis [8, 23, 7, 52] that also use neural field representation, we only need to sample the texture field at the locations of the surface points (as opposed to dense samples along a ray). This greatly reduces the computational complexity for rendering high- high-resolution images and guarantees to generate multi-view consistent images by construction.
3.2 Differentiable Rendering and Training
In order to supervise our model during training, draw inspiration from Nvdiffrec [42] that performs multi-view 3D object reconstruction by utilizing a differentiable renderer. Specifically, we render the extracted 3D mesh and the texture field into 2D images using differentiable renderer [33], and supervise our network with a 2D discriminator, which tries to distinguish the image from a real object or rendered from the generated object.
Differentiable Rendering We assume that the camera distribution C that was used to acquire the images in the dataset is known. To render the generated shapes, we randomly sample camera c from C, and utilize a highly-optimized differentiable rasterizer Nydiffrast [33] to render the 3D mesh into a 2D silhouette well as an image where each pixel contains the coordinates of the corresponding 3D point on the mesh surface. These coordinates are further used to query the texture field to obtain the RGB values. Since operate directly the extracted mesh, we can render high-resolution images with high efficiency, allowing our model to be trained with image resolution as high as 1024x 1024.
Discriminator & Objective We train our model using an adversarial objective. We adopt the discriminator architecture from StyleGAN [31], and use the same non-saturating GAN objective with RI regularization [37]. We empirically find that using two separate discriminators, one for RGB images and another one for silhouettes, yields better results than a single discriminator operating on both. Let D2 denote the discriminator, where can either be an RGB image or silhouette. The adversarial objective is then be defined as follows:
L(Dar, G) = (e
e))) + (g.
+ (1))
(I)
where g(u) defined as g(u) = - log(1 + exx((-ul Pa is the distribution of real images, R denotes rendering, and X is a hyperparameter. Since Rt is differentiable, the gradients can be backpropagated from 2D images to our 3D generators.
To remove internal floating faces that are not visible in any of the views, we further regularize the geometry generator with cross-entropy loss defined between the SDF values of the neighboring vertices [42]:
Regularization
Trng
Σ I.EES
H (a(s)) sign (s²)) + H (s))), sign (s:)),
(2)
where denotes binary cross-entropy loss and denotes the sigmoid function. The sum in Eq. is defined over the set of unique edges Se in the tetrahedral grid, for which sign(s;) * sign(s)).
The overall loss function is then defined
L = L(D G) + L( musks G) + p.ivgg
(3)
where u is hyperparameter that controls the level of regularization.
5

---

Cregryy
11
(1)
GJ)
Carggry
(S.
PID
(1)
LFD
CD
LFD
CD
Odi
3D
LED
CD
LTD
CD
On
3D
PeinFlas
Poin [90)
50.68
4023
OceNet [16)
27.20
42.63
1717
0.61
OueNer [S8]
10.14
47.95
4551
2.04
GAN
Che
GRAP (57)
1.57
1.57
0012
10.5
49.95
52.85
Miske
GRAP [50)
4384
50.64
4518
2.40
$3.00
113.99
[X
Ous
66.78
58.39
1491
0.71
10.25
10.25
Ous
67.12
07.12
3631
1.72
05.60
65.60
Ouss (ingroved G7
59.00
47.95
1473
0.81
10,50
10,60
Oues Comproved G3
69.56
65.75
3593
1.79
48,90
48,90
PoinsFow (90)
OccNer [S8]
61.10
67.11
3494
3.98
OcoNet [38]
56.18
75.25
4418
2.39
P1
120.53
30.54
Chai
GRAP
50.23
79.25
4055
6.30
43.82
61.03
Anineal
GRAF [52]
00.67
61.80
5053
4.81
42.07
52.48
|X]
74.16
58.43
Out
00.08
69.91
3167
3.72
23.28
23.28
Ours
79.77
78.65
3798
2,02
28.33
26.33
23.17
74.16
Oues (inquova
71.96
71.96
3125
3.96
22.41
22.41
Oue (impureed (3)
74.16
62.02
3767
1.97
27.18
27.18
Table 2: Quantitative evaluation of generation results: t: the higher the better, 4: the lower the better. The best scores are highlighted in bold. MMD-CU MDDCD scores multiplied by 103. The results of Ours (improved G) were obtained after the review process by improving the design of the generator network architecture G' (see Appendix A.5 for more details).
Animal
Chair
Motorbile
PointFlow
OceNet
PiGAN
GRAF
EG3D
Ours
Ours-tex
Figure Qualitative comparison of GET3D to the baseline methods in terms of extracted 3D geometry, GET3D is able to generate shapes with much higher geometric detail across all categories.
Experiments
We conduct extensive experiments evaluate our model. We first compare the quality of the 3D textured meshes generated by GET3D to the existing methods using the ShapeNet [9] and Turbosquid [4] datasets. Next, we ablate design choices in Sec. 4.2. Finally, demonstrate the ficxibility of GET3D by adapting it to downstream applications in Sec. 4.3. Additional experimental results and implementation details are provided Appendix.
4.1 Experiments on Synthetic Datasets
Datasets For evaluation on ShapeNet [9], we use three categories with complex geometry Car, Chair, and Motorbike, which contain 7497, 6778, and 337 shapes, respectively. We randomly split each category into training (70%), validation (10 %), and test (20 %), and further remove from the test set shapes that have duplicates in the training set. To render the training data, we randomly sample camera poses from the upper hemisphere of each shape. For the Car and Chair categories, we use 24 random views, while for Motorbike we use 100 views due to less number of shapes. As models in ShapeNet only have simple textures, we also evaluate GET3D on an Animal dataset (442 shapes) collected from TurboSquid [4], where textures are more detailed and we split into training, validation and test as defined above. Finally, to demonstrate the versatility of GET3D, we also provide qualitative results on the House dataset collected from Turbosquid (563 shapes), and Human Body dataset from Renderpeople [2] (500 shapes). We train separate model each category.

---

Aninal
Car
Chair
Motorbike
PiGAN
GRAF
EG3D
Ours
Figure 4: Qualitative comparison of GET3D to the baseline methods in terms of generated 2D
images. GET3D generates sharp textures with high level of detail.
Shapes generated by GET3D rendered in Blender. GET3D generates high-quality shapes with diverse texture, high-quality geometry, and complex topology. Zoom-in for details.
Figure 5:
Baselines We compare GET3D groups of works: 1) 3D generative models that rely 3D supervision: PointFlow [59] and OccNet [38]. Note that these methods only generate geometry without texture. 2) 3D-aware image generation methods: GRAF [52], PiGAN [7], and EG3D (8).
Metrics To evaluate the quality of our synthesis, we consider both the geometry and texture of the generated shapes. For geometry, adopt metrics from [5] and use both Chamfer Distance (CD) and Light Field Distance [10] (LFD) to compute the Coverage score and Minimum Matching Distance. For OceNet [38], GRAF [52], PiGAN [7] and EG3D [8], we use marching cubes to extract the underlying geometry. For PointFlow [59], we use Poisson surface reconstruction to convert point cloud into a mesh when evaluating LFD. To evaluate texture quality, we adopt the FID [26] metric commonly used to evaluate image synthesis. In particular, for each category, we render the test shapes into 2D images, and also render the generated 3D shapes from each model into 50k images using the same camera distribution. We then compute FID on the two image sets. As the baselines from 3D-aware image synthesis [52, 7, 8] do not directly output textured meshes, we compute FIID score in two ways: (i) we use their neural volume rendering to obtain 2D images, which we refer to as FID-Ori, and (ii) we extract the mesh from their neural field representation using marching cubes, render it, and then use the 3D location of each pixel to query the network to obtain the RGB values. refer to this score, that more aware of the actual 3D shape, as FID-3D. Further details on the evaluation metrics are available in the Appendix B.3.
Experimental Results We provide quantitative results Table. and qualitative examples in Fig. and Fig. 4. Additional results are available in the supplementary video. Compared to OccNet [38] that
7

---

Figure 6: Shape interpolation. We interpolate both geometry and texture latent codes from left to right.
Figure 7: Shape variation. We locally perturb each latent code to generate different shapes.
uses 3D supervision during training, GET3D achieves better performance in terms of both diversity (COV) and quality (MMD), and our generated shapes have more geometric details. PointFlow [59] outperforms GET3D in terms of MMD on CD, while GET3D is better in MMD on LFG. We hypothesize that this is because PointFlow directly optimizes on point locations, which favours CD. GET3D also performs favourably when compared to 3D-aware image synthesis methods, achieve significant improvements over PiGAN [7] and GRAF [52] in terms
metrics on all datasets. Our generated shapes also contain more detailed geometry and texture. Compared with recent work EG3D [8]. We achieve comparable performance on generating 2D images (FID-ori), while significantly improve on 3D shape synthesis in terms of FID-3D, which demonstrates the effectiveness of our model on learning actual 3D geometry and texture.
Since we synthesize textured meshes, we can export our shapes into Blender'. We show rendering results in Fig. and 5. GET3D is able to generate shapes with diverse and high quality geometry and topology, very thin structures (motorbikes), well complex textures cars, animals, and houses.
Shape Interpolation GET3D also enables shape interpolation, which can be useful for editing purposes. We explore the latent space of GET3D in Fig. 6, where we interpolate the latent codes to generate each shape from left right. GET3D able faithfully generate smooth and meaningful transition from one shape to another. We further explore the local latent space by slightly perturbing the latent codes to random direction. GET3D produces novel and diverse shapes when applying local editing in the latent space (Fig. 7).
4.2 Ablations
We ablate our model in two ways: 1) w/ and w/o volume subdivision, 2) training using different image resolutions. Further ablations are provided in the Appendix C.3.
Ablation of Volume Subdivision As shown in Tbl. 2, volume subdivision significantly improves the performance classes with thin structures (e.g., motorbikes), while not getting gains on other classes. We hypothesize that the initial tetrahedral resolution already sufficient to capture the de- tailed geometry on Chairs and Cars, and hence the subdivision cannot provide further improvements.
COV (%t. 7)
MMD (1)
Class
Img Res
FID (1)
LFD
LFD
128"
9.28
8:25
2224
1.30
39.21
Car
5122
52.32
44.13
1593
0.50
13.19
10244
10.25
128
38.25
33.98
3886
5.90
13.04
Chair
5122
10242
69.08
67.87
3167
3,74
23.28
Mbiie
1024
1122
68.49
65.75
3421
1.74
74.04
Animal
3122
77.53
78.65
3W28
2.01
29.75
Ablating Different Image Resolutions We ab- late the effeect of the training image resolution in Tbl. 3. As expected, increased image resolution
1024
79.78
3798
28.33
Table 3: Ablating the image resolution. f: higher
is better, J: lower is better.
We use xatlas [61] to get texture coordinates for the extracted mesh, from where we can warp our mesh into a 2D plane and obtain the corresponding 3D location on the mesh surface for any position on the 2D plane. We then discretize the 2D plane into an image, and for each pixel, we query the texture field using corresponding 3D location to obtain the RGB color get the texture map.

---

Base Color
Roughness
Metallic
Normal
Relight
Specular
Relight
Specular
Figure 8: Material generation and relighting. Despite being unsupervised, our model generates reasonable material properties, and can be realistically rendered with real-world HDR panoramas (bottom right). Normals are computed from the generated mesh. Note how specular effects change under two different lighting conditions.
improves the performance in terms of FID and shape quality, as the network can see more details, which are often not available in the low-resolution images. This corroborates the importance of training with higher image resolution, which are often hard make use of for implicit-basee methods.
4.3 Applications
4.3.1 Material Generation for View-dependent Lighting Effects
GET3D can easily be extended to also generate surface materials that are directly usable modern graphics engines. In particular, we follow the widely used Disney BRDF [6, 29] and describe the materials in terms of the base color (RR*), metallic (RR), and roughness (IE) properties. As result, we repurpouse our texture generator to now output a 5-channel reflectance field (instead of only RGB). To accommodate differentiable rendering of materials, we adopt an efficient spherical Gaussian (SG) based deferred rendering pipeline [12]. Specifically, we rasterize the reflectance field into G-buffer, and randomly sample an HDR image from a set of real-world outdoor HDR panoramas Stight { Lsa) K. where Lsg E I is obtained by fitting 32 SG lobes to each panorama. The SG renderer [12] then uses the camera e to render an RGB image with view-dependent lighting effects, which we feed into the discriminator during training. Note that GET3D does not require material supervision during training and learns to generate decomposed materials in an unsupervised manner.
We provide qualitative results of generated surface materials in Fig. 8. Despite unsupervised, GET3D discovers interesting material decomposition, e.g., the windows are correctly predicted with smaller roughness value to be more glossy than the car's body, and the car's body is discovered as more dielectric while the window more metallic. Generated materials enable us to produce realistic relighting results, which can account for complex specular effects under different lighting conditions.
4.3.2 Text-Guided 3D Synthesis
Similar to image GANs, GET3D also supports text-guided 3D content synthesis by fine-tuning pre-trained model under the guidance of CLIP [51]. Note that final synthesis result is textured 3D mesh. To this end, we follow the dual-generator design from StyleGAN-NADA [19], where a trainable copy G2 and frozen copy G5 of the pre-trained generator are adopted. During optimization G, and Gy both render images from 16 random camera views. Given a text query, we sample 500 pairs of noise vectors 2 and Z2. For each sample, we optimize the parameters of G, to minimize the directional CLIP loss [19] (the source text labels are "car", "animal" and "house" for the corresponding categories), and select the samples with minimal loss. To accelerate this process, we first run a small number of optimization steps for the 500 samples, then choose the top 50 samples with the lowest losses, and run the optimization for 300 steps. The results and comparison against SOTA text-driven mesh stylization method, Text2Mesh [39], are provided in Fig. 9. Note that, [39] requires mesh of the shape as an input to the method. We provide our generated meshes from the frozen generator input meshes to it. Since it needs mesh vertices to be dense to synthesize surface details with vertex displacements, we further subdivide the input meshes with mid-point subdivision to make sure each mesh has 50k- 150k vertices average.

---

"burned
car"
"polygon
car"
"fuffyy
animal"
"tiger"
"haunted house"
Text2Mesh
Our results
Figure 9: Text-guided 3D synthesis. Note that Tex12Mesh [39] requires 3D mesh geometry as input. To fulfil the requirement, we provide our generated geometry its input mesh.
Conclusion
We introduced GET3D, novel 3D generative model that able to synthesize high-quality 3D textured meshes with arbitrary topology. GET3D is trained using only 2D images as supervision. We experimentally demonstrated significant improvements on generating 3D shapes over previous state-of-the-art methods multiple categories. We hope that this work brings one step closer to democratizing 3D content creation using A.I..
Limitations While GET3D makes significant step towards a practically useful 3D generative model of 3D textured shapes, it still has some limitations. In particular, we still rely on 2D silhouettes as well as the knowledge of camera distribution during training. As a consequence, GET3D was currently only evaluated on synthetic data. A promising extension could use the advances in instance segmentation and camera pose estimation mitigate this issue and extend GET3D real-world data. GET3D is also trained per-category; extending it to multiple categories in the future, could help us better represent the inter-category diversity.
Broader Impact We proposed novel 3D generative model that generates 3D textured meshes, which can be readily imported into current graphics engines. Our model is able to generate shapes with arbitrary topology, high quality textures and rich geometric details, paving the path for democratizing A.I. tool for 3D content creation. As machine learning models, GET3D is also prone to biases introduced in the training data. Therefore, abundance of caution should be applied when dealing with sensitive applications, such as generating 3D human bodies, as GET3D is not tailored for these applications. We do not recommend using GET3D if privacy or erroneous recognition could lead to potential misuse or any other harmful applications. Instead, we do encourage practitioners to carefully inspect and de-bias the datasets before training our model depict a fair and wide distribution of possible skin tones, races or gender identities.
Disclosure of Funding
This work funded by NVIDIA. Jun Gao, Tianchang Shen, Zian Wang and Wenzheng Chen acknowledge additional revenue in the form of student scholarships from University of Toronto and the Vector Institute, which are not in direct support of this work.
References
[1] Autodesk Maya, http::://ww...ttooeeeteett http::/wwwwttt...oppp/pppcc////e Accessed: 2022-05-19.
[2] Renderpeople, http:///h//////ppp//////eeeepppp com/ Accessed: 2022-05-19.
[3] Sketchfab, https::/skettchhabbbb com// Accessed: 2022-05-19.
[4] Turbosquid by Shutterstock, http:://cckkhtttss/////s http://.wwbbouuuddotoo com/ Accessed: 2022-05-19.
[5] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference machine learning, pages 40-49. 49, PMLR, 2018.
10

---

[6] Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. ACM SIGGRAPH, volume 2012. pages 1-7. vol. 2012, 2012.
[7] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021.
[8] Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay. Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 16123-16133, 2022.
[9] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1r1.v.111..0312,, 2015.
[10] Ding-Yon Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retricval. In Computer graphics forum, volume 22, pages 223-232. Wilcy Online Library, 2003.
[11] Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation baaiattooo---edddddee differentiable renderer. In Advances In Neural Information Processing Systems, 2019.
[12] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[13] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[14] Blender Online Community. Blender 3D modelling and rendering puckage. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[16] Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE TRANSACTIONS on Information and Systems, 74(1):214-224, 1991.
[17] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[18] Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In 2017 International Conference on 3D Vision (3DV), pages 402-411. 1. IEEE, 2017.
[19] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan- nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 113) 2022.
[20] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacohson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020.
[21] Jun Gao, Chengcheng Tang, Vignesh Suapaathi-SSuaaaaniinn Jiahui Huang, Hao Su, and Leonidas J Guibas. Deepspline: Data-driven reconstruction of parametric curves and surfaces. arXiv preprint
MXXiv:191.03378/ 2019.
[22] Jun Gao, Zian Wang, Jinchen Xuan, and Sanja Fidler. Beyond fixed grid: Learning geometric image representation with deformable grid. In European Conference on Computer Vision, pages 108-125. Springer, 2020.
[23] Jiatao Gu, Lingjic Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022.
[24] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yi Yu Liu. GANeraft: Unsupervised 3D Neural Rendering of Minecraft Worlds. In ICCV, 2021.
[25] Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In The IEEE International Conference Computer Vision (ICCV), October 2019.
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[27] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming- Yu Liu. Multimodal conditional image synthesis with -exeeettoof-epperts GANs. In ECCV, 2022.
[28] Moritz lbing, Gregor Kobsik, and Leif Kobbelt. Octree transformer: Autoregressive 3d shape generation hierarchically structured sequences. arXiv preprint arXiv:2111.12480, 2021.
11

---

[29] Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 4(3), 2013.
[30] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alies-free generative adversarial networks. In Proc. NeurlPS, 2021.
[31] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 1401-4410, 2019.
[32] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[33] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for perforrefor differentiable rendering. ACM Transactions on Graphics, 39(6), 2020.
[34] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 2144):163-169 1987.
[35] Sehastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and Nate Kushman. Inverse graphics gan: Learning to generate 3d shapes from unstructured 2d data. arXiv preprint urXiv:2002.12672 2020.
[36] Andrew Luo, Tianqin Li, Wen-Hao 7hang, and Tai Sing Lee. Surfgen: Adversarial 3d shape synthesis with explicit surface discriminators. In Proceedings of the IEFE/CVF International Conference Computer Vision, pages 16238-16662 16248, 2021.
[37] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In International Conference on Machine Learning (ICML). 2018.
[38] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460 4470, 2019.
[39] Oscar Michel, Roi Bar-On. Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492-13502, 2022.
[40] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[41] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Lconidas Guibas. Structurenct: Hicrarchical graph networks for 3d shape generation. ACM Transactions on Graphics (TOG), Siggraph Asia 2019, 38(6)ee 242, 2019.
[42] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280 8290, 2022.
[43] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. ICML, 2020.
[44] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural
feature fields. In Proc. IEEE Conf. on Computer Vision and Pottern Recognition (CVPR), 2021.
[45] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531 145400 2019.
[46] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher Shhiaccemnnrhiizmaaa...... Stylesdf: High-resolution 3d-consistent image and geometry generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13503-13513, 2022.
[47] Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Yan 7hu. Semantic image synthesis with spatially- adaptive normalization. In Proceddings of the IEFE Conference on Computer Vision and Pattern Recogni- tion, 2019.
[48] Dario Pavllo. Jonas Kohler, Thomas Hofmann, and Aurelien Lucchi. Learning generative models of textured 3d meshes from real-world images. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[49] Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurelien Lucchi. Convolu- tional generation of textured 3d meshes. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[50] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), 2020.
12

---

[51] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Muchine Learning, pages 3748.8763. PMLR, 2021.
[52] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for al-ae image synthesis. Advances Neural Information Processing Systems (NeurlPS), 2020.
[53] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. ARXIV, 2022.
[54] Tianchang Shen, Jun Gao, Amlan Kar, and Sanja Fidler. Interactive annotation of 3d object geometry using 2d scribbles. In European Conference on Computer Vision, pages 751-767. Springer, 2020.
[55] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yus Yu Liu, and Sanja Fidler. Deep marching tetrahedra:
hybrid representation for high-resolution 3d shape synthesis. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[56] Edward Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning, pages 87-96. PMLR, 2017.
[57] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d generaaive-advvesariaal modeling. Advances in neural information processing systems, 29, 2016.
[58] Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. In CVPR, 2022.
[59] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongic, and Bharath Hariharan. Point- flow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4541-4550, 2019.
[60] Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 3dstylenet: Creating shapes with geometrie and texture style variations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12456-12465, 2021.
[61] Jonathan Young. xatlas, 2021.
[62]
Linqi Zhou, Yilun Du and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5826-5835, 2021,
[63] Peng Zhou, Lingxi Xie, Binghing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on conditionally-ii inndpeeee pixel synthesis. arXiv preprint ar 100.9981, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We provide extensive experiments in Sec.
(b) Did you describe the limitations of your work? [Yes] We provide the discussion on limitations in Sec. 5, and fallure cases in Appendix.
(c) Did you discuss any potential negative societal impacts your work? [Yes] We provide the discussion in the Broader Impact paragraph with further discussions in the main paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] We have htt////ttaaaaaaa/a///
released the code here /ttpps//gitaii
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] We provide training details in both See, the main paper and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] Training existing 3D models, including ours, on large-scale 3D datasets is too computation costly to repeat multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide in the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used ShapeNet [9] core dataset, Turbosquid [4], Renderpeople [2] in Sec. 4.1. We also used official code to reproduce baselines with citations. In particular, OccNet [38], PiGAN [7]. Pointflow [59]. Eg3D [8].
(b) Did you mention the license of the assets? [Yes] We provided the license of ShapeNet, Turbosquid and RenderPeople the Appendix.
(c) Did you include any new assets either in the supplemental material or as URL? [No] We did not include new assets in Supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We provide discussion this in the Appendix.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] We provide discussion on this in Appendix.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ZzGCMQ-7anQ", "reviewer": "Reviewer_vLJX", "summary": "This paper proposed GET3D, a generative model that synthesizes high-quality 3D meshes with textures.\nThe authors put the recent 3D shape reconstruction work [40] into a styleGAN architecture and demonstrated that this framework is capable to generate diverse 3D meshes with various textures. The experiments on multiple datasets show the proposed method is superior to the prior arts.", "questions": "I do not have major concerns. Just some minor questions and comments as listed in the weakness section.\n\nOverall I think this is a high-quality paper. The idea is straightforward and the results are convincing. Though I have some minor concerns, I lean into accepting this paper.", "limitations": "The authors listed two major limitations in their work: 1. the evaluation was done on the synthetic data only and 2. per category training. The authors also discussed the potential social impact of their work applying on privacy or biased data. All these sound sufficient to me.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The writing is clear and the idea is easy to follow.\n\nThe authors closely followed the most recent 3D mesh reconstruction work nvdiffrec [40] and developed further to incorporate the 3D reconstruction pipeline into a GAN. I am glad to see that without many bells and whistles, the proposed GET3D works pretty well on multiple datasets.\n\nThe experiments are sufficient overall, though I do have some minor comments.\n\nWeakness:\n1. It is great that the generated 3D textured meshes can be directly used in the production tools like Blender. I wonder how the texture is represented in the final output as I don't think these tools can utilize texture fields? Is it easy to get the UV map from the proposed model? Or does \"texture\" only mean \"colored vertices\"?\n2. Based on Table 1, [46] actually is closer to what the authors proposed in this paper (without the texture part though). Why [46] is excluded from the comparison?\n3. What is the illumination model in the differentiable rendering part? If I understand correctly, it is a Phone model with only 1.0 ambient lighting.\n4. Though in L255, the authors claimed that \"hence the subdivision cannot provide further improvements\", in Table 2, subdiv actually hurts the performance seriously. Any idea?\n5. Shape interpolation is impressive. It would be great to see the texture interpolation as well (though the title of Fig 6 is shape interpolation, the texture has also changed...why...).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mcwlix9NNcg", "reviewer": "Reviewer_LrUW", "summary": "- A generative model of 3D models for 3D content creation should output a mesh of detailed geometry and any topology and must be trained using 2D images without 3D supervision. The proposed method is the first that has these properties. The proposed realizes it by combining a 3D representation of meshes with any topology (DMTet) [51] (S.3.1.1), texture fields [43] (S.3.1.2), differentiable rendering [31], and StyleGAN [29] (S.3.2).\n- The experiments conducted on ShapeNet, Turbosquid, and Renderpeople demonstrate that the variety and quality of generated 3D models by the proposed method are better than existing 3D mesh generation models and 3D-aware image generation models. Also, the proposed method is applied to (1) unsupervised decomposition of geometry, light, and material, and (2) text-guided 3D model generation.", "questions": "It is written that it is difficult to extract meshed textures from neural fields (l.45, l.85), but what exactly is the difficulty? That is understandable when volume rendering is used, but when SDF is used, I guess the difficulty is the same as the proposed method using texture fields.", "limitations": "It is described that it is difficult to apply the proposed method to natural datasets because the distribution of camera poses and silhouettes images are assumed to be known. No other serious limitations were found.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strength\n\n- The proposed method can handle topology changes as it uses DMTet as a 3D representation, different from The existing generative model of meshes [46]. It is a significant step toward realistic 3D content generation models.\n- Comparison with existing 3D generation methods [46,11] are conducted on multiple datasets, and quantitative improvement is certain in the geometry diversity and visual quality metrics. Also, it is a bit surprising that the proposed method generates better images than recent 3D-aware image generation models inspired by NeRF.\n- Qualitative results are impressive and clearly outperform the baselines. Improvement of texture is explained by the image resolution in training (Table 3), which is also an interesting finding.\n- Though no quantitative comparison is provided, the quality of text-guided 3D model generation is significantly better than Text2Mesh [37].\n\n#### Weakness\n\n- The proposed method is a straightforward combination of existing methods [51,43,31,29], and each component is technically not novel, although what is realized by the combination is novel.\n- Training the proposed method requires the distribution of camera parameters and silhouette images, so it practically requires a synthetic dataset of 3D models, as noted in l.300. Therefore, the cost of training datasets is still not very lower than existing methods that require 3D supervision.\n- The generated shape and texture look a bit coarse when zoomed up, though they are finer than [46,11].\n- DMTet is not very good at representing high-resolution scenes as it stores information in 3D grids. Therefore, it is difficult by the proposed method to generate a large-scale scene containing multiple objects, such as bedrooms, unlike 2D GANs.\n- The result of unsupervised geometry, lighting, and material decomposition look appealing, but I am not very sure the proposed method is actually good because quantitative comparison is not provided.\n\nAlthough the technical contribution is not very strong, I recommend acceptance because of the impressive results shown in the experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JUdeoF7_cMs", "reviewer": "Reviewer_n6SS", "summary": "This paper proposes a 3D textured mesh generator trained using 2D images, known camera distributions and object silhouettes. The generator is composed of a geometry generator, which generates the SDF value and vertice deformations to a tetrahedron field. The mesh is then extracted using differentiable marching tets. The texture generator generates a texture field, which predicts the reflectance of a point in space using its coordinates and the latent codes. By differentialbly rendering them into images and sillhouettes, image-based discriminator is used to provide the signal for training. \nIn addition, the authors also presented two variants, first is generating the material maps instead of reflectance, where the generated objects can be relit. Second, using a clip discriminator, the generator can be tailored to perform the task of text-to-textured-mesh. ", "questions": "In general I find this paper above the acceptance bar. The following questions are intended to elicit more insights and details of the proposed methods:\n\n1. How robust is the proposed method to distribution shifts of camera poses? Since it samples from the distribution of camera poses, does it really matter to have access to the exact camera pose, which seems to be a limitation of the method to apply to real images?\n\n2. In addition to camera pose, there's also the matter of camera intrinsics. For real images, those are usually unknow and varying as well. Will the method robust enough to handle that?\n\n3. How delicate is the system to train successfully? Though GANs are hard to tune overall, it would be nice to have some insights on how easy it is for others to re-train the system on different categories.", "limitations": "The authors provided adequate discussions on the limiation of the method.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nOriginality:\n\nEven though the core compnent of the system is based on prior works(nvdffrast, tet marching), the system itself is novel and nontrivial to setup. As far as I know, the proposed system is the first of its kind to generate high-quality textured mesh. \n\nSignificance:\n\nThe proposed system tackels multiple drawbacks of previous results; it uses mesh as the final geometric representation, but also not constrained by topology nor utilizes a template mesh. The quality of the result is also appealing, making it a promising candidate for generating assets for graphics applications. \n\nQuality:\n\nThis paper is of high quality. In addition to technical details, this paper presents multiple experiments with two variants of the method to support the validity of their design choices and the potential of the method. \n\nClarity:\n\nThis paper is well written and easy to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I2X9p8gxZt", "reviewer": "Reviewer_9TSe", "summary": "This paper proposes a generative method to produce textured 3D mesh that can be directly used in general 3D rendering engines. The overall pipeline contains a geometry generator, a texture generator, a differentiable rendering module, and two discriminators. The geometry generator exploits DMTet as surface representation, and a tri-plane texture field is used for the texture generator.  Besides, the differentiable module is used to predict the 2D silhouette and 3D on-surface coordinates that are used to query the corresponding RGB color in the texture field. The two discriminators are inspired by StyleGAN for silhouette and rendered RGB images to achieve RGB-supervised training, respectively.  In general, this paper is a combination of DMTet, texture field, NVdiffrec, and StyleGAN. While the performance excell SOTA method, the novelty is limited. ", "questions": "Please provide some discussion about the weakness ", "limitations": "Yes ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this work is the generated texture mesh can be directly used in general 3D rendering engines, which is not suitable for previous works. Meanwhile, the supervision is 2D images that are more widely available than 3D geometry data. \nThe weakness of this work contains:\n1. Although this work only requires 2D images as supervision, all experiments are trained with rendered RGBs with known camera parameters and light variation. The application for in-the-wild RGB images needs more discussion. Hence the application scenario of this method is limited. \n2. The author claims their method can generate high-quality 3D textures meshes, but local details (e.g. face) of the human character are missed (as in Fig.1, Fig.5). A qualitative comparison of local detail with previous SOTA methods (e.g. EG3D) will make this claim more valid. \n3. The lack of quantitative comparison of human character. \n\nTypo: \nP7, L222, \"Tbl. 2\". \nSupp. P3, L101, \"Fret\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept