Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape
Authors: Rundi Wu
Ruoshi Liu
Carl Vondrick
Changxi Zheng
Venue: ICLR 2024 poster (2024)
Keywords: 3D generation
shape analysis and synthesis
diffusion models
Primary area: generative models
TL;DR: A diffusion model trained on a single 3D textured shape to produce high-quality variations.

Abstract: Synthesizing novel 3D models that resemble the input example as long been pursued by graphics artists and machine learning researchers. In this paper, we present Sin3DM, a diffusion model that learns the internal patch distribution from a single 3D textured shape
and generates high-quality variations with fine geometry and texture details. Training a diffusion model directly in 3D would induce large memory and computational cost. Therefore, we first compress the input into a lower-dimensional latent space and then train a diffusion model on it. Specifically, we encode the input 3D textured shape into triplane feature maps that represent the signed distance and texture fields of the input. The denoising network of our diffusion model has a limited receptive field to avoid overfitting, and uses triplane-aware 2D convolution blocks to improve the result quality. Aside from randomly generating new samples, our model also facilitates applications such as retargeting, outpainting and local editing. Through extensive qualitative and quantitative evaluation, we show that our method outperforms prior methods in generation quality of 3D shapes.

Published as a conference paper at ICLR 2024
SIN3DM: LEARNING A DIFFUSION MODEL FROM A SINGLE 3D TEXTURED SHAPE
Rundi Wu, Ruoshi Liu, Carl Vondrick, Changxi Zheng
Columbia University
frundi, , r1 rliu, vondrick, exz exx] co umb i edu
https: wtdmm.nn.mmmmmmmmmmmmmmmmmmm.m.m wit.u..iio.oo
ABSTRACT
Synthesizing novel 3D models that resemble the input example has long been pursued by graphies artists and machine learning researchers. In this paper, we present Sin3DM, diffusion model that learns the internal patch distribution from a single 3D textured shape and generates high-quality variations with fine geom- etry and texture details. Training a diffusion model directly in 3D would induce large memory and computational cost. Therefore, we first compress the input into lower-dimensional latent space and then train diffusion model on it. Specifi- cally, we encode the input 3D textured shape into triplane feature maps that rep- resent the signed distance and texture fields of the input. The denoising network of our diffusion model has a limited receptive field to avoid overfitting, and uses triplane-aware 2D convolution blocks improve the result quality. Aside from randomly generating new samples, our model also facilitates applications such retargeting, outpainting and local editing. Through extensive qualitative and quantitative evaluation, we show that our method outperforms prior methods in generation quality of 3D shapes.
INTRODUCTION
Creating novel 3D digital assets is challenging. It requires both technical skills and artistic sensi- bilities, and is often time-consuming and tedious. This motivates researchers to develop computer algorithms capable of generating new, diverse, and high-quality 3D models automatically. Over the past few years, deep generative models have demonstrated great promise for automatic 3D content creation (Achlioptas et al. 2018: Nash et al. [020)] Gao et al. (022). More recently, diffusion models have proved particularly efficient for image generation and further pushed the frontier 3D generation (Gupta et al. 2023 Wang et al. 2022b).
These generative models are typically trained on large datasets. However, collecting large and diverse set of high-quality 3D data, with fine geometry and texture, is significantly more challenging than collecting 2D images. Today, publicly accessible 3D datasets (Chang et al. 2013: Deitke et al.
Single training shape
Generated shapes
Figure I: Trained on a single 3D textured shape (left), Sin3DM is able to produce a diverse new sam- ples, possibly of different sizes and aspect ratios. The generated shapes depict rich local variations with fine geometry and texture details, while retaining the global structure of the training example. Top: acropolis (choly kurd 2021); bottom: industry house (Lukas camota 2015).

---

Published as a conference paper at ICLR 2024
2022
remain orders of magnitude smaller than popular image datasets (Schuhmann et al. 2022) insufficient to train production-quald 3D generative models. In addition, many artistically designed 3D models possess unique structures and textures, which often have no more than one instance to learn from. In such cases, conventional data-driven techniques may fall short.
In this work, we present Sin3DM, a diffusion model that only trains on a single 3D textured shape. Once trained, our model is able to synthesize new, diverse and high-quality samples that locally resemble the training example. The outputs from our model can be converted to 3D meshes and UV-mapped textures (or even textures that describe physics-based rendering materials), which can be used directly in modern graphies engine such as Blender (Community (0018) and Unreal En- gine (Epic Games, (019). We show example results in Fig. [and include more in Sec. 4 Our model also facilitates applications such as retargeting, outpainting and local editing.
We aim to train a diffusion model on a single 3D textured shape with locally similar patterns. Two key technical considerations must be taken into account. First, we need an expressive and memory- efficient 3D representation. Training diffusion model simply on 3D grids would induce large memory and computational cost. Second, the receptive field of the diffusion model needs to be small, analogously to the use of patch discriminators in GAN-based approaches (Shaham et al.) 2019). A small receptive field forces the model to capture local patch features.
The training process of our Sin3DM consists of two stages. We first train autoencoder to compress the input 3D textured shape into triplane feature maps (Peng et al.] 2020), which are three axis- aligned 2D feature maps. Together with the decoder, they implicitly represent the signed distance and texture fields of the input. Then we train a diffusion model on the triplane feature maps to learn the distribution of the latent features. Our denoising network is 2D U-Net with only one-level of depth, whose receptive field is approximately 40% of the feature map size. Furthermore, we enhance the generation quality by incorporating riplane-aware 2D convolution blocks, which consider the relation between triplane feature maps. At inference time, we generate new 3D textured shapes by sampling triplane feature maps using the diffusion model and subsequently decoding them with the triplane decoder.
To our best knowledge, Sin3DM is the first diffusion model trained on a single 3D textured shape. We demonstrate generation results on various 3D models of different types. We also compare to prior methods and baselines through quantitative evaluations, and show that our proposed approach achieves better quality.
2 RELATED WORK
3D shape generation Since the pioneering work by (Funkhouser et al. 2004). data-driven methods for 3D shape generation has attracted immense research interest. Early works this direction follow synthesis-by-analye
approach (Merrell [007 Booeloh et al. 2010 Kalogerakis al.] 2012 2, Xu 2012). After the introduction of deep generative networks such as GAN (Goodfellow et al.- researchers start to develop deep generative models for 3D shapes. Most of the existing works primarily focus on generating 3D geometry of various representations, including voxels (Wul
jet al.
2014
et al.
2016; Chen et al. 2021), point clouds (Achlioptas al. 2018,, Yang al. 20T9: Cai a
2020
Li Li al.]|220
meshes (Nash et al. 2020 Pavllo et al. (021). implicit fields (Chen & Zhang
2019
Park | Mescheder et al.
2019 Liu al. 2022 Liu & Vondrick] 2023), structural
primitives (Li et al. [2177] 7/ et al. 2019; Jones et al. 2020 and parametric models (Chn et al.,
2020
Wu et al. 2021; Payaramaa et (022). Some recent works take step forward to generate
3D textured shapes (Gupta et al. [023] Gao et al. 2022, Nichol et al. 2022, Jun & Nichol 2023).
All these methods rely on large 3D dataset for training. Yet, collecting high-quality 3D dataset
is much more expensive than images, and many artistically designed shapes have unique structures that are hard to learn from a limited collection. Without the need of large dataset, from merely a single 3D textured shape, our method is able to learn and generate its high-quality variations.
Another line of recent works (Poole et al. 2022 Lin et al. 2023 Wang et al., 2022al Metzer et al.) 2022 Liu et al.) 2023) use gradient-based optimization produce individual 3D models by lever- aging differentiable rendering techniques (Mildenhall et al. 2020 Laine et al., 2020) and pretrained text-to- -magt generation models (Rombach et al. (2022). However, these methods have long in- ference time due to the required per-sample optimization, and the results often show high saturation artifact. They are unable to generate fine variations of an input 3D example.
2

---

Published as a conference paper at ICLR 2024
Triplane Encoding
Venc
Triplana latont h
Vdec
along
AvgPool
2D ReeBlock
Input
2-99i5
BD Cooon
AvgPool
2D ResBlock
aalong x-axis
AvgPool
2D ResBlock
Interpolation
hy
h
h t Po
ho
O)
SDF MLP
Tex MLP
Po
Pa
Pe
Latent Diffusion Model
p))
c(p)
Figure 2: Method overview. Given an input 3D textured shape, we first train triplane auto-encoder to compress into an implicit triplane latent representation h. Then we train a latent diffusion model on it to learn the distribution of triplane features. See Fig. [3 for the structure of our denoising network po- inference time, we sample a new triplane latent using the diffusion model and then decode it to a new 3D textured shape using the triplane decoder docco
Single instance generative models The goal of single instance generative models to learn the internal patch statistics from a single input instance and generate diverse new samples with similar local content. Texture synthesis an important use case for such models and has been an active area of research for more than a decade Efros & Leung (1999); Han et al. (0008: Zhou et al. (2018);
Niklasson et al. 2 Rodriguez- Pardo Garees (2022). Beyond stationary textures, the seminal work SinGAN (Shaham et al. 2019) explores this problem by training a hierarchy of patch GANs (Goodfellow et al. 2014) on an pyramid of a nature image. Many follow-up works improve upon from various perspectives (Hinz et al.) 2021 Shocher et al. 2019; Granot et al. [222 Zhang (et 2021). Some recent works use a diffusion model for single image generation by limiting the
receptive field of the denoising network (Nikankin et al. [2022] Wang et al. 2022c) constructing
multi-scale diffusion process (Kulikov et al. 2022). Our method is inspired by the above generative
models trained on single image.
The idea of single image generation has been extended to other data domains, such as videos (Haim al. 202 Nikankin al. 2022), audio (Greshler al. (022), character motions (Li et al. 2022
Raab el al.
2023). 3D shapes (Hertz et al., 2020 Wu Wu & Zheng (2022) and radiance fields (Karnewar
al.j 2022] Son et al. 2022 Li al. 2023). In particular, SSG (Wu Zheng, 2022) is the most
relevant prior work. It uses scati-sale GAN architecture and trains a voxel pyramid of the input 3D shape. However, it only generates un-textured meshes (i.e., the geometry only) and the geometry quality is limited by the highest training resolution of the voxel grids. By encoding the input 3D) textured shape into neural implicit representation, our method is able to generate textured meshes with high resolutions for both geometry and texture.
Li et al. (2023) uses a patch matching approach to synthesize 3D scenes from a single example. It specifically focuses on 3D natural scenes represented as grid based radiance fields (Plenoxels (Fridovich- Keil et al. 2022)), ) which cannot be casily re-lighted and we found its converted meshes often broken.
Diffusion models Diffusion models (Sohl- Dickstein et al. 2015 Ho et al. 2020 Song et al. 2020
class of generative models that use a stochastic diffusion process to generate samples matching the real data distribution. Recent works with diffusion models achieved state-of-the-at art performance for image generation (Rombach et al.) 2022 Dhariwal & Nichol, 2021 Saharia al.) 2022 Ramesh eet al. 2022). Following the remarkable success in image domain, researchers start to extend diffu- sion models 3D (Zeng et al. 2022 Nichol et al. 2022 Cheng al. 2022 Shue et al. 2022 Gupta et al. 2023 Anciukevičius et al. 2023 Karnewar et al. 2023) and obtain better performance than prior GAN- dataset such as ShapeNet (Chang et al.) 2013) and Objaverse (Deitke et al. [0222] 2023). In con- trast, we explore the diffusion model trained on single 3D textured shape to capture the patch-level variations.
methods. These works typically train diffusion models on a large scale 3D shape
3

---

Published as a conference paper at ICLR 2024
3 METHOD
Overview Sin3DM learns the internal patch distribution from a single textured 3D shape and gen- erates high-quality variations. The core of our method is denoising diffusion probabilistic model (DDPM) (Ho et al. 2020). The receptive field of the denoising network is designed to be small, analogously the use of patch discriminators in GAN-based approaches (Shaham al.) 2019). With that, the trained diffusion model is able to produce patch-level variations while preserving the global structure (Wang et al. 2022c Nikankin et al. 2022).
Directly training diffusion model on a high resolution 3D volume is computationally demanding. To address this issue, first compress the input textured 3D mesh into a compact latent space, and then apply diffusion model in this lower- dimenssiaa space (see Fig. 2). Specifically, we encode the geometry and texture of the imput mesh into an implicit triplane latent representation (Peng et al.j 2020 Chan et al. (02T). Given the encoded triplane latent, we train diffusion model on it with denoising network composed of triplane-aware convolution blocks. After training. we can generate a new 3D textured mesh by decoding the triplane latent sampled from the diffusion model.
3.1 TRIPLANE LATENT REPRESENTATION
To train a diffusion model on a high resolution textured 3D mesh, we need a 3D representation that is expressive, compact and memory efficient. With such consideration,
adopt the triplane representation (Peng et al. 2020; Chan et al., (022)) to model the geometry and texture of the input 3D mesh. Specifically, we train an auto-encoder to compress the input into triplane representation.
Given a textured 3D mesh M. we first construct a 3D grid of size H x W X D to serve as the input to the encoder. At each grid point p. we compute its signed distance d(p) the mesh surface and truncate the value by threshold ea. For points whose absolute signed distance falls within this threshold, we set their texture color d(p) E TR'S to be the same as the color of the nearest point on the mesh surface. For points outside the distance threshold, we assign their color values to be zero After such process, we get 3D grid G E R x of truncated signed distance and texture values. In our experiments, we set max W, D) 256.
Next, we use an encoder Vhene to encode the 3D grid into a triplane latent representation h Venn( (G,), which consists of three axis-aligned 2D feature maps
h (hays h hys),
where hey E CC''''xW' has E RC'H'x and hys E RCCW''D' >D'' with C' being the number of channels. H', W" D' are the spatial dimensions of the feature maps. The encoder Venc composed of one 3D convolution layer and three average pooling layers for the three axes, as illustrated in Fig. 2
The decoder thaer consists of three 2D ResNet blocks (wader dee We and two separate MLP heads (wige vilex) ) for decoding the signed distances and texture colors, respectively. Consider a 3D position E R². The decoder first refines the triplane latent h using 2D ResNet blocks and then gather the triplane features at three projected locations of p.
Say = interp (w (hay); Pay); dec
f== interp (hæz), Paz);
fø= interp dec (hyz), Pyz),
where interp(:: 9) performs bilinear interpolation of 2D feature map at position q. The interpolated features are summed and fed into the MIP heads to predict the signed distance d and color é,
d(p) Vaee [fay + fax + fyz),
(3)
a(p) = vees (fay + fax + syz).
We train the auto-encoder using a reconstruction loss
((v) = [pppp - d(p) +
(4)
where P is point set that contains all the grid points and 5 million points sampled near the mesh surface. d(p) and c(p) are the ground truth signed distance and color at position p, respectively.
4

---

Published as a conference paper at ICLR 2024
TriplaneConv
ResBlock
ConvXY
TripleneConv
U-Net Pa
GroupNorm SilU
Shift and Scale
h
ConvXZ
Triplaneoovv
GroupNorm
ConvYz
h,
average pooling
expand to 2D
(a) U-Net structure
(b) TriplaneConv block
Figure 3: Left: denoising network structure. Our denoising network is a fully convolution U-Net composed of four ResBlocks and its bottleneck downsamples the input by 2. Right: triplane-aware convolution block. A TriplaneConv block considers the relation between triplane feature maps. Inside ConvXY, we apply axis-wise average pooling to h and hyz. yielding two feature vectors, which are then expanded to the original 2D dimension by replicating along y(or x)) axis. The two expanded 2D feature maps are concatenated with hzy and fed into regular 2D convolution layer.
Note that a seemingly simpler option is to follow an auto-decoder approach (Park et al. 2019), i.e., optimize the triplane latent h directly without encoder. However, we found the resulting triplane latent h to be noisy and less structured, making the subsequent diffusion model hard to train. An encoder naturally regularizes the latent space.
3.2 TRIPLANE LATENT DIFFUSION MODEL
After compressing the input into compact triplane latent representation, train denoising diffu- sion probabilistic model (DDPM) (Ho al. 2020) to learn the distribution of these latent features.
At high level, diffusion model is trained to reverse Markovian forward process. Given triplane latent hg = (hæy. hzz, hys), the forward process q gradually adds Gaussian noise to the triplane features, according to variance schedule (3.)T-0
(h. h = (h. V 3,, 3.I).
(5)
The noised data at step t can be directly sampled a closed form solution h, = Va,h + VI & where E is random noise drawn from (OO,1) I) and āg == II a. = II (3)) With large enough T, ht approximately random noise drawn from (0.1).
B.).
A denoising network Pe is trained to reverse the forward process. Due to the simplicity of the data distribution of single example, instead of predicting the added noise c, we choose predict the clean input and thus train with the loss function
(()) = E, ho - (o, t)112
(6)
Denoising network structure. A trriihhtfforaad option for the denoising network is to use the original U-Net structure from (Ho et al. 2020), treating the triplane latent as images. However, this leads to "overfitting", namely the model can only generate the same triplane latent the input hg- without any variations. In addition, it does not consider the relations between the three axis- alis-ed feature maps. Therefore, we design our denoising network to have limited receptive field avoid "overfitting". and use triplane-aware 2D convolution to enhance coherent triplane generation.
Figure 3a illustrates the architecture of our denoising network, It's a fully-convolutional U-Net with only one-level of depth, whose receptive field covers roughly 40% region of triplane feature map of spatial size 128. On the examples used in our experiments, we found that these parameter choices produce consistently plausible results and reasonable variations while keeping the input shape's 's global structure. In each ResBlock in our U-Net, we use a triplane-awane aware convolution block (see Fig. 3b). similar to the one introduced in (Wang al. 2022b). It introduces cross-plane feature interaction by aggregating features via axis-aligned average pooling. As we will show in Sec. 4.3 effectively improves the final generation quality.
5

---

W Published as a conference paper at ICLR 2024
Figure 4: Retargeting results. By changing the spatial dimensions of the sampled Gaussian noise hy, we can resize the input to different sizes and aspect ratios. The training examples are labeled by blue boxes. From left to right, small town (Pedram Ashoori, 2020). wooden fence (REÁRCH
Studio 2018), train wagon (3ddominator 2019) and antique pillar (oguzhnkr 2017).
3.3 GENERATION
At inference time, we generate new 3D textured shape by first sampling new triplane latent using the diffusion model, and then decoding it using the triplane decoder Maec. Starting from random Gaussian noise hy ~ (O,, we follow the iterative denoising process (Sohl-Dickstein et al,
2015 Ho al. 2020
h V 1
to Var(L 1 āt-1) 1-a
h O,E
until / = 1. Here, ~ No, I) for all the last step (6 when / = 1) and of = Be. After obtaining the sampled triplane latent ho. first decode signed distance grid at resolution 256, from which we extract the mesh using Marching Cubes (Lorensen Cline 1987). To get the 2D texture map, we xatlas (Young) 2023) warp the extracted 31) mesh onto 2D plane and get the 2D texture coordinates for each mesh vertex. The 2D plane is then discritized into a 2024 x 2048 image. For each pixel that covered by warped mesh triangle, we obtain its corresponding 3D location via barycentric interpolation, and query the decoder viees to obtain its RGB color.
3.4 IMPLEMENTATION DETAILS
For all examples tested in the paper, we use the same set hyperparameters. The input 3D grid has resolution 256, i.e., max H. W, D) 256, and the signed distance threshold ca is set to 3/256. The encoded triplane latent has a spatial resolution 128, iee. max H', W" D') 128, and the number of channels C' 12.
We train the triplane auto-encoder for 25000 iterations using the Adam optimizer (Loshchilov & Hutter 2017) with initial learning rate 5 and a batch size of 216, The triplane latent diffusion model has a max time step T = 1000. We train for 25000 iterations using the AdamW optimizer with an initial learning rate 5e 3 and batch size of 32. With the above settings, the training usually takes 2 ~ 3 hours on an NVIDIA RTX A6000. Please see the Table 3 for detailed network configurations. We also include the source code in the supplementary materials.
4 EXPERIMENTS
4.1
QUALITATIVE RESULTS
We show a gallery of generated results in Fig. and Fig. 9 results in Fig. and Fig. 4 where generate new shapes of different sizes and aspect ratios while keeping the local content. This is achieved by changing the spatial dimension of the sampled Gaussian noise hy. The generated shapes able to preserve the global structure of the input example, while presenting reasonable local variations terms of both geometry and texture. In the supplementary materials, provide webpage for interactive view of the generated 3D models.
We also show some shape retargeting

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison. 1: lower is better; t: higher is better. We compare our method to SSG (Wu& & Zheng, 2022) and Sin3DGerLLi et al. (2023) in terms of geometry quality (G- (G-Qual.), geometry diversity G-Div.), texture quality (T-Qual.) and texture diversity (T-Div.). The last column
the average score over the 10 testing examples. Red: best score, Orange: second best score.
Examples
Metrics
Methods
Acropolis
Canyon
Cliff Stone
Fight Piliar
House
Swars
Small Town
Tree
Wall
Wood
Avg.
Ours
0,059
0,075
0.075
0.181
0.069
0.064
0.600
0.283
0.113
0.043
0.156
G G-Qual.
SSG
0.055
0,077
0.150
0.247
0.017
0.179
0.928
0.376
0,244
0.111
0.238
Sin3DGen
4.983
5.141
0.560
8.609
3.479
1,665
8.063
3.390
Ours
0.139
0.166
0.285
0.169
0.010
0,518
0.528
0,204
0,078
0.140
0.224
G-Div.
SSG
0.091
0.218
0.342
0.175
0.008
0.514
0.463
0.040
0.101
0.134
0.209
Sin3DGen
0.154
0,239
0.260
0.196
0.136
0.255
0.519
0.599
0.135
0.113
0.261
Ours
0.024
0.032
0,006
0.012
0.019
0.114
0,051
0.007
0.013
0.004
0.028
T-Qual.
SSG
0,070
0.079
0.064
0.225
0.334
0.068
0.099
SinJDGen
0.039
0.042
0.009
0.017
0.026
0.020
0.135
0.027
0.045
0.004
0.036
Ours
0.006
0.218
0.132
0.151
0.053
0.116
0.258
0,234
0.048
0.056
0.133
T:Div. 1
SSG
0.066
0.267
0.187
0.194
0.079
0.132
0.250
0.181
0.080
0.063
0.150
Sin3DGen
0.061
0.232
0.117
0.119
0.041
0.151
0,289
0,137
0,059
0.028
0.123
Ours
SSG
Ours
SSG
Figure 5: Visual comparison. We compare the generated results from our method and SSG (Wu & Zheng 2022). The inputs of these two examples are shown in Fig. g Note that our mesh surfaces are much cleaner (see the zoomed-in columns), and our textures have much more details (see the zoomed-in wood surfaces). Please see Fig. 14 for the visual comparison to Sin3DGer Li et al. 2023
4.2 COMPARISON
We compare our method against SSG (u & Zheng 2022). a GAN-based single 3D shape generative
model, and Sin3DGen Li et al. (2023), , a recently proposed method that applics patch matching on the Plenoxels representation Fridovich- Keil et al. (2022.. Since SSG only generates geometry, extend it to support texture generation by adding extra dimensions for RGB color at its generator's output layer. For fair comparison, we train it the same sampled 3D grid (resolution 256) that used as our encoder input. In the Fig. of the appendix, we show comparison to the original SSG on geometry generation only, by removing the texture component (x)ee) in our model. For Sin3DGen, since it synthesize radiance fields that cannot be casily relighted, we generate its training data with the same lighting condition that we use for evaluation.
Evaluation Metrics To quantitatively evaluate the quality and diversity for both the geometry and texture of the generated 3D shapes, we adopt the following metrics. For geometry evaluation, we first voxelize the input shape and the generated shapes at resolution 128. Geometry Quality (G-Qual.) is measured against the input 3D shape using SSFID (single shape Fréchet Inception Distance) (Wu & Zheng 2022). Geometry Diversity (G-Div.) is measured by calculating the pair-wise IoU distance (1 IoU) among generated shapes.
For texture evaluation, we first render the input 3D model from 8 views at resolution 512. Each generated 3D model is then rendered from the same set of views. For the rendered images under each view, compute the SIFID (Single Image Fréchet Inception Distance) (Shabam et al.) 2019) against the image from the imput model, and also compute the LPIPS metric (Zhang et al. 2018) between those images. Texture Quality (T- (T-Qual.) is defined the SIFID averaged over different views. Similarly, Texture Diversity (T-Div.) is defined as the averaged LPIPS metric.
We select 10 shapes in different categories our testing examples. For each input example, generate 50 outputs to calculate the above metrics. Formal definition of the above evaluation metrics included in the Sec. of the appendix.
7

---

Published as a conference paper at ICLR 2024
Table Ablation study, Metric values are averages over the 10 testing examples used in Table
G-Qual.
G-Div.
T-Qual. ! T-Div.
Ours
0.156
0.224
0.028
0.133
w/o triplaneConv
0.347
0.310
0.068
0.171
w/o encoder
0.608
0.355
0.060
0.179
e-prediction 0.925
0.402
0.156
0.187
(a) Outpainting
(b) Patch duplication
Figure 6: Controlled Generation. Left: outpainting, which seamlessly extend the input 3D shape beyond its boundaries. Right: patch duplication, which copies a patch of the input to specified locations of the generated outputs. Wood (All-about-Blend Blttt--BleeerrDD 2020), stone tiles (skinny hands) 2021
house (Lukas carnota, 2015) and sandstone mountain (Marti David Rial 2021
Results We report the quantitative evaluation results in Table and highlight the visual difference in Fig. 5 Fig. [4] Compared to the baselines, our method obtains better scores for geometry and texture quality, while having similar scores for diversity. The generated shapes from SSG Wu & Zheng 2002)) often have noisy geometry and blurry textures. This is largely because it is limited by the highest voxel grid resolution it trains on. Sin3DGen Li et al. (2023), based on Plenoxels radiance fields, often produces broken mesh surfaces and distorted textures (see Fig. 14] In contrast, method is able to generate 3D shapes with high quality geometry and fine texture details.
4.3 ABLATION STUDY
We conduct ablation studies validate several design choices of our method. compare our proposed method with the following variants:
Specifically, we
Ours (w/o triplaneConv), in which we do use the triplane-aware convolution (Fig. 3b) in the denoising network. Instead, we simply use three separate 2D convolution layers for each plane, without considering the relation between triplane features.
Ours (w/o encoder). in which we remove the triplane encoder v/enc and fit the triplane latent in an auto-decoder fashion (Park al. 2019). The resulting triplane latent less structured, making the subsequent diffusion model hard train.
Ours (E-prediction). in which the diffusion model predicts the added noise e instead of the clean input he (see Eqn. 6). In single example case, ho is fixed and therefore easier to predict. Predicting the noise adds extra burden to the diffusion model.
As shown in Table 2 all these variants lead lower scores for geometry and texture quality. The diversity scores increase at the cost much lower result quality. Please refer to Fig. for visual comparison. We also visually compare results of using different receptive field size Fig. 13
4.4 CONTROLLED GENERATION
Aside from randomly generating novel variations of the input 3D textured shape, we can also control the generation results by specifying region of interest. Let m denote a spatial binary mask indicat- ing the region of interest. Our goal is to generate new samples such that the masked region stays as close as possible to our specified content yo while the complementary region (1 - m) are syn- thesized from random noise. Specifically, when applying the iterative denoising process (Eqn. 77). we replace the masked region with yo- That is, h O m + Yo @ m, where O is element- wise multiplication. To allow smooth transition around the boundaries, adjust such that the borders
8

---

Published as a conference paper at ICLR 2024
Rendered
Geometry
Base Color
Metallic
Roughness
Normal
Input
Generated
Input
Generated
Figure 7: Results on 3D models with PBR material. Here we show two examples with PBR materials. For each example, show the input model top and two generated models below. Cliff stone (DJMaesen 2021), damaged brick wall (Max Ramirez 2016).
between 0 and are linearly interpolated. Note that no re-training needed and all changes are made inference time.
We demonstrate two use cases in Fig. 6 and Fig. 15 1) Outpainting, which seamlessly extends the input 3D textured shape beyond its boundaries. This is achieved by setting Yo to be the padded triplane latent hy of the input (padded by zeros). The mask m corresponds to the region occupied by the input. 2) Patch duplication, which copies the patch of the input the certain locations of the generated outputs where other parts are synthesized coherently. In the case, we take hg and copy the corresponding features get Yo- The mask corresponds to the regions of the copied patches.
4.5 SUPPORTING PBR MATERIAL
PBR PPhysics-Based Rendering) materials are commonly used in modern graphics engines, they provide more realistic rendering results. Our method can be easily extended to support input 3D shape with PBR material. In particular, we consider the material in terms of base color (R²), metallic (R), roughness (R) and normal (IR3). Then, the input 3D grid has 9 channels total (ie., GM E xDx9). We add two additional MI.P heads in the decoder one for predicting metallic and roughness; the other for predicting normal. We demonstrate some examples in Fig. 7 and Fig. I0
5 DISCUSSION AND FUTURE WORK
In this work, we present Sin3DM, a diffusion model that is trained on a single 3D textured shape with locally similar patterns. To reduce the memory and computational cost, compress the input into triplane feature maps and then train a diffusion model to learn the distribution of latent features. With a small receptive field and triplane-award aware convolutions, our trained model is able to synthesize faithful variations with intricate geometry and texture details.
While the use of triplane representation significantly reduces the memory and computational cost, we empirically observed that the generated variations primarily occur along three axis directions. Our method is also limited controlling the trade-off between the generation quality and diverse, which is only possible by changing the receptive field size in our diffusion model. Generative models that learn from a single instance lack the ability to leverage prior knowledge from external datasets. Combing large pretrained models with single instance learning, possibly through fine- tuning (Ruiz et al., 2022 Zhang et al. 2022). is another interesting direction for synthesizing more diverse variations.

---

Published as a conference paper at ICLR 2024
REFERENCES
3ddominator.
Train
wagon,
https::/skkttchhaatt
train-war waw.:e/0022800033333338300020000 tribution.
2019.
License: CC At-
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. tions and generative models for 3d point clouds. In International conference on machine learning. pp. 40-49. PMLR, 2018.
Learning representa-
Ahmad Riazi. htt.::ac....cccooooeeeeeooeeeaaa CC Attribution.
Voronoi card stand.
https://sketchrabb.. http://ttttccaaaa...ooa//ee-eaee -odedddelll
2013. License:
All-about-Blender-DD Photo-realistic floating wood.
https://ww.ccttzzttt com/
(ree-3ddmoddelee
2020. Li-
cense: Royalty Free.
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
12608- 12618, 2023.
Martin Bokeloh, Michael Wand, and Hans-Peter Seidel. A connection between partial symmetry and inverse procedural modeling. In ACM SIGGRAPH 2010 papers, pp. 1-10. 2010.
Ruojin Cai, Guandao Yang. Hadar Averbuch-Elow Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference Computer Vision, pp. 364-381. Springer, 2020.
Eric R Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awan 3d generative adversarial networks. arXiv preprint arXiv: 2rXiv:21 12.07945, 2021.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi. and Fisher Yu. ShapeNet: An Information-Rie 3D Model Repository. Technical Report arXiv:1512.030 [cs.GR], Stanford University - Princeton University - Toyota Technological Institute Chicago, 2015.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 45-54, 2020.
Zhiqin Chen, Vladimir Kim, Matthew Fisher, Noam Aigerman, Hao Zhang, and Siddhartha Chaudhuri. Decor-gan: 3d shape detailization by conditional refinement. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15740-15749, 2021.
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, and Liangyan Gui. Sd- fusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv preprint
arXiv:2212.04493, 2022.
choly
https: http://tttbbsquuiiiott cow.://mii..mmmdmmmmmeeellllll acropolis 3ds free/610880 2021. License: Educational Uses.
kurd.
akropolis.
Blender Online Community. Blender - a 3D modelling and rendering packagge Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. TRRLhtp::// www. http://wwwwwwewe.....ec
wwwnder................. org
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiama Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of anno- tated 3d objects. arXiv preprint arXiv:2212.08055 2022.
10

---

Published as a conference paper at ICLR 2024
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.0500 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
DJMaesen.
Terrain
2.
wtw.chaaa..commmmmmdeeee..ee 2018. License:
CC Attri-
bution.
DJMaesen.
Cliff.
loon/3d-modelll
C
2021. License: CC Attribution.
Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. Proceed-
ings of the seventh IEEE international conference on computer vision, volume 2, pp. 1033-1038. IEEE, 1999.
Epic Games. Unreal engine, 2019. URL /tLhtttp:
wew.ineeeennnneenee com
Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5501-5510, 2022.
Thomas Funkhouser, Michael Kazhdan, Philip Shilane, Patrick Min, William Kiefer, Ayellet Tal, Szymon Rusinkiewicz, and David Dobkin. Modeling by example. ACM transactions graphics (TOG), 23(3):552-663, 2004.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 15:31841-31854 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In
Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL httts:::/ hhttps://proceedngss neurips...u........... cc cc/paperr/01///1le Paper. pddf
Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani. Drop the gan: In defense of patches nearest neighbors single image generative models. arXiv preprint arXiv:2103.15545. 2021.
Gal Greshler, Tamar Shaham, and Tomer Michaeli. Catch-a-waveform: Learning to generate au- dio from single short example. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp- 20916-20928. Curran Associates, Inc., 2021.
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.0533 2023.
Niv Haim, Ben Feinstein, Niv Granot, Assaf Shocher, Shai Bagon, Tali Dekel, and Michal Irani. Diverse generation from single video made possible. arXiv preprint arXiv:2109.0852 2021.
Charles Han, Eric Risser, Ravi Ramamoorthi, and Eitan Grinspun. Multiscale texture synthesis. In ACM SIGGRAPH 2008 papers, pp. 1-8. 2008.
Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. Deep geometric texture synthesis.
ACM Trans. Graph 39(4), 2020. ISSN 0730-0301. doi: 10. 10.1145/3386669333924777 URL
hittps://dol. org.orgggoo........ooo 10 1141111111//1 111.1/1181666..111 3392477
Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. Improved techniques for train- ing dingle-image gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1300-1309, 2021.

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-6857 2020.
Implive.
/ttps::/sketcchaft con/3d-models/ Lig.taccpplls-ee112222e288666660
Fighting pillar
2021. License:
CC Attribttion-SarrAAlike
Pradeep Kumar Jayaraman, Joseph G Lambourne, Nishkrit Desai, Karl DD Willis, Aditya Sanghi, and Nigel JW Morris. Solidgen: An autoregressive model for direct b-rep synthesis. arXiv preprint arXiv:2203. 2022.
JB3D.
Old towers ruins.
httpss//sketcc..ab.com// modelss wtw.uno-222133560000000061886661
pack of - old towwrs-in--une
2019. License: CC Attribution.
R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, and Daniel Ritchie. Shapeassembly: Learning to generate programs for 3d shape structure synthesis. ACM Transactions on Graphics (TOG), Siggraph Asia 2020, 39(6):rticcl 234, 2020.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. A probabilisticc model for component-based shape synthesis. Acm Transactions on Graphics (TOG), (11) 2012.
Animesh Karnewar, Oliver Wang, Tobias Ritschel, and Niloy Mitra. 3ingan: Learning 3d gener- ative model from images of a self-similar scene. In 2022 International Conference on 3D Vision (3DV), pp. 342-34222 352. IEEE, 2022.
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training a 3d diffusion model using 2d images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14423-18433. 2023.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. arXiv preprint arXiv:2211.165522 2022
Laetitia Irata.
Temple of ikov reimagined.
ht.ps://sseecchfab..cmmm//
2019.
License: CC Attribution.
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for pigh-peeformance differentiable rendering. ACM Transactions on Graphics, 396), 2020.
Jun Li. Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG), 36(4):1-14, 2017.
Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung. Ganimator: Neural motion synthesis from single sequence. ACM Transactions on Graphics (TOG), 41()): 138, 2022.
Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics 40(4):1111 2021.
Weiyu Li, Xuelin Chen, Jue Wang, and Baoquan Chen. Patch-based 3d natural scene generation from single example. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16762-16772. 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
12

---

Published as a conference paper at ICLR 2024
Ruoshi Liu and Carl Vondrick. Humans as light bulbs: 3d human reconstruction from thermal reflec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12531-12542, 2023.
Ruoshi Liu, Sachit Menon, Chengzhi Mao, Dennis Park, Simon Stent, and Carl Vondrick. Shadows shed light on 3d objects. arXiv preprint arXiv:2206.0899 2022.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1eto-1-to-3: Zero-shot one image 3d object. arXiv preprint arXiv:2303./1325, 2023.
William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface struction algorithm. SIGGRAPH '87, pp. 163-169, New York, NY, USA, 1987. Association for Computing Machinery. ISBN 0897912276. doi: 10.1145/3700..34422
URL https:
//doi. 145337401.7742
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
2017.
Lukas carnota. Industrial building. https://www wtt.:.cgtaade...r com/ comm/rree-3d-mmodell.
2015. License: Royalty Free.
Marti David Rial. High detail sandstone mountain.
2011688//dww loadlooldoeee loappgeaadpgee 2021. License: Royalty Free.
Max Ramirez, Damaged wall (pbr). https://sketcchaat mttp:/chh...a.coom/mmmmmetteees. (aangg/c.aa1--gg000000e...b.aaa.
2016. License: CC
Attribution.
Paul Merrell. Example-based model synthesis. In Proceedings of the 2007 symposium Interactive 3D graphics and games, pp. 105-112, 2007.
Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of IÉEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or, Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint trXiv:2211.076000 2022,
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Struc- turenet: Hierarchical graph networks for 3d shape generation. arXiv preprint 1rXiv:1988.0055 2019.
Charlic Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference Machine Learning, pp. 7220- 7229. PMLR, 2020.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:22/2.0875/. 2022.
Yaniv Nikankin, Niv Haim, and Michal Irani. Sinfusion: Training diffusion models single image or video. arXiv preprint 2rX1v:2211.11743, 2022.
Eyvind Niklasson, Alexander Mordvintsev, Ettore Randazzo, and Michael Levin. Self-organising textures. Distill, 602):c00027-003, 2021.
nikola.
https:/ homm/ddmmdeels [Do-oooycceee0000eeaaaee22211111 Attribution.
Low poly
trees.
2020. License: CC
oguzhnkr.
Antique pillar.
https://sketchfat com 3d-models/
omi.aeeellaao30022244a4333222110 2017.
License: CC
Attribution.
13

---

Published as a conference paper at ICLR 2024
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Dario Pavllo, Jonas Kohler, Thomas Hofmann, and Aurelien Lucchi. Learning generative models of textured 3d meshes from real-world images. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13879- 13889,13889,,0 2021.
Pedram Ashoori. Small town. http:///w..ttraaada.cccorreeeeee
2020. License: Royalty Free.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. 523-540. Springer, 2020.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. Single motion diffusion. arXiv preprint arXiv:2302.05905, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204400001 2022.
REÁRCH
Studio.
Palisade wall.
palisade seaade-wall--a wwwll-ll----- low Lowll---w-o-------- polppoollllae 1bb1 http:///ooooss444ooeoeeddooooaad
2018.
License: CC Attribution.
Carlos Rodriguez-Pardo and Elena Garces. Seamlessgan: Self-supervised synthesis of tileable tex- ture maps. IEEE Transactions on Visualization and Computer Graphics, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695. 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint Xiv:22200 12242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Newral Informa- tion Processing Systems, 35:36479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08-02, 2022.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from a single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4570-4580, 2019.
Shahriar Shahrabi. The vast land.
https:://kettccfaat com/ 3d-models/ the-vast land- 733380225524443ef99995 2tt.:/54444222999922722222993999 2021. License:
CC
Attribution.
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Ingan: Capturing and retargeting the" dna" of a natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4492-4501, 2019.
14

---

Published as a conference paper at ICLR 2024
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. arXiv preprint 20x.:2221111111.7,,7
skinny hands. Stylized stone tiles.
https///sketccfaa..cmmm stylized stone-totttt tiles-iel------- (ecccceaaaaaaaaa8833733999900919
2021. Li-
cense: CC Attribution.
SLASH / RENDAR.
http:://Beetcc..a..ommm/.0mmeeee old- httpaaacw..caaaaaaaa...000001111
Industrial pipe.
2020. License: CC Attribution.
Jascha Sohl- Diikkteein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 1256-2261 PMLR, 2015.
Minjung Son, Jeong Joon Park, Leonidas Guibas, and Gordon Wetzstein. Singraf: Learning 3d generative radiance field for a single scene. arXiv preprint arXiv:22 1rX122221..17660, 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
stray. Wooden irregular wall.
bttps://www.cggzaaerr com/ fdee-- 2015. License: Royalty
Free.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond Yeh, and Greg Shakhnarovich. Score ja- cobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022a.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu. Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. arXiv preprint arXiv:2212.061335 2022b.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffussio: Learning a diffusion model from a single natural image. arXiv preprint arXiv:2211.12445, 2022c.
Werniech. Large stone arch
https:// www.tuurbssqud..cmm com/ 3d-models/
large- ston--atne-aaae rch- 3d- 1 1908101 2022. License: TurboSquid Standard License.
Werniech ottp:f/ooe.....aaaaaaaaaaaaaaaaa
van der
Heever.
Stalagmites.
https: //ww. www.
cgtrader. www.
stalagmites 2atp:/altee-22222----020000----0 26200036-6666-------------------
2019.
License:
Royalty Free.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a proba- bilistic latent space of object shapes via 3d generative-advvesarial modeling. Advances in neural information processing systems, 29, 2016.
Rundi Wu and Changxi Zheng. Learning to generate 3d shapes from single example. ACM Transactions Graphics (TOG), 41(6):1-19, 2022.
Rundi Wu, Chang Xiao, and Changxi Zheng. Deepead: A deep generative network for computer- aided design models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6772 6782, 2021.
Kai Xu, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. Fit and diverse: Set evolution for inspiring 3d shape galleries. ACM Transactions Graphics (TOG), 31(4):1-10, 2012.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 4541 4550, 2019.
Jonathan Young. xatlas.
https: https://githuu.... www.:ccc//jpccccccccc /atee//xaalas
2023.
15

---

Published as a conference paper at ICLR 2024
Xiaohui Zeng. Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, volume 35, pp. 10021-10039. Curran Associates, Inc., 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. arXiv preprint arXiv:22/2.0445: 2022.
ZiCheng Zhang, Cong Han, and TianDe Guo. Exsingan: Learning an explainable generative model from single image. arXiv preprint arXiv:2/05.07350, 2021.
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Non- stationary texture synthesis by adversarial expansion. ACM Trans. Graph., 37(4). jul 2018. ISSN 0730-0301.
doi: 10.11453197517:221225. URL /ttoss///el..org10..111
3197517. 3201285
Simon
Ustal.
Canyon landscape.
canyon-
2020. License:
CC Attribution.
16

---

Published as a conference paper at ICLR 2024
A
RESULTS GALLERY
Input
Generated
Acropolis
Canyon
Fighting Pillar
House
Small Town
Ruined Tower
Wood
Tree
Cliff Stone
Figure 8: Results gallery. We show the input 3D textured shapes on the left, and several randomly
generated samples the right. From top bottom, acropolis (choly kurd 2021 canyon (Simon
Ustal 2020), fighting pillar (ImpJive] 2021). house (Lukas carnota] 2015). small town (Pedram
Jahhoori|2020) ruined tower (1B3D22019) wood (AllabouttBleedeer, 2020), tree (nikola 2020
and cliff stone (DJMaesen 2021).
17

---

Published as a conference paper at ICLR 2024
Input
Generated
Antique Pillar
Stalagmites
Temple of Ikov
Train
Vast Land
Pipe Stone Arch Voronoi Stand
Wall
Figure Results gallery. From top to bottom, antique pillar (oguzhnkr 2017 stalagimites
(Werniech van der Heever [2019) temple of ikov (Lactitia Irata 2019), train wagon (3ddominator 2019), vast land (Shahriar Shahrabi 2021). voronoi (Ahmad Riazi 2013), stone arch (Werniech)
2022 pipe (SLASH RENDAR 2020), wall (stray) 2015).
18

---

Published as a conference paper at ICLR 2024
Rendered
Geometry
Base Color
Metallic
Roughness
Normal
Input
Generated
Input
Generated
Figure 10: More results on 3D models with PBR material. Damaged wall (Max Ramirez 2016 terrain [DMaesen 2018).
Table Network architectures. IN: instance normalization layer. GN: group normalization layer. Downsampll lin; average pooling. Upsampling ing: bilinear interpolation with scaling factor 2. For the decoder MIP, we add a skip connection to the middle layer (Park al. 2019).
Module
Layers
Out channels
Kernel size
Stride
Vonc-Comy
Conv3D+IN+tant.
2
Conv2D+IN+Sin
64
doe ResBlock
Conv2D
64
I
Conv2D (shortcut)
64
1
[LineartReL0]
256
doc -LL
Linear
or 3
TriplaneConv
64
TriplaneResSlock
64
Downsampling
2
TriplareResslock
128
pa
TriplaneResšlocc
128
Upsampling
TriplaneResslock
64
GX++S1L04trripaaeCCoav
12
1
B NETWORK ARCHITECTURES
C EVALUATION METRICS
For G-Qual. and G-Div,, we refer readers to (Wu & Zheng diversity score based loU.
2022) for the calculation of SSFID and
19

---

Published as a conference paper at ICLR 2024
T-Qual. and T-Div. computed as follows. We uniformly select 8 upper views (elevation angle 45%) and render the textured meshes in Blender. Let (M) and 1,(G) denote the rendered images at the i-th view, of the reference mesh M and the generated mesh Gj. respectively. T-Qual. and T-Div. are then defined as
T-Qual.
(IF)
i=1
(8)
8
T-Div.
k(k
i=1
where we set n = 50. SSFlD(Shaham et al. 2019) and LPIPS (Zhagg et al. 2018) are distance measures.
D MORE EVALUATIONS
Visual comparison for the ablation study (Sec. 4.3) is shown in Fig. and Fig. T2 Ablation different receptive field sizes is shown in Fig. 13 We also compare to SSG (Wu & Zheng 2022) on geometry generation only. by removing the texture component (eedee in our model. In Table and Fig. 17 we show the quantitative and qualitative evaluation results their 10 testing examples.
Table 4: Quantitative comparison geometry generation. 4: lower is better; t: higher is better. Please refer to (Wu Zheng 2022) for metries definition. Numbers are average values over 10 testing examples.
LP-IoU f LP-F-score
SSFID
Ours SSG
0.572 0.477
0.699
0.068
0.594
0.074
Ours
wlo encoder
h,
h
bgg
h
h
Figure 11:
Visualization of triplane feature maps. Without encoder, the learned triplane feature maps are noisy and less structured, which poses difficult task for the subsequent diffusion model.
Input
Ours
w/o triplaneConv
w/o encoder
c-prediction
Figure 12: Ablation study. We show visual results from our method and compared variants.
20

---

Published as a conference paper at ICLR 2024
2 ResBlocks
4 ResBlocks
6 ResBlocks
Figure 13: Ablation over different receptive field sizes.. We change the receptive field size by using different numbers of ResBlock the U-Net. 2, 4, 6 ResBlocks have receptive fields that cover roughly 20%, 40%, 80% of the input. the receptive field is too small, the generated shapes exhibit rich patch variations but fail to retain the global structure. If the receptive field is too large, the model overfits to the input example and is unable to produce variations.
Ours
Li
Ours
Li al
Figure 14: Visual comparison to Sin3DGerLi al. 2023). The generated 3D shapes from Sin3DGen often depict bad geometry and sometimes their textures are distorted. In comparison, our generated shapes have high quality geometry with fine textures details.
21

---

Published as a conference paper at ICLR 2024
MOLE
a) Retargeting
b) Outpainting
c) Patch duplication
Figure 15: More examples for retargeting, outpainting and patch duplication.
Input
Generated
Input
Generated
Figure 16:
Failure cases. For inputs that have strong semantic structures, our method produces variations that do not necessarily maintain its semantics.
22

---

Published as a conference paper at ICLR 2024
Input
Ours
SSG
Figure 17: Visual comparison SSG (Wu al. 2016) on geometry generation. For most cases, our generated geometry are cleaner and sharper. But for examples with thin structures, our results are more likely to be broken (e.g., the vase).
23


Reviews:
[{"review_id": "SsLrs6wUgI", "reviewer": "Reviewer_Udpj", "summary": "This method introduces a way to generate new textured 3D shapes from a single example. To accomplish this, the input shape is represented using triplane feature maps, which encode signed distance as well as color. These feature maps are used to train an auto encoder, and a diffusion model is then trained on this auto encoder latent space. Then, by modifying the noise map, e.g., changing the resolution or masking regions, during the diffusion inference process, novel 3D shapes whose internal distribution resembles that of the original are generated.", "questions": "How consistent/realistic are the normals generated in Figure 7? It would be interesting to show them used in relighting.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The overall idea of using latent diffusion models to aid in learning the internal distribution of a 3D shape and the specific design choices in the method---e.g., the shape representation and the steps taken to prevent overfitting---are clever. The results shown also look nice and consistent with minimal artifacts.", "weaknesses": "It is mentioned several times that a unique benefit of the proposed method is that inputs and outputs meshes. This is not the case, as instead the 3D representation is triplanes encoding signed distance as well as texture color. While this can indeed be converted to a mesh using marching cubes, it's not fair to claim that the method outputs a mesh. This brings me to my main concern with the paper. While the authors cite and briefly discuss [Li et al. 2023], no qualitative or quantiative comparisons are provided, with the explanation that Li et al. do not produce parameterized meshes. While I agree that there are merits to the proposed representation, [Li et al. 2023] seems like the most natural point of comparison---many of the experiments are nearly identical to this in this work. For this reason, I cannot recommend acceptance as is.\n\nTypos:\n\nAbstract: \"example as long\" -> \"example has long\"\n4: \"The decoder first refine...\" -> \"The decoder first refines... gathers...\"\nFigure 3: replicating along which axis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "79r4AV9TdL", "reviewer": "Reviewer_vrWR", "summary": "This paper introduces a diffusion model trained on a single 3D textured object. The goal here is to learn a generative model that can regenerate local patch wise patterns of the single object. The key idea is to train a path-based diffusion model on encoded latents of the single objects and use a triplane-base decoder to represent the 3D content. For the decoding part, the authors propose a novel triplane-aware convolutional architecture and 3D convolutional based encoder. The authors provide results on various 3D assets and provide an ablation study to support their claims.", "questions": "In my view, the diversity metrics measure the average distance between generated samples and it seems like the generated samples should be very different from each other to have an improved diversity. I’m unsure if this is a good metric. The goal you want to achieve is a good similarity of local details while having a diversity in the global structure. Maybe a patch-based distance vs. a global feature can help with that. Please comment on that.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The related work section covers relevant topics and helps to set the work in context.\nThe authors tackle a novel problem that I was not aware of before.\nThe results look plausible and claims are supported.", "weaknesses": "One point that remains unclear to me is how the triplane based representations can handle larger extensions of objects, e.g.  in the example of the building. How can the triplane represent those? Do you increase the size?\n\nThe authors claim that the model learns a distribution over patches. It is unclear to me how you choose the size of patches/ receptive field and how you identify that it is nor overfitting. I think there is more evaluation needed to show that it is not overfitting. Maybe an ablation on the receptive field size can lead to interesting insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YKeWOCuVC", "reviewer": "Reviewer_s6cL", "summary": "This paper aims at training a diffusion model to generate tri-plane that could be decoded into 3D objects. It first compress the 3D meshes into tri-planes by training a auto-encoder. Next, it trains a diffusion model to generate the encoded triplane. By altering the input noise, the model could achieve retargeting, outpainting and local editing.", "questions": "1. What would happens if multiple objects instead of single object are encoded and fed into diffusion model in training? Would it not be able to generate realistic samples?\n2. What does retargeting mean in this context? I could only get the meaning of outpaining and local editing.\n3. How do the input noise and target tri-plane feature match each other when training the diffusion model on only one input shape? One input shape could only correspond to one tr-plane feature maps, while different noise is sampled in each iteration.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The edited geometry after altering the noise is satisfactory. It has rich local variation while retaining the global structure.\n2. The writing is clear.", "weaknesses": "1. After training, the model could only generate 3D contents with the same global structure. Since we already have the basic 3D mesh at the beginning, it seems that the application is a bit narrow and limited. Besides, achieving this goal needs to train two models, an auto-encoder and a diffusion model. The outcome does not seem to match the effort.\n2. The novelty of this work lies at the usage of tri-plane as the input of diffusion model and the introduction of tri-plane convolution. Both aspects do not offer a solid contribution. Using tri-plane as a compact 3D representation has been explore by many prior works. The contributino tri-plane convolution also lacks complete analysis, such as discussion and comparison with similar modules that could fuse three axis information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0fRArp4VxO", "reviewer": "Reviewer_cGW2", "summary": "This paper introduces a novel generative model for generating 3D textured shapes, given a single example as input. Building upon recent work on diffusion models and tri-plane representations for shape representations, the proposed model can generate variations of the input shape. The method is evaluated across a variety of metrics of quality and diversity and compared to previous work on this topic. An ablation study is included, showing the impact of each of the components of the model. Finally, important applications are shown, including generating 3D shapes with BRDF properties, as well as different edition possibilities.", "questions": "- What are the results of this method with unstructured shapes? Most of the results shown contain repeating patterns (like columns, windows or bricks). What would happen with a human 3D shape, for instance? \n- Can this method generate tileable texture shapes? For example, it would be valuable to generate a tileable version of the shape in Figure 6 (a).\n- Regarding the resolution of the method, it is mentioned that the maximum resolution across one dimension is 256 for the input shape, and that the receptive field is 128, while it is also mentioned that the receptive field is 40% of the input resolution. Could the authors clarify this?\n- Does this model allow for texture transfer or for structural analogies? See \"Non-Stationary Texture Synthesis by Adversarial Expansion\", \"Drop the GAN\" or \"Neural Photometry-guided Visual Attribute Transfer\" for examples of this. It would be very powerful to be able to condition the generation of one or more of the tri-planes with a structure from another image. \n- Is there a middle ground between using and not using the $\\epsilon$-prediction? I am concerned that this introduces a trade-off between diversity and quality that may be limiting the impact of the proposed method.\n- Can the authors show more examples of results of this method? I think it would be valuable to see more results on controlled generation, retargeting and PBR generation. \n- What is the impact of using only a 2-level U-Net for the denoising, rather than \n\nWriting improvement suggestions:\n- I suggest the author mention the need for relightable assets less frequently. I think the introduction motivates the problem well, however, it is mentioned more times during the paper, hindering its readability. \n- On the first paragraph of the introduction: \"often time-consuming and tedious\", as the current text is written, is referring to \"artistic sensibilities\", which I would disagree that are \"time-consuming and tedious\". I understand where the authors are going with this argument, however, the current phrasing is a bit confusing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces a novel generative model for 3D textured shape generation, capable of synthesizing unseen variations of a 3D texture shape given a single example as input. Generative models trained on a single example (image, texture, video, etc) are a long-standing problem in the literature, and this paper proposes, to the best of my knowledge, the first method for 3D textured shapes. \n- The proposed method is sound, the design decisions are solid and well-justified, and borrow from the literature of both neural rendering and single image diffusion models in interesting ways. \n- Some ideas in this paper may be valuable for many downstream applications, including generative models trained on datasets of 3D textured shapes, or for dreamfusion-like models, where the representation that is learned is a radiance field.\n- The quantitative evaluation proposed in this paper is comprehensive, measuring not only quality but also diversity.\n- The applications proposed in this paper demonstrate that the method can be used for controllable generation and edition. Further, it can also synthesize BRDF properties, which makes this model applicable for downstream computer graphics pipelines.\n- The paper is well structured and it is mostly well written. The quality of the figures is very high. \n- The triplane convolution is smart and I believe it amy have an impact on many future works.  \n- The supplementary material provides valuable insights on the quality of the results and the impact of different individual components.\n- Extensive implementation details are provided, and code is included in the submission, increasing the paper's reproducibility.", "weaknesses": "- The paper writing is sometimes overly convoluted and the same arguments are repeated too many times. In my opinion, this makes the paper at times a bit hard to read. (See the Questions section). \n- In some cases, showing more examples could have been benefitial to this paper. \n- While limitations are somewhat adequately discussed, it would have been interesting to see failure cases. \n- The applications (4.4 and 4.5) sections are limited in the amount of examples shown and it makes it hard to fully understand what level of control is actually provided to the user. \n- The related work section can be, in my opinion, improved. In particular, I believe that this paper should mention an important use-case for single-sample generative models, which is texture synthesis. This is even more important as this paper is working with Textured 3D shapes, and most of the examples shown contain repetitions (eg bricks or windows). I suggest including important work like \"Non-stationary texture synthesis by adversarial expansion\", Zho et al., Siggraph 2018; \"“Self-organising textures\", Niklasson et al. 2021; and \"SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps\", Rodriguez-Pardo et al. TVCJ 2022. \n- Some components of the model are detrimental to the diversity of the generated samples. This is a traditional trade-off in many generative models, where increased diversity can lead to less realistic outputs. However, I believe that this paper leans heavily towards realism, hindering diversity. I believe that some level of control on this tradeoff should be allowed to the user, and changing the receptive field of the model is, in my eyes, the only possible way to achieve this. This could be discussed in the limitations section or some future work could be hinted to address this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)