Large-Vocabulary 3D Diffusion Model with Transformer
Authors: Ziang Cao
Fangzhou Hong
Tong Wu
Liang Pan
Ziwei Liu
Venue: ICLR 2024 poster (2024)
Keywords: Large-Vocabulary 3D Objects; 3D Generation; Transformer; Diffusion model
Primary area: generative models

Abstract: Creating diverse and high-quality 3D assets with an automatic generative model is highly desirable. Despite extensive efforts on 3D generation, most existing works focus on the generation of a single category or a few categories. In this paper, we introduce a diffusion-based feed-forward framework for synthesizing massive categories of real-world 3D objects \textit{with a single generative model}. Notably, there are three major challenges for this large-vocabulary 3D generation: \textbf{a}) the need for expressive yet efficient 3D representation; \textbf{b}) large diversity in geometry and texture across categories; \textbf{c}) complexity in the appearances of real-world objects. To this end, we propose a novel triplane-based 3D-aware \textbf{Diff}usion model with \textbf{T}rans\textbf{F}ormer, \textbf{DiffTF}, for handling challenges via three aspects. \textbf{1}) Considering efficiency and robustness, we adopt a revised triplane representation and improve the fitting speed and accuracy. \textbf{2}) To handle the drastic variations in geometry and texture, we regard the features of all 3D objects as a combination of generalized 3D knowledge and specialized 3D features. To extract generalized 3D knowledge from diverse categories, we propose a novel 3D-aware transformer with shared cross-plane attention. It learns the cross-plane relations across different planes and aggregates the generalized 3D knowledge with specialized 3D features. \textbf{3}) In addition, we devise the 3D-aware encoder/decoder to enhance the generalized 3D knowledge in the encoded triplanes for handling categories with complex appearances. Extensive experiments on ShapeNet and OmniObject3D (over 200 diverse real-world categories) convincingly demonstrate that a single DiffTF model achieves state-of-the-art large-vocabulary 3D object generation performance with large diversity, rich semantics, and high quality. More results are available at https://difftf.github.io/

Published as a conference paper at ICLR 2024
LARGE-VOCABULAR, 3D DIFFUSION MODEL WITH TRANSFORMER
Ziang Cao', Fangzhou Hong', Tong 2u Liang Pan 1.3 Ziwei Liu' Ss-Lab, Nanyang Technological University, "hee Chinese University of Hong Kong, Shanghaii Artificial Intelligence Laboratory {ziang cao, fangzhou. hong, liang. pan, ziwei wiw.)enu...deeeee. sg 0200li cuhk. edu. hk
ABSTRACT
Creating diverse and high-quality 3D assets with an automatic generative model is highly desirable. Despite extensive efforts 3D generation, most existing works focus on the generation of a single category or a few categories. In this paper, we introduce diffusion-based feed-forward framework for synthesizing massive categories of real-world 3D objects with a single generative model. Notably, there are three major challenges for this large- vocabulary 3D generation: a) the need for expressive yet efficient 3D representation; b) large diversity in geometry and texture across categories; c) complexity in the appearances of real-world objects. To this end, we propose novel triplane-based 3D-aware Diffusion model with TransFormer, DiffTF, for handling challenges via three aspects. 1) Considering efficiency and robustness, we adopt revised triplane representation and improve the fitting speed and accuracy. 2) To handle the drastic variations in geometry and texture, regard the features of all 3D objects combination of generalized 3D knowledge and specialized 3D features. To extract generalized 3D knowledge from diverse categories, we propose novel 3D-aware transformer with shared cross-plane attention. It learns the cross-plane relations across different planes and aggregates the generalized 3D knowledge with specialized 3D features. In addition, we devise the 3D-aware encoder/decoder to enhance the generalized 3D knowledge in the encoded triplanes for handling categories with complex appearances. Extensive experiments on ShapeNet and OmniObject3D (over 200 diverse real-world categories) convincingly demonstrate that single DiffTF model achieves state-of-the-art large-vocabulary 3D object generation performance with large diversity, rich semantics, and high quality. More results are available at fefffffggiituu github.ioi.
INTRODUCTION
Creating diverse and high-quality 3D content has garnered increasing attention recently, which could benefit many applications, such as gaming, robotics, and architecture. Recently, various advanced techniques have achieved promising results in 3D generation (Sitzmann et al., 2019 Müller et al.. 2022, Achlioptas al. 2018, Nichol et al. 2022 Chan et al., 2022] Mo et al. (023). However, many of these methods excel in a specific category but struggle to maintain robustness across a wide range of objects. Consequently, there an urgent need to develop an approach capable of generating high-quality 3D objects across a broad vocabulary.
Compared with the prior works single-category generation, large- vocablloyy objects have three special challenges: a) the urgent requirement for expressive yet efficient 3D representation: b) wide diversity across massive categories; c) the complicated appearance of real-world objects. Since the explicit ways are easy to evaluate but compute-intensive intensive and implicit ways are easy extend but time- consuming evaluating, we adopt an efficient hybrid 3D representation, i.e., Triplane feature (Chan) al., 2022). Besides, benefiting from state-of-the-art (SOTA) performance and reasonable denoising process, diffusion-based methods have attracted much attention in 3D object generation (Müller et al.
"Corresponding author

---

Published as a conference paper at ICLR 2024
Lange Vocabuloy Generation
Interpolation
Figure 1: Visualization on Lorge-vocabulary 3D object generation. DiffTF can generate high- quality 3D objects with rich semantic information and photo-realistic RGB. Top: Visualization of the generated results. Bottom: Interpolation between generated results.
2022; Shue et al. 2022). Therefore, we try to build a novel diffusion based feed-forward framework for large-vocatee vocabulary 3D object generation with single generative model. Specifically, in this paper, to handle the special challenges, we try to propose a Diffusion model with 3D-aware TransFormes i.., DiffTF for general 3D object generation. In contrast to shape generation methods, method can generate highly diverse 3D objects with semantic textures.
To meet the efficiency requirement of a) challenge, we adopt revised sampling method in triplane fitting for boosting efficiency. Besides, to achieve better convergence, we constrain the distribution of triplanes in an appropriate range and make the values of triplanes smooth using normalization and strong regularization including total variation (TV) regularization (Dey et al. (006).. To handle the challenge caused by the large diversity and complex appearance of real-world objects, we regard all objects as a combination of generalized 3D prior knowledge and specialized 3D features. It is superior two aspects: 1) it can benefit from voggevoocbblary objects compared with treating categories as independent properties. Since the generalized 3D knowledge is extracted from massive categories, it has strong generalized capability and robustness. 2) it can decrease the complexity generation. By separating the generalized information from different categories, the specialized feature extractor can pay more attention to the special information. To this end, we propose a novel 3D-aware transformer transformer adopts special share-weight cross-plane attention
to achieve this goal. Since module, the extracted 3D prior knowledge is generalized to different planes of different categories. As for specialized 3D features, we apply different self-attention integrate the 3D interdependencies into the encoded features, Consequently, our transformer can boost the performance in handling diverse objects. Additionally, complicated categories, we built the 3D-aware encoder/decoder. Compared with the traditional 2D encoder that inevitably affects the 3D relations, our model can integrate 3D awareness into encoded features and enhance the 3D-related information. Finally, attributed to the effective generalized 3D knowledge and specialized 3D features extracted from various categories, DiffTF can generate impressive large- vocabulary 3D objects with rich and reasonable semantics.
enhance 3D awareness and semantic information in triplanes for
To validate the effectiveness our method in large-vocabulary 3D object generation, we conduct ex- haustive experiments on two benchmarks, i.e., ShapeNet (Chang et al. 2015)) and OmniObject3D (Wu) et al. 2023). Note that OmniObject3D contains 216 challenging categories with intricate geometry and texture, e.g., pine cones, pitaya, various vegetables. As shown in Fig. I benefiting from the generalized 3D knowledge and the specialized 3D features extracted by 3D-aware modules, a single DiffTF model achieves superior performance in many complex and diverse categories, e.g., litchi,
2

---

Published as a conference paper at ICLR 2024
conch, pitaya, and dinosaur. Besides, generated results have rich semantics and impressive performance in terms of topology and texture. The figure also intuitively proves that the latent space of our method is smooth semantically.
RELATED WORK
In this section, we introduce the recent related improvements in 3D generative models including GAN-based and diffusion-basee methods, as well as transformer structure.
2.1 TRANSFORMER
In recent years, Transformer Vaswanni al.
2017) has rapid progress in many fields including
image recognition (Dosovitskiy et al. [200bb] Touvron et al. [202T). object detection (Carion et al.
2020: Zhu et al. 2220)), tracking (Cao et al.
2022 2023), segmentation (Zheng al. 2021
Strudel et al. 2021). and image generation Van den Oord et al. 2016 Jiang et al. [20 Mo et al.
2023). Some works (Chen et al., 2020 Child et al. 2019) prove the remarkable of transformer when
predicting the pixels autoregressively. Based on the masked token, MaskGIT. (Chang et al.) 2022)
achieve promising generation performance. DiT (Peebles & Xie 2022) adopts the transformer as the backbone of diffusion models of images. Based on the 2D version, Mo et al. (2023) propose a DiT-3D for point cloud generation. Despite impressive progress in transformer-based generative models, they are optimized on a single or few categories. In this paper, we propose a 3D-aware transformer for diverse real-world 3D object generation that can extract the generalized 3D prior knowledge and specialized 3D features from various categories.
2.2 3D GENERATION
With satisfying performance in 2D image synthesis, generative adversarial networks (GANs) inspire
research in 3D via generating meshes (Gao et al. 2022). , texture (Siddiqui et al. 2022), voxel (Chen
ct al. 2019), NeRFss (Chan et al. 2021 Gu et al. 2021 Niemeyer & Geiger| 202 Schwarz et al.
2020 Zhou et al. 2021), Triplane (Chan et al. 2022), and point cloud (Achlioptas al., 2018).
Since the NeRFs and Triplane-based methods can adopt 2D images as supervision without any 3D assets. those two branches of methods have received much attention recently. By adding a standard NeRF model in the GAN model, Pi-GAN (Chan et al. (021) and GRAF (Schwarz et al. (2020) can generate novel view synthesis of the generated 3D objects. Since the high memory loading and long training time of NeRF, adopting high-resolution images is hard to afford which impedes the generative performance. GIRAFFE (Niemeyer & Geiger (02T) propose a downsample uppamaee structure to handle this problem. By generating low-resolution feature maps and upsampling the feature, GIRAFFE (Niemeyer & Geiger 202T) indeed improves the quality and resolution of output images. To address the 3D inconsistencies. StyleNeRF (Gu et al. 202 design convolutional stage to minimize the inconsistencies. To boost the training efficiency further, EG3D (Chan et al.) (002)) propose efficient 3D representation, i.e., triplane. Due to its promising efficiency, in this work, adopt the revised triplane as our 3D representation.
In contrast to GANs, diffusion models are relatively unexplored tools for 3D generation. A few
valuable works based on NeRF (Poole et al. 2022 Li et al. 2022), point cloud (Nichol et al. 2022
Luo & Hu | Zeng al., 2022). triplane (Shue et al. 2022 Wang al. 2022 Gu al. 2023)
show the huge potential of diffusion model. DeamFusion (Poole et al. 2022 presents a method to
gain NeRFs data and apply the generation based on the pre-train 2D text-image diffusion model.
Similarly, NFD (Shue et al.] (022) views the triplane as the flattened 2D features and utilizes the 2D diffusion. is indeed that adopting pre-train 2D diffusion can accelerate the training process. However, 2D prior knowledge also limits the capacity of the model in 3D-related information. To handle this problem, Rodin (Wang et al. 2022) proposes a 3D-aware convolution module in diffusion. Based local CNNebased awareness, it can provide local 3D-related information for single-category generation. However, it is hard to maintain robustness when facing large categories for local 3D awareness. To this end, in this paper, introduce global 3D awareness into our diffusion- based feed-forward model to extract the 3D interdependencies across the planes and enhance the relations within individual planes.
3

---

Published as a conference paper at ICLR 2024
3Dwaee Generation Transformer
Noise
Label & Timestop
6 256 x (256 x 18 x 256 x 256
6 256 x (256
Flatten
I.
Q
3D-aware Transformer
aKVaxt axtsux
Bay
aKVaxts-x
Boy
Pra
Pyz
Fer
Fyz
3D awarn mmoder
3D awar decoder
For rendering
Reshage
Color
II.
Volume rendering
Cart my Feo Byy
Triplane Decoder
Density
Generated Triplane
Camera params
Gemerated 3D Object
Figure 2: Pipeline DiffTE. Sampling the 3D content from DiffTF has two individual steps: 1) using trained diffusion model to denoise latent noise into triplane features, and 2) adopting the trained triplane decoder to decode the implicit features into the final 3D content.
METHODOLLOGY
In this section, we will introduce the detailed structure of DiffTF. The sampling pipeline DiffTF is shown in Fig. More details about the two-step training process are released in the Appendix.
3.1 3D REPRESENTATION FITTING
An efficient and robust 3D representation is significance to train a 3D diffusion model for large- vocabulary object generation. In this paper, adopt the triplane representation inspired by (Chan ct al. 2022). Specifically, it consists of three axis-aligned orthogonal feature planes, denoted as F Fzy. F yz: Fu: I Cxx W and multilayer perceptron (MLP) decoder for interpreting the sampled features from planes. By projecting the 3D coordinates on the triplane, we can query the corresponding features (p)) Cat (aa (p), Fyz(p)) Fax (p)) Rx3 of any 3D position p R x3 via bilinear interpolation. given the position p e IR and view direction d IR? the view-dependent color R and density a E R² can be formulated as:
c(p, d), (p) MLP
(1)
where 7(-) represents the positional encoding. Detail structure of MLP, shown Appendix.
Since the fitted triplane is the input of diffusion, the triplanes of diverse real-world objects should be the same space. Therefore, we need to train robust nategor-iindepenent shared decoder. More details about the decoder released in Appendix. Besides, to constrain the values of triplanes, adopt strong L2 regularization and TVloss regularization (Dey et al. 2006) as follows:
TV(F(p)) Σ [F(p).. 1. F()) 4, [F()) +] i. 2.
(2)
Additionally, to further accelerate the speed of triplane fitting, the merely samples from the points the foreground. To avoid the low performance in predicting background RGB, we set threshold to control the ratio of only sampling from the foreground sampling from the whole image. Besides, we introduce
mask. According to the volume rendering(Max 1995). the final rendered images C(p, d) and foreground mask (p) can be achieved by: C(p, Σ T:(1 exp(-o;5)))c; where T; = exp(- E o2d,), (p) = ΣN T₁(1 cxp(-oo6;)) Therefore, the training objective of triplane fitting can be formulated as:
Breground-backkrooun
L -= (MS(((p..)) G.) (S o, +A(V))))))))) (())
(3)
where G. and G, represent the ground-truth of RGB and alpha while Aι and X2 coefficients.
4

---

Published as a conference paper at ICLR 2024
3D-aware Transformer x L
Cross-plane Attention
Share- Mhire-weigtt
Add & Norm
Multi-Head Cross tbention
MLP
K.
Add & Norm
Add & Nom
MLP
Cross-plane A Atention
V. k
Add Nomn
Add & Norm
Self-Atteation
Atentinn
Impur
Impur
Input Bay
Positional Embedding
Encoded Triplanes
Patchify
Figure 3: The detailed structure of our proposed 3D-aware modules. We take the feature from the xy plane Fay as an example. Relying on the extracted generalizable 3D knowledge and specialized one. our model can achieve impressive adaptivity among various categories.
3.2 3D-AWARE DIFFUSION WITH TRANSFORMER
3.2.1 3D-AWARE ENCODER/DECODER
In general, low-resolution features tend to contain richer semantic information and exhibit greater robustness, while high-resolution features are beneficial for capturing detailed features. Therefore, to enhance the 3D-related information in triplanes and avoid compute-intensive operation in the 3D-aware transformer, we propose new 3D-aware encoder/decoder to encode the original triplanes F' Cat( my: F F E CC W x3Hf with 3D awareness.
To avoid compute-intensive decrease the resolution features. Note that since the triplanes have 3D relations, the convolution should be performed
intensive operations in the encoder/dccoder, adopt single convolution to
denote the output individual planes respectively. For simplification,
of convolution as F' E CxC' x3H² W' = W/n, H' = H/n, where n represents the stride of convolution. Following the patchify module in (Dosovitskiy et al. 2020a), we adopt patchify layer (denote as 9) to convert the spatial features into a sequence M tokens. Thus, given the patch size ps, the dimension of output feature sequences can be formulated as: F' -FFF") RCx3M where M = W' /ps * H' fp'ps. Since the attention operation is ppsition-independent, the learnable positional embedding is applied after the patchify module (denoted the output as F). Considering triplanes from different categories may have differences feature space, we introduce the conditional information using an adaptive norm layer inspired by (Peebles & Xie 2022).
To extract the generalized 3D knowledge across different planes, introduce the cross-plane attention whose details are shown in Fig. 3 It takes the target plane (for example xy-plane) as the query while treating the whole triplane as key and values. The 3D constraint feature space between xy-plane and three planes can be obtained as:
Fay = ay + a1 MultiHead by, F,F)
where an2 represents the calibration vector obtained by MLP. To maintain the 3D-related information, we adopt share-weight attention module. Consequently, the encoded feature with 3D awareness can be concatenated: F Cat(F zn. Fy., Faz), where Fyz and F.s are the 3D constraint features from yz and xz planes respectively.
In the end, the sequence features can be restored to spatial features via an integration layer (denoted g-1(F) E RCXW'xH' The detail structure is released in Appendix.
--1): F
5

---

Published as a conference paper at ICLR 2024
3.2.2 3D-AWARE TRANSFORMER
Benefiting from the 3D-aware transformer encoder, the 3D-related information is enhanced in the encoded high-level features. To extract the global generalized 3D knowledge and aggregate it with specialized features further, we build the 3D-aware transformer illustrated in Fig. consists of self- and cross prane- attention. The former aims to enhance the specialized information of individual planes while the latter concentrates on building generalized 3D prior knowledge across different planes.
Similar to the 3D-aware encoder, we adopt the patchify module and the learnable positional em- bedding. For clarification, we denote the input of the 3D-aware transformer as F. Since the encoded features have in terms of plane-dependent and plane-independent information, the transformer two branches. Specifically, the self-attention module in the left branch can build interdependencies within individual planes. By exploiting 2D high-level features in different planes, our transformer will exploit the rich semantic information. Therefore, the output of enhanced features can be formulated as follows:
build
f = Norm(F F + MLP(fi)) F Norm( + (F,,
(5)
where Norm represents the layer normalization.
As for the right branch, it focuses on extracting the global 3D relations across different planes. Similarly, to enhance the explicit 2D features in each plane, we adopt an additional self-attention module before cross-plane attention. Meanwhile, to avoid the negative influence of 2D semantic features, we apply the residual connection. Consequently, the 3D-related information of xy planes can be formulated as:
pfinal Norm xy MLP((EC) my ryy
F Norm F + MultiHead Fi F F²))
(6)
Norm Pay + MultiHead( Fry, Fags Fay))
ay
By concatenating the features from three planes and integration layer (g (g-1) similar to 3D- aware encoder, the final features containing 3D-related information can restored to the spatial Features:Fa = G (Cat(C((PF))) my y=z= 12
final fifinal Fifinel)) E RCXW'xx x1''
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Datasets. Following most previous works, we the ShapeNet (Chang al. 2015) including Chair, Airplane, and Car for evaluating the 3D generation which contains 6770, 4045, and 3514 objects, respectively. Additionally, experiments on a most recent 3D dataset, OmniObject3D (Wu et al. 2023). OmniObject3D is large-vocabulary real scanned 3D dataset, containing 216 challenging categories of 3D objects with high quality and complicated geometry, egg., toy, fruit, vegetable, and art sculpture.
evaluate the aarge-oocabllay 3D object generation,
conduct the
Evaluation Metries. Following prior work (üüller al. 2022 Shue al. 2022). we adopt two well-known 2D metrics and two 3D metrics: a) Fréchet Inception Distance (Heusel et al. 2017) (FID-50k) and Kernel Inception Distance (Binkowski et al.
2018) (KID-50k); b) Coverage Score
(COV) and Minimum Matching Distance (MMD) using Chamfer Distance (CD). All metrics are evaluated resolution of 128 x 128. More details are released in Appendix.
4.2 COMPARISON AGAINST STATE-OF-TIE--AM METHODS
In this section, compare methods with state-of-the the-art art methods, including GAN-bace methods: EG3D (Chan et al. 2022), GET3D (Gao et al. 2022) and two diffusion-baa based methods:
DiffRF Müller et al. 2022) and NED (Shue et al.) 2022).
Large-voca 3D object generation on OmniObject3D. The class-conditional quantitative results on OmniObject3D are shown in Table Compared with NFD w/texture that uses the 2D CNN

---

Published as a conference paper at ICLR 2024
EG3D
GETSD
DiffRF
NFD w/ texture
Our
Figure 4: Qualitative comparisons to the SOTA methods in terms of generated 2D images and 3D shapes on OmniObject3D. Compared with other SOTA methods, our generated results are more realistic with richer semanties.
Methods
FID
KID(%)
COV(S))
MMD(%.)
FG3D (Chan et al.
2022
41.56
1.0
14.14
28.21
GET3D (Go 2022
49.41
1.5
24.42
13.57
DifRR Müiller al. 2022)
147.59
8.8
17.86
16.07
NFD w/ texture (Shu al.
2022
122.40
4.2
33.15
10.92
DifITF (Ours)
25.36
43.57
6.64
Table 1: Quantitative comparison of conditional generation on the OmniObject3D dataset.
DifTFF outperforms other SOTA methods in terms of 2D and 3D metries by large margin.
diffusion, our 3D-aware transformer achieves promising improvement in both texture and geometry metrics, especially in texture metric. It proves the effectiveness of the global 3D awareness introduced by our transformer. Compared DifRR adopting 3D CNN on voxel, the diffusion on triplane features can benefit from higher-resolute outperform the SOTA GAN-based methods with a significant improvement. The visualization illustrated in Fig 4 intuitively demonstrates the superiority of DiffTF in terms of geometry and texture. Particularly, our generated objects have rich semantics that make our results easy to determine the category. More qualitative results released in Appendix.
representation to boost performance. Additionally, our methods
Single-category generation on ShapeNet. To validate our methods in a single category, we provide the quantitative and qualitative results ShapeNet (Chang al.) 2015) in Fig 5 and Table 2 Compared with other recent SOTA methods, our methods achieve sigmificant improvements in terms of all metrics in three categories. Our method is more stable and robust in all three categories. As

---

Published as a conference paper at ICLR 2024
ShapeNet Car
ShapeNet. Plane
ShapeNet Chair
EC3D
GET3D
DifRRe
NFD w/ texeture
Ours
Figure 5: Qualitative comparison of DiffTF against other SOTA methods ShapeNet. It
intuitively illustrates the promising performance of our method in texture and topology.
Category
Method
FID
KID(%))
COV(S)
MMD(%.)
EG3D Chan et al. 2022
45.26
2.2
35.32
3.95
GET3D Gao el al 2022
41.41
1.8
37.78
3.87
Car
DifRF Muilerr et al. 2022
75.09
5.1
29.01
4.52
NFD w/ texture (hue et DiffTF (Ours)
2022
106.06
5.5
39.21
3.85
36.68
1.6
53,25
2.57
EG3D Chan al. 2022
29.28
1.6
18.12
4.50
GET3D (Gao et al. 2022
26.80
1.7
21.30
4.06
Plane
DifiR (Miller et al 2022
101.79
6.5
37.57
3,99
NFD w/ texture Shue et al. DifTTF (Ours)
2022
126.61
6.3
34.06
2.92
14.46
0.8
45.68
2.58
EG3D Chan et al. GET3D (Gao al. 2022
2022
37.60
2.0
19.17
10.31
35.33
1.5
28.07
9.10
Chair
DifRR Muller el al 2022)
99.37
4.9
17.05
14.97
NED w/ texture Shue et al. 2022
87.35
2.9
31.98
7.12
DiffTF (Ours)
35.16
1.1
39.42
5.97
Table 2: Quantitative comparison of unconditional generation on the ShapeNet. It proves the impressive performance of our DifTTF on the single-category generation.
shown in Fig our generated results contain more detailed semantic information which makes our generated results more realistic. More qualitative results are released in Appendix.
4.3
ABLATION STUDY
In this section, we conduct the ablation studies mainly in two ways: 1) diffusion: w/ and w/ our proposed 3D-aware modules; 2) triplane fitting: w/ and w/o normalization and regularization.
8

---

Published as a conference paper at ICLR 2024
TP Regu
TP Norm
CP
On-TF
CP-TF
FID
KID(S
COV(
MMD(
X
X
125.21
2.6
21.31
16.10
K
x
X
113.01
2.3
30.61
12.04
X
X
X
109.81
2.0
33.15
10.92
V
x
X
39.13
1.4
37.15
9.93
N/A
52.35
2.0
36.93
9.97
N/A
25.36
0.8
43.57
6.64
Table 3: Ablation studies on OmniObject3D. The TP Regu and TP Norm represent the triplane regularization and triplane normalization. Besides, we denote the proposed cross-plane attention in the encoder, original transformer and our 3D-aware transformer as CP, Ori-TF, and CP-TF.
Ablating normalization and regularization. Since the triplane fitting is the foundation of the diffusion model, we also study the influence of normalization and regularization in tri- plane fitting and diffusion training. As shown in Fig. 6 the triplane features are more smooth and clear. With effective regularization, the generated objects have better shapes and geometry information. Since the distribution of triplanes is overwide (from -10 to 10) without constraint, is essential to adopt normalization to accelerate the convergence. As illustrated in Table 3 by adopting the preprocess on triplane features, the diffusion model can get better performance.
Studies of 3D-aware transformer modules. Com- pared with 2D CNN, 3D-aware encoder/decoder can encode the triplanes while enhance the 3D relations. As shown in Fig. attributing the enhanced 3D fea- tures, it can raise the overall generative performance. Notably, since the original transformer merely utilizes 2D self-attention, the introduction of the 2D interdepen- dencies will break the latent 3D relations built by the encoder, thereby impeding the performance shown in 3 In contrast to the original transformer, our 3D- aware transformer can effectively extract generalized 3D-related information across planes and aggregate it with specialized one for large- vocabulary generation. In conclusion, attributed to extracted generalized 3D knowledge and specialized 3D features, our 3D-awar transformer is capable of strong adaptivity for large- vocabulary 3D objects. More qualitative and network architecture comparisons are released in Appendix.
Wo Regularization
w/ Regularization
Badlin
RedisecCP
Bindine/CFiQI-T RINII
Figure 6: Top: Ablations regularization. Bottom: Ablations on 3D-aware modules.
5 CONCLUSION
In this paper, in contrast to the prior work optimizing the generative model on a single category, we propose diffusion-based feed-forward framework for synthesizing various categories of real-world 3D objects with a single model. To handle the special challenges in large-vocabulary 3D) object generation,
improve the efficiency of triplane fitting: 2) introduce the 3D-aware transformer to boost the generative performance in diverse categories; 3) propose the 3D-aware encoder/decoder to handle the categories with complicated topology and texture. Separating generalized and specialized information from 3D objects can 1) enable more effective utilization of diverse 3D objects; 2) assist the specialized feature extractor focus on specialized features, thereby reducing the complexity of the vorge-vocabllary generation. Based on the extracted generalized and specialized 3D prior knowledge, our DiffTF can generate diverse high-quality 3D objects with rich semantics. The exhaustive comparisons against SOTA methods validate the promising performance of our methods. We believe that our work can provide valuable insight for the 3D generation community.
Limitation. While our method makes significant improvement in 3D generation, it still has some limitations: 1) the triplane fitting speed (~-3 minutes per object) is still time-consuming when scaled up to 1 million objects; 2) the proper noise scheduling scheme for triplane generation is not well- analyzed, impeding the improvement of generative performance, especially in detail. In our future work, will try handle those problems boost the generative performance further.

---

Published as a conference paper at ICLR 2024
Ethics statement. In this paper, we proposed novel large- vocabulary 3D diffusion model that generates high-quality textures and geometry with a single model. Therefore, our method could be extended to generate DeepFakes generate fake images videos of any person intending to spread misinformation or tarnish their reputation.
ACKNOWLEDGMENTS
This study supported by Shanghai Artificial Intelligence Laboratory (P23KS00010, 20222D0160201), the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE- T2EP20221- 0012), NTU NAP, and under the RIE2020 Industry Alignment Fund Industry Col- laboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018.
Mikolaj Binikowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint 1801 401,, 2018.
Ziang Cao, Changhong Fu, Junjie Ye, Bowen Li, and Yiming Li. Hift: Hierarchical feature transformer for aerial tracking. In Proceedings of the IEEE/CVF International Conference Computer Vision,
pp. 15457-15466, 2021.
Ziang Cao, Ziyuan Huang, Liang Pan, Shiwei Zhang. Ziwei Liu, and Changhong Fu. Tetrack: Temporal contexts for aerial tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 14798-14808, 2022.
Ziang Cao, Ziyuan Huang, Liang Pan, Shiwei Zhang, Ziwei Liu, and Changhong Fu, Towards real-world visual tracking with temporal contexts. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 213-229. Springer, 2020.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5799-5809, 2021.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In CVPR, pp. 16123-16133, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-ries 3d model repository. arXiv preprint
2015.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Kevin Chen, Christopher Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pp. 100-116. Springer, 2019.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691- 1703. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10500 2019.
Nicolas Dey, Laure Blanc-Feraud, Christophe Zimmer, Pascal Roux, Zvi Kam, Jean- SannCChrsttophe Olivo-Marin, and Josiane Zerubia. Richardson-luce algorithm with total variation regularization for 3d confocal microscope deconvolution. Microscopy research and technique, 69(4):260-26) 2006.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11.0//9999.. 1929, 2020a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020b.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-31854 2022.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style- 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021.
Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In International Conference on Machine Learning, pp. 11808-11826. PMLR, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equílibrium. Advances in neural information processing systems, 30, 2017.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34: 14745-14758, 2021.
Gang Li, Heliang Zheng, Chaoyue Wang, Chang Li, Changwen Zheng, and Dacheng Tao. 3ddesigner: Towards photorealistic 3d object generation and editing with text-guided diffusion models. arXiv preprint 1rXiv:2211.14100 2022.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, 1(2)999-108, 1995.
Shentong Mo, Enze Xie, Ruihang Chu, Lewei Yao, Lanqing Hong, Matthias Nießner, and Zhenguo Li. Dit-3d: Exploring plain diffusion transformers for 3d shape generation. arXiv preprint 2rX1v:2307.01831, 2023.
Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. arXiv preprint urXiv:2212.0/206, 2022.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:22/2.08751, 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1145--1 1464, 2021.
11

---

Published as a conference paper at ICLR 2024
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33: 20154-20166, 2020.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. arXiv preprint arXiv:2211.16677 2022,
Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Texturify: Generating textures on 3d shape surfaces. In Computer Vision-ECCY 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III. pp. 72-88. Springer, 2022.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR, pp. 2437-2446, 2019.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 7262-7272, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. arXiv preprint arXiv:2212.06133, 2022.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang. Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. arXiv preprint arXiv:2301.07525 2023.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:22 10.0697, 2022.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from perspective with transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6881-6890, 2021.
Peng Zhou, Lingxi Xie. Bingbing Ni, and Qi Tian. Cips-3d: 3d-aware generator of gans based on conditionally-indeeeddd pixel synthesis. arXiv preprint arXiv:21 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint urXiv:2010.04159, 2020.
12


Reviews:
[{"review_id": "bTNWcu01WT", "reviewer": "Reviewer_yEnj", "summary": "This paper proposes a two-stage 3D generation method, where 1) in the first stage, triplanes and a shared decoder are fitted on Shapenet/OmniObject 3D dataset. 2) in the second stage, a diffusion model is trained on the fitted triplanes. The author proposes a novel transformer-based denoiser (based on the proposed cross-plane attention) for training the diffusion model, and show state-of-the-art 3D generation results.", "questions": "1. What is the prediction objective of DiffTF’s diffusion model? Is it \\epsilon-prediction, x_0 prediction or v-prediction?\n2. What’s the size of the shared triplane decoder?\n3. For the 3D-aware transformer part, as the triplane resolution is only 16x16 and the patch size is 2, this means that there’re only 8x8=64 tokens on each plane, and 64x3=192 tokens in total. Why not just do self-attention over all these 192 tokens, instead of using the cross-plane attention? Would this simpler architecture also work? In fact, looking at figure 3(b), such simple self-attention is already used in the 3D-aware encoder/decoder.\n4. How does the proposed cross-plane attention compared against Rodin’s 3D-aware convolution? Does it make sense to compare with this work qualitatively and quantitatively, as the cross-plane attention seems a major contribution of this work?\n\tRodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion\n5. It seems that the model is trained per-category on shapenet cars, chairs, airplanes? Why not train a class-conditioned generator just like what has been done on the OmniObject3d data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. It makes a lot of sense to use attention to model the inter- and cross-plane relations in a triplane, and learn such relations on massive amount of data. Ablation studies show that this is very effective.    2. The proposed method achieves state-of-the-art generation results on both Shapenet and OmniObject3D datasets in terms of geometry and texture quality, and diversity.", "weaknesses": "1. I feel that this paper’s writing is very confusing, and could be improved significantly for clarity.\n    - The method section shows that this is a two-stage model; but Fig. 2 seems to suggest that this is a single-stage model. The authors might want to make Fig. 2 clearer.\n    - For figure 4, since the generation results on OmniObject3D are class-conditioned, would it make sense to arrange the results in a specific order of classes in order for the readers to easily compare different methods?\n    - I have to dig into the appendix to find the architecture of the shared triplane decoder; there’s no reference to appendix in section 3.1. What’s worse, even looking at the appendix, I’m still unable to find out the width of the MLP layers for the triplane decoder.\n    - I find it a bit hard to imagine what exactly the architectures are for the different variants in the ablation “Studies of 3D-aware transformer modules”. The authors might want to elaborate more in the appendix, or better draw figures to the ablated architectures.\n    - Section 4.3 mentioned about the importance of triplane normalization. But this is never mentioned in the method section. In fact, the authors might want to show a histogram of the unnormalized triplane values in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tboxGF4CqE", "reviewer": "Reviewer_heX9", "summary": "The paper presents a 3D generation pipeline based on triplane diffusion. To achieve this, triplane fitting is conducted on the train set, and the output triplanes are used as the inputs to the diffusion model. Initializing as three pure noise planes, the paper uses a transformer-based architecture with cross-plane attention to enhance the 3D-awareness of the generation.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* The task of 3D generation, especially on large-scale open vocabulary datasets is challenging and well-motivated.\n* The paper is elegantly written and easy to understand.\n* Through ablation studies were done to justify its design choices.\n* Evaluations were done with reasonable metrics using both older ShapeNet dataset, and new large-scale OmniObject3D. Large margin of improvements can be observed.", "weaknesses": "* The contribution of the paper mainly comes from its transformer-based architecture and combining existing tricks in designing such architectures. The overall workflow has been introduced and well studied by prior works such as [1] and [2].\n* Upon the efforts of enhancing 3D-awareness of the generation, the rendered results still suffer from noisy geometry and texture details. It is not well-discussed what could be the source of these limitations.\n\n\n\n[1] Shue, et al. 3d neural field generation using triplane diffusion, in CVPR, 2023.\n[2] Chen, et al. Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction, in ICCV, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4tCoCodYHa", "reviewer": "Reviewer_veme", "summary": "This paper aims at increasing robustness over a wide range of objects for 3D content generation. It uses tri-plane as a compact 3D representation. To fuse the information on three axis, it proposes to use cross attention, named 3D-aware transformer. Experiments on Omniobjects and ShapeNet show that the proposed method produce realistic results.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The generated 3D objects are realistic. The geometry is better than prior works.\n2. Using cross-attention to achieve information fusing of tri-plane is interesting and also efficient in terms of 3D processing.\n3. Directly training a 3D diffusion model is hard. The paper is one of the good start.", "weaknesses": "1. The paper lacks the introduction of the whole pipeline which links each module together and also the training pipeline which demonstrate which module is pre-trained and which is trained in each step. I am still confused how Step 1 and 2 are related, especially how the fitted tri-plane are used, or is it only the tri-plane decoder is used.\n2. The improvement over GAN-based method is not obvious when the texture is on. They all seem realistic.\n3. The time complexity should be analyzed, with comparisons over prior methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)