DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Authors: Shentong Mo
Enze Xie
Ruihang Chu
Lanqing HONG
Matthias Nießner
Zhenguo Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Models
Transformers
3D Shape Generation

Abstract: Recent Diffusion Transformers (i.e., DiT) have demonstrated their powerful effectiveness in generating high-quality 2D images. However, it is unclear how the Transformer architecture performs equally well in 3D shape generation, as previous 3D diffusion methods mostly adopted the U-Net architecture. 
To bridge this gap, we propose a novel Diffusion Transformer for 3D shape generation, named DiT-3D, which can directly operate the denoising process on voxelized point clouds using plain Transformers. Compared to existing U-Net approaches, our DiT-3D is more scalable in model size and produces much higher quality generations.
Specifically, the DiT-3D adopts the design philosophy of DiT but modifies it by incorporating 3D positional and patch embeddings to aggregate input from voxelized point clouds.
To reduce the computational cost of self-attention in 3D shape generation, we incorporate 3D window attention into Transformer blocks, as the increased 3D token length resulting from the additional dimension of voxels can lead to high computation.
Finally, linear and devoxelization layers are used to predict the denoised point clouds. 
In addition, we empirically observe that the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet.
Experimental results on the ShapeNet dataset demonstrate that the proposed DiT-3D achieves state-of-the-art performance in high-fidelity and diverse 3D point cloud generation.

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Shentong Mo Enze Xie² Ruihang Chu Lewei Yao? Lanqing Hong" Matthias NieBner' Zhenguo Li²
MBBUAL, Huuawei Noah's i Ark Lab, 'CUHK, 4TUM https://DiT- 3D github.
Abstract
Recent Diffusion Transformers (e.g. DiT [1]) have demonstrated their powerful effectiveness in generating high-quality 2D images. However, it is still being determined whether the Transformer architecture performs equally well in 3D shape generation, as previous 3D diffusion methods mostly adopted the U-Net architecture. To bridge this gap, propose novel Diffusion Transformer for 3D shape generation, namely DīT-3D, which can directly operate the denoising process voxelized point clouds using plain Transformers. Compared to existing U-Net approaches, our DiT-3D more scalable in model size and produces much higher quality generations. Specifically, the DīT-3D adopts the design philosophy of DiT [1] but modifies it by incorporating 3D positional and patch embeddings adaptively aggregate input from voxelized point clouds. To reduce the compu- tational cost of self-attention in 3D shape generation, we incorporate 3D window attention into Transformer blocks, the increased 3D token length resulting from the additional dimension of voxels can lead to high computation. Finally, linear and devoxelization layers are used to predict the denoised point clouds. addition, our transformer architecture supports efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on the ShapeNet dataset demonstrate that the pro- high-fidelity and diverse 3D point cloud generation. In particular, our DiT-3D decreases the I-Nearest Neighbor Accuracy of the state-of-the-art method by 4.59 and increases the Coverage metric by 3.51 when evaluated on Chamfer Distance.
posed DiT-3D achieves state-of-the-art performance
1 Introduction
In recent times, there has been growing interest in exploring the potential of diffusion transformers
for high-fidelity image generation, as evinced by a series of scholarly works [1, 2, 3, 4]. Notably, a seminal work by Peebles al. [1] proposed the replacement of the widely-used U-Net backbone with scalable transformer. Specifically, the proposed method operates on latent patches by training latent 2D diffusion models. However, the efficacy of plain diffusion transformers for 3D shape generation has yet to be explored, as most existing 3D diffusion approaches continue to adopt the U-Net backbone.
Generating high-fidelity point clouds for 3D shape generation is a challenging and significant problem. Early generative methods [5, 6, 7] addressed this problem by directly optimizing heuristic loss objectives, such as Chamfer Distance (CD) and Earth Mover's Distance (EMD). More recent works [8, 9, 10, 11] have explored the usage of the generative adversarial network (GAN)-based
"Correspondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure 1: Examples of high- fidelity and diverse 3D point clouds produced from DiT-3D.
high-fidelity
and flow-based models to generate 3D point clouds from probabilistic perspective. Recently, researchers [12, 13, 14, 15] have turned to various denoising diffusion probabilistic models (DDPMs) to generate entire shapes from random noise. For instance, PVD [12] employed the point-voxel representation of 3D shapes as input to DDPMs. They reversed the diffusion process from observed point clouds to Gaussian noise by optimizing variational lower bound to the likelihood function. Recently, the Diffusion Transformer (DiT) [1. 2] has been shown to surpass the U-Net architecture in 2D image generation, owing its simple design and superior generative performance. Consequently, we investigate the potential of the Diffusion Transformer for 3D generation. However, extending the DiT to 3D poses two significant challenges: (1) Point clouds are intrinsically unordered, unlike images where pixels are ordered; and (2) The tokens in 3D space have additional dimension compared to 2D images, resulting in a substantial increase in computational cost.
This work introduces DiT-3D, novel diffusion transformer architecture designed for 3D shape generation that leverages the denoising process of DDPM on 3D point clouds. The proposed model inherits the simple design of the modules in DīT-2D, with only minor adaptations to enable it to generalize to 3D generation tasks. To tackle the challenge posed by the unordered data structure of point clouds, we convert the point cloud into voxel representation. DÝT-3D employs 3D positional embedding and 3D patch embedding on the voxelized point clouds to extract point-voxel features and effectively process the unordered data. Furthermore, address the computational cost associated with large number of tokens in 3D space, we introduce a 3D window attention operator instead of the vanilla global attention in DiT-2D. This operator significantly reduces training time and memory usage, making DiT-3D feasible for large-scale 3D generation tasks. Finally, we utilize linear and devoxclization layers to predict the noised point clouds in the reversed process to generate final 3D shapes.
In order to address the computational cost associated with a large number of tokens in 3D space, we also introduce a parameter-efficien tuning method utilize the pre-trained DiT-2D model on ImageNet as initialization for DiT-3D (window attention shares the same parameters with vanilla attention). Benefiting from the substantial similarity between the network structure and parameters of DiT-3D and DiT-2D, the representations learned on ImageNet significantly improve 3D generation, despite the significant domain disparity between 2D images and 3D point clouds. To our knowledge, are the first achieve parameter-efficient fine-tuning from 2D ImageNet pre-trained weights for high-fidelity and diverse 3D shape generation. In particular, we highly decrease the training parameters from 32.8MB to only 0.09MB.
We present a comprehensive evaluation of DiT-3D on diverse set of object classes in the ShapeNet benchmark, where it achieves ffte-off-the-nd performance compared to previous Mon-DDPM and DDPM-based shape generation methods. Qualitative visualizations further emphasize the efficacy of DiT-3D in generating high-fidelity 3D shapes. Extensive ablation studies confirm the significance of 3D positional embeddings, window attention, and 2D pre-training in 3D shape generation. Moreover, demonstrate that DiT-3D is casily scalable regarding patch sizes, voxel sizes, and model sizes. Our

---

findings align with those of DiT-2D, where increasing the model size leads to continuous performance improvements. In addition, our aarameter-efficien fine-tuning from DiT-2D ImageNet pre-trained weights highly decreases the training parameters while achieving competitive generation performance. By only training 0.09MB parameters of models from the source class to the target class, we also achieve comparable results of quality and diversity in terms of all metrics.
Our main contributions can be summarized as follows:
We present DiT-3D, the first plain diffusion transformer architecture for point cloud shape generation that can effectively perform denoising operations on voxelized point clouds.
We make several simple yet effective modifications on DřT-3D, including 3D positional and patch embeddings, 3D window attention, and 2D pre-training on ImageNet. These modifications significantly improve the performance of DiT-3D while maintaining efficiency.
Extensive experiments on the ShapeNet dataset demonstrate the state-of thetthe superiority of DīT-3D over previous non-DDPM and DDPM baselines in generating high-fidelity shapes.
2 Related Work
3D Shape Generation. 3D shape generation aims synthesize high-fidelity point clouds or meshes using generative models, such as variational autoencoders [16, 17, 18], generative adversarial net- works [19, 8, 20], and normalized flows [9, 10, 11]. Typically, PointFlow [9] utilized a probabilistic framework based on the continuous normalizing flow to generate 3D point clouds from two-level hierarchical distributions. ShapeGF [21] trained score-matching energy-basee network to learn the distribution of points across gradient fields using Langevin dynamics. More recently, GET3D [14] leveraged signed distance field (SDF) and texture field as two latent codes learn generative model that directly generates 3D meshes. In this work, we mainly focus on denoising diffusion probabilistic models for generating high-fidelity 3D point clouds starting from random noise, where point and shape distributions are not separated.
Diffusion Models. Diffusion models [22, 23, 24] have been demonstrated to be effective in many generative tasks, such as image generation [25], image restoration [26], speech generation [27], and video generation [28]. Denoising diffusion probabilistic models (DDPMs) [22, 23] utilized a forward noising process that gradually adds Gaussian noise to images and trained a reverse process that inverts the forward process. In recent years, researchers [29, 12, 13, 30, 15, 31, 32] have tried to explore diverse pipelines based on diffusion probabilistic models to achieve 3D shape generation. For example, PVD [12] applied DDPM based on PVCNNs [33] on the point-voxel representation of 3D shapes with structured locality into point clouds. To improve the generation quality, LION [13] used two DDPMs to learn a hierarchical latent space based on global shape latent representation and a point-structured latent space separately. Different from them, we will solve the 3D shape generation problem in our approach by designing a plain transformer-batt based architecture backbone to replace the U-Net backbone for reversing the diffusion process from observed point clouds to Gaussian noise. Meanwhile, our 3D plain diffusion transformer supports multi-class training with learnable class embeddings as the condition and parameter-efficient fine-tuning with modality and domain transferability differ from DDPM- based 3D generation approaches discussed above.
Transformers in Diffusion Generation. Diffusion Transformers [1, 2, 3, 4] have recently shown their
impressive capacity to generate high-fidelity images. For instance, Diffusion Transformer (DiT) [1]
proposed a plain diffusion Transformer architecture to learn the denoising diffusion process on
latent patches from pre-trained pre-trained variational autoencoder model in Stable Diffusion [34]. U-ViT [2] incorporated all the time, condition, and noisy image patches as tokens and utilized a Vision transformer( ViT) [35]-based architecture with long skip connections between shallow and deep layers. More recently. UniDiffuser [3] designed a unified transformer for diffusion models to handle input types of different modalities by learning all distributions simultaneously. While those diffusion transformer approaches achieve promising performance in 2D image generation, how a plain diffusion transformer performs on 3D shape generation is still being determined. contrast, we develop novel plain diffusion transformer for 3D shape generation that can effectively perform denoising operations on voxelized point clouds. Furthermore, the proposed DiT-3D can support parameter-efficieet fine-tuning with transferability across modality and domain.
3

---

2D Latent Diffusion Transformer
3D Voxel Diffusion Transformer
Scale
Scale
FFN
Reshape
Y2.Pa
Linear
Y2.82
Scale, Shift
Scale, Shift
Layer Norm
Layer Norm
Layer Norm
Layer Norm
N
DiT-2D Block
N
DiT-3D Block
Scale
Scale
Global
30 Window
+20
Embed
Attention
+3D PE
Embed
Yu.Pi
Yr.fg
Scale,
Latent
Timestop
Vosel
Timestep
Layer
Layar
MLP
Input Tokens
Condition
Input
Condition
Illustration of the proposed Diffusion Transformers (DIT-3D) for 3D shape generation. The plain diffusion transformer takes voxelized point clouds as input, and a patchification operator is used generate token-level patch embeddings, where 3D positional embeddings are added together. Then, multiple transformer blocks based on 3D window attention extract point-voxel representations from all input tokens. Finally, the unpatchified voxel tensor output from a linear layer devoxelized to predict the noise in the point cloud space.
Figure 2:
Method
Given set of 3D point clouds, we aim to learn plain diffusion transformer for synthesizing new high-fidelity point clouds. We propose a novel diffusion transformer that operates the denoising process of DDPM on voxelized point clouds, namely DiT-3D, which consists of two main modules: Design DiT for 3D Point Cloud Generation in Section 3.2 and Efficient Modalit//Doman Transfer with Parameter-efffcienn Fine-tuning in Section 3.3.
3.1 Preliminaries
In this section, we first describe the problem setup and notations and then revisit denoising diffusion probabilistic models (DDPMs) for 3D shape generation and diffusion transformers 2D images.
Problem Setup and Notations. Given a set S = {pr} of 3D shapes with M classes, our goal is to train a plain diffusion transformer from these point clouds for generating high- figh-fidelity point clouds. For each point cloud p.. we have N points for x. y, coordinates, that is p; E IR N'3 Note that have class label for the 3D shape p;, which is denoted {yi M with yi for the ground-truth category entry as 1. During the training, we take the class label as input to achieve classifier-fret guidance in conditional diffusion models, following the prior diffusion transformer (i.e., DiT [1]) images.
Revisit DDPMs on 3D Shape Generation. To solve the 3D shape generation problem, previous work [12] based on denoising diffusion probabilistic models (DDPMs) define forward noising process that gradually applies noise to real data Xę (x-)) \xxx vI 3, 8,I), where B, is Gaussian noise value between 0 and In particular, the denoising process produces a series of shape variables with decrcasing levels of noise, denoted xy, Xy-1..... Xg, where xy is sampled from a Gaussian prior and Xg is the final output. With the reparameterizztion trick, we can have x, Vā,X, + -I- o,cc where e ~ I). a; = 1 3r and ā, = -I a;-
For the reverse process, diffusion models are trained to learn denoising network 0 for inverting forward process corruption as po (x+ xx)) = (x; p (x,, (), of)). The training objective is to maximize variational lower bound of the negative log data likelihood that involves all of xo. XT

---

as
L -o + DxL DxL X xo)llpo (x
where DxL (-1l)) denotes the KL divergence measuring the distance between two distributions. Since both po(x: (xxx((x))) and g(x+11 Xx, Xo) are Gaussians, we can reparameterize [x, 1) to predict the noise (x( 1). In the end, the training objective can be reduced to a simple mean-squared loss between the model output eØ (xe: () and the ground truth Gaussian noise e as: <imple - ||e Ex(x,, 1)]l After po(xx-1 (xx))) is trained, new point clouds can be generated by progressively sampling X ~ (x xx)) by using the reparameterization trick with initialization of XT ~ (0.
Revisit Diffusion Transformer (DiT) on 2D Image Generation. To generate high-fidelity 2D images, DiT proposed to train latent diffusion models (LDMs) with Transformers the backbone, consisting of two training models. They first extract the latent code z from an image sample x using an autoencoder with encoder f-uc(-) and decoder (-ec()), that is, z feod (x). The decoder is used to reconstruct the image sample & from the latent code z, i.e., & = face (z) Based on latent codes z, latent diffusion transformer with multiple designed blocks is trained with time embedding t and class embedding c, where a self-attention and a feed-forward module are involved in each block. Note that they apply patchification latent code to extract sequence of patch embeddings and depatchification operators are used to predict the denoised latent code z.
Although DDPMs achieved promising performance 3D shape generation, they can only handle single-class training based on PVCNNs [33] as the encoder to extract 3D representations, and they cannot learn explicit condiionn embeddings. Furthermore, their single-class pre-trained model to new classes with parameter-efficien fine-tuning. Mcanwhile, empirically observe that the direct extension of DiT [1] on point clouds does not work. To address this problem, we propose a novel plain diffusion transformer for 3D shape generation that can effectively achieve the denoising processes voxelized point clouds, illustrated in Figure 2.
are not able to directly transfer
3.2 Diffusion Transformer for 3D Point Cloud Generation
To enable denoising operations using a plain diffusion transformer, propose several adaptations to 3D point cloud generation in Figure within the framework of DiT [1]. Specifically, our DiT-3D model accepts voxelized point clouds token-level patch embeddings. We add 3D positional embeddings to these embeddings and extract point-voxel representations from all input tokens using multiple transformer blocks based on 3D window attention. Finally, we apply a devoxelized linear layer to the unpatchified voxel output, allowing us predict the noise in the point cloud space.
input and employs patchification operator to generate
Denoising on Voxelized Point Clouds. Point clouds are inherently unordered, unlike images where pixels follow specific order. We encountered difficulty in our attempt to train diffusion transformer on point coordinates due to the sparse distribution of points in the 3D embedding space. To address this issue, we decided to voxelize the point clouds into dense representations, allowing the diffusion transformers to extract point-voxel features. Our approach differs from DiT [1], which utilizes latent codes train the latent diffusion transformer. Instead, we directly train the denoising process voxelized point clouds using the diffusion transformer. For each point cloud P; E R NX3 with N points for x, y: 2 coordinates, we first voxelize it as input V, R XVVXx
3D Positional and Patch Embeddings. With the voxel input v, E VVVVVX we introduce patchification operator with a patch size p
p to generate a sequence of patch tokens t E Lx3 = (V/p)² denotes the total number of patchified tokens. A 3D convolution layer is applied on patch tokens to extract patch embeddings E LxD. where D is the dimension of embeddings. To adapt to our voxelized point clouds, we add frequency-based sine-cosine 3D positional embeddings instead of the 2D version in DiT [1] to all input tokens. Based on these patch-level tokens, we introduce time embeddings t and class embeddings c as input to achieve multi-class training with learnalle class embeddings as the condition, which differs from existing 3D generation approaches with U-Net as the backbone.
3D Window Attention.
Due to the increased token length resulting from the additional dimension in 3D space, the computational cost of 3D Transformers can be significantly high. To address this issue, introduce efficient 3D window attention into Transformer blocks blocks to propagate voxet
5

---

features in efficient memory usage. For the original multi-head self-attention process with each of the heads Q, K, V have the same dimensions D, where L (V/p)² 3 the length of input tokens, we can have the attention operator as:
QKT T VDn V)
Attention(Q,K,, Softmax
(2)
where D2 is the dimension size of each head. The computational complexity of this process is OLL²), which will be largely expensive for high voxel resolutions. Inspired by [36], extend the 2D window attention operator to a 3D one for 3D input tokens instead of vanilla global attention.
This process uses window size of R to reduce the length of total input tokens
L
K = Reshape(
D. ).)))
R3
K Linear( D R², ())
where K is the input tokens to be reduced. Reshape ( (1ss D. R²) denotes to reshape K to the L x (D. I²²), and Linear(Ciss in. (.u denotes to a linear layer with a Cin- dimensional tensor as input and a -dimenninnii tensor as output. Therefore, the new K has the shape of RS x D. As a result, the complexity of the self-attention operator in Equation (2) is reduced from (lL) to (2) In our experiments, we set to the default setting.
one with shape of Te
Devoxelized Prediction. Since the transformers blocks are implemented on voxelized point clouds, we can not directly use a standard linear decoder predict the output noise e t) from point clouds. In order to generate the output noise, we devoxelize output tokens from the linear decoder. We first apply the final layer norm and linearly decode each token into a p x p X p x L x tensor, where L is the total number of input tokens. Then we unpatchify the decoded token into voxel tensor with the shape of V x x V x 3. Finally, the unpatchified voxel tensor devoxelized into a N x 3 tensor as the output noise (x,, !), matching with the ground truth Gaussian noise in the point cloud space.
Model Scaling, Our DīT-3D is designed to be scalable, adapting to varying voxel sizes, patch sizes, and model sizes. Specifically, can flexibly accommodate voxel dimensions of 16, 32, 64, patch dimensions of 2, 4, 8, and model complexity ranging from Small, Base, Large and Extra I Large, as demonstrated in DiT [1]. For instance, a model designated as DiT-3D-S/4 refers that utilizes the Small configuration of the DiT model [1], with patch size p of 4.
3.3 Efficient Modality/Domain Transfer with Parameter-efficien Fine-tuning
Leveraging the scalability of the plain diffusion transformer, we investigate parameter-efficien fine- tuning for achieving modality and domain transferability. To facilitate modality transfer from 2D to 3D, we can leverage the knowledge pre- pre-trained on large-scale 2D images using DiT [1]. For domain transfer from source class to target classes, we train DiT-3D on a single class (e.g. chair) and transfer the model's parameters to other classes (e.g. airplane, car).
Modality Transfer: 2D (emageNet) - 3D (ShapeNet). As large-scale pre-trained DiT checkpoints are readily available, we can skip training our diffusion transformer from scratch. Instead, we can load most of the weights from the DiT [1] pre-trained on ImageNet [37] into our DiT-3D and continue with fine-tuning. To further optimize training efficiency, we adopt the parameter-efficient fine-tuning approach described in recent work, DiffFit [4], which involves freezing the majority of parameters and only training the newly-added scale factors, bias term, normalization, and class condition modules. It's worth noting that we initialize y to 1, which is then multiplied with the frozen layers.
Domain Transfer: Source Class + Target Class. Given a pre-trained DiT-3D model on chair data, we can the parameter-efficient fine- fine-tuning approach to extend its applicability to new categories. Specifically, following the same methodology as described above, we leverage the fine-tuning strategy of DiffFit and obtain satisfactory generation results.

---

Table 1: Comparison results (%) on shape metrics of our DiT-3D and baseline models,
Chair
Airplane
Car
Method
I-NNA (1)
COV (1)
1-NNA (
COV (1)
I-NNA (J)
COV (t)
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
F-GAN [8]
83.69
99.70
24.27
15.13
98
96.79
30.12
14.32
94.46
99.01
19.03
6.539
I-GAN (CD)
68.58
83.84
41.99
29.31
87.30
93.95
38.52
21.23
66.49
88.78
38.92
23.58
L-ANN (EMD)
71.90
64.65
38.07
44.86
89.49
76.91
38.27
38.52
71.16
66.119
37.78
45.17
PointFlow [9]
62.84
60.57
42.90
50.00
75.68
70.74
47.90
46.41
58.10
$6.25
46.88
50.00
Soffllow [10]
59.21
60.05
41.39
47.43
76.05
65.80
46.91
47.90
64.77
60.09
42.90
44,60
SetVAE [18]
58.84
60.57
46.83
44.26
76.54
67.65
43.70
48.40
59.94
59,94
49.15
46.59
DPF-Net [11]
62.00
58.53
44.71
48.79
75.18
65.55
46.17
48.89
62.35
54.48
45.74
49.43
DPM [29]
60.05
74.77
44.86
35.50
76.42
86.91
48.64
33.83
68.89
79.97
44.03
34.94
PVD [12]
57.09
60.87
36.68
49.24
73.82
64.81
48.88
52.09
54,55
53.83
41.19
50.56
LION [13]
53.70
52.34
48.94
52.11
67.41
61.23
47.16
49.63
53.41
51.14
50.00
56.53
GRT3D [14]
75.26
72.49
43.36
42.77
75.26
72.49
15.04
18.38
MeshDiffusion [15]
53.69
57.63
46,00
46.71
66.44
76.26
47.34
42.15 15
81.43
87.84
34.07
25.85
DIT-3D (ours)
49.11
50.73
52.45
54.32
62.35
58.67
53.16
54.39
48.24
49.35
50.00
56.38
3.4
Relationship to DiT [1]
Our DiT-3D contains multiple different and efficient designs for 3D shape generation compared with DiT [i] on 2D image generation:
We effectively achieve the diffusion space on voxelized point clouds, while DiT needs the latent codes from a pre-trained variational autoencoder the denoising target.
Our plain diffusion transformer first incorporates frequency-basee sine-cosine 3D positional embeddings with patch embeddings for voxel structure locality.
We are the first to propose efficient 3D window attention in the transformer blocks for reducing the complexity of the self-attention operator in DIT.
We add devoxelized operator to the final output of the last linear layer from DiT for denoising the noise prediction in the point cloud space.
Experiments
4.1 Experimental Setup
Datasets. Following most previous works [12, 13], use ShapeNet [38] Chair, Airplane, and Car as our primary datasets for 3D shape generation. For each 3D shape, we sample 2,048 points from 5,000 provided points in [38] for training and testing. We also use the same dataset splits and pre- pre-processing in PointFlow [9], which normalizes the data globally across the whole dataset.
Evaluation Metrics. For comprehensive comparisons, we follow prior work [12, 13] and use Chamfer Distance (CD) and Earth Mover's Distance (EMD) as our distance metrics in computing I-Nearest Neighbor Accuracy (1-NNA) and Coverage (COV) as main metrics to measure generative quality. I-NNA calculates the leave-one-out accuracy of the 1-NN classifier to quantify point cloud generation performance. which is robust and correlates with generation quality and diversity. A lower I-NNA score is better. COV measures the number of reference point clouds matched to at least one generated shape, correlating with generation diversity. Note that higher COV score is better but does not measure the quality of the generated point clouds since low-quality but diverse generated point clouds can achieve high COV scores.
Implementation. Our implementation size is 32 x 32 32 x 3, i..., V = 32. The final linear layer is initialized with zeros, and other weights initialization follows standard techniques in ViT [35]. The models were trained for 10,000 epochs using the Adam optimizer [40] with learning rate of le 4 and a batch size of 128. We set T= 1000 for experiments. the default setting, we use S/4 with patch size p = 4 as the backbone. Note that we utilize 3D window attention in partial blocks (i.e., 0,3,6,9) and global attention in other blocks.
based on the PyTorch [39] framework. The input voxel
7

---

Figure 3: Qualitative visualizations of high-fidhe figh-fidelity and diverse 3D point cloud generation.
4.2 Comparison to State-of-the-art Works
In this work, we propose a novel and effective diffusion transformer for 3D shape generation. In order to validate the effectiveness of the proposed DiT-3D, we comprehensively compare it to previous non-DDPM and DDPM baselines. 1) r-GAN, I-GAN [8]: (201'ICML)) generative models based on GANs trained on point clouds (I-GAN) and latent variables (I-GAN); 2) PointFlow [9] (2019' ICCV): probabilistic framework to generate 3D point clouds from two-level hierarchy of distributions with continuous normalizing flow; 3) SoftFlow [10] (2020' NeurIPS): a probabilistic framework for training normalizing flows on manifolds to estimate the distribution of various shapes; 4) SerVAE [18] (2021 (CVR): a hierarchical variational autoencoder for sets to learn latent variables for coarse-to- dependency and permutation invariance: 5) DPF-Net [11] (2020' ECCV): discrete latent variable network that builds on normalizing flows with affine coupling layers; 6) DPM [29] (202I'ICCV): the first DDPM approach learn the reverse diffusion process for point clouds Markov chain conditioned on shape latent; 7) PVD [12] (2021 (CCC): a strong DDPM baseline based on the point-voxel representation of 3D shapes; LION [13] (2022' NeurIPS): a recent method based two hierarchical DDPMs in global latent and latent points spaces; 9) GET3D [14] :S)r a generative model that directly generates explicit textured 3D meshes based two latent codes (a 3D SDF and a texture field); 10) MisfDiffusion [15] (2023'CLLR) a very recent DDPM method using graph structure of meshes and deformable tetrahedral grid parametrization of 3D mesh shapes.
For chair generation, we report the quantitative comparison results in Table 1. As can be seen, we achieved the best performance in terms of all metrics compared to previous Non-DDPM and DDPM baselines. In particular, the proposed DiT-3D significantly outperforms DPF-Net [11], the current state-of-the-arr normalizing flows baseline. decreasing by 12.89 I-NNA @ CD & 7.80 I-NNA @ EMD, and increasing by 7.74 COV @ CD 3.8 COV EMD. Moreover, we achieve superior performance gains compared to Misfuisfusion [15], the current state-of-the t-thhhe-n art DDPM baseline meshes, which implies the importance replacing the U-Net with plain diffusion transformer from observed point clouds for generating high-fidelity 3D shapes. Meanwhile, our DiT-3D outperforms LION [15] by large margin, where we achieve the performance gains of 4.59 1 I-NNA @CD & 1.61 1-NNA @ FMD, and 3.51 COV @CD & 2.21 COV EMD. These significant improvements demonstrate the superiority of our method in 3D shape generation. In addition. significant gains in airplane and car generations can be observed in Table These qualitative results also showcase the effectiveness of applying a plain diffusion transformer to operate the denoising process from point clouds for generating high-fidelity and diverse shapes, as shown in Figure 3.
4.3 Experimental Analysis
In this section, we performed ablation studies to demonstrate the benefit of introducing three main 3D design components (voxel diffusion, 3D positional embeddings, and 3D window attention) in 3D shape generation. We also conducted extensive experiments explore the efficiency of 3D window attention, modality and domain transfcrability, and scalability.
8

---

Table 2: Ablation studies on 3D adaptation components of our DiT-3D.
Voxel Diffusion
3D
3D Window
Training Cost (hours)
I-NNA (L)
COV (1)
Pos Embed
Attention
CD
EMD
CD
EMD
X
86.53
99.86
99.93
7.768
4.653
X
91.85
67.46
69.47
38.97
41.74
/
X
91.85
51.99
49.94
54.76
57.37
/
/
/
41.67
49.11
50.73
52.45
54.32
Table 3: Transferability studies on modality and domain with parameter-efficiet fine-tuning.
ImageNet Pre-triin
Effiiint
Params
I-NNA ())
COV (1)
Source
Target
I-NNA
(1)
(1)
Fine-tuning
(MB)
EMD
EMD
Domain
Domain
(MB)
CD
EMD
CD
EMD
32.8
51.99
49.94
54.76
57.37
Airplane
Clair
Chair
Chair
33.8
52.56
51,99
49.94
54.76
56.32
57.37
x
3288
49.07
49.76
53.26
55.75
Airplane
Airplane
32.8
62.11
58.31
38.04
54.88
/
0.09
50.87
50.23
52.59
55.36
Chair
Airplane
0.09
63.58
59.17
53.25
53.68
(a) Modality transfer.
(b) Domain transfer.
Ablation on 3D Design Components. In order to validate the effectiveness of the introduced 3D adaptation components (voxel diffusion, 3D positional embeddings, and 3D window attention), ablate the necessity of each module and report the quantitative results in Table 2. Note that no voxel diffusion means we directly perform the denoising process on point coordinates without voxelized point clouds and devoxelization prediction. We can observe that adding bearable voxel diffusion to the vanilla baseline highly decreases the results of 1-NNA (by 32.40 @CD and 30.46 @AUC) and increase the performance of COV (by 31.202 @ CDD and 37, 07.77 @ EMD). which demonstrates the benefit of voxelized point clouds and devoxelization prediction in denoising process for 3D shape generation. Meanwhile, introducing 3D positional embedding in the baseline with voxel diffusion also increases the shape generation performance in terms of all metrics. More importantly, incorporating 3D window attention and two previous modules together into the baseline significantly decreases the training cost by 44.86 hours and results of I-NNA by 50.75 @CD and 49.2 @ EMD, and raises the performance of COV by 44.682 @CD and 49.667 @ EMD. These improving results validate the importance of the proposed 3D adaptation components in the plain diffusion transformer to operate the denoising process from observed point clouds for 3D shape generation.
Influence of 2D Pretrain (ImageNet), In order show the modality transferability of the proposed approach from 2D ImageNet pre-trained weights to 3D generation with parameter-efficient fine-tuning, we report the ablation results of ImageNet pre-train and efficient fine-tuning on chair generation in Table 3a. From comparisons, two main observations can be derived: 1) With the initialization with 2D ImageNet pre-trainee pre-trained weights, the proposed DiT-3D improves the quality of shape generation by decreasing 1 1-NNA by 2.92@t @CD and 0.18 @ EMD. 2) Incorporating harameter-efffiiee fine-tuning into 2D ImageNet pre-trained weights highly decreases the training parameters while achieving competitive generation performance.
Transferability in Domain. addition, we explore the parameter-efficient fine-tuning for domain transfcrability in Table 3b. By only training 0.09MB parameters of models from the source class to the target class, we can achieve a comparable performance of quality and diversity in terms of all metrics. These results indicate that our DiT-3D can support flexible transferability on modality and domain, which differs from previous 3D generation methods [12, 13] based on U-Net as the backbone of DDPMs.
Scaling Patch size, Voxel size and Model Size. To explore the scalability of our plain diffusion transformer to flexible designs, we ablate the patch size from {2, 4, 8), voxel size from {16, 32, 64), and the model size from (S/4, B/4, L/4, XIJ4). As seen in Table 4a, when the patch size is 2, the proposed DiT-3D achieves the best performance. This trend is also observed in the original DiT [1] work for 2D image generation. In addition, increasing the voxel size from 16 64 for the input of the diffusion denoising process raises the performance in terms of all metrics, as shown in Table 4b. More importantly, we can still observe performance gains by scaling up the proposed plain diffusion transformer to XL/4 when the model is trained for 2,000 epochs. These promising results further demonstrate the strong scalability of our DiT-3D to flexible patch size, voxel size, and model sizes for generating high-fidelity 3D shapes.

---

Table 4: Scalability studies on flexible patch, voxel, and model sizes.
Patch Size
I-NNA (1)
COV
(t)
Voxel Size
I-NNA (J)
COV (t)
Model
Panms (MB)
I-NNA
COV (1)
CD
EMD
CD
EMD
CD)
EMD
CD
EMD
8
53.84
51.20
50.01
52.49
16
54,00
50.60
50.73
52.26
56.31
35.82
47.21
51.99
49.94
54.76
57.37
32
51.99
49.94
54.76
57.37
130.2
52.96
55.59
54.91
5009
53.00
2
51.78
49.69
54.54
55,94
64
50.32
49.73
55.45
57.32
51.95
$2.90
52.71
(a) Patch size.
(b) Voxel size.
(c) Model size.
Conclusion
In this work, we present DiT-3D, a novel plain diffusion transformer for 3D shape generation, which can directly operate the denoising process voxelized point clouds. Compared to existing U-Net approaches, our DiT-3D more scalable in model size and produces much higher quality generations. Specifically, we incorporate 3D positional and patch embeddings to aggregate input from voxelized point clouds. We then incorporate 3D window attention into Transformer blocks to reduce the computational cost of 31) Transformers, which can be significantly high due to the increased token length resulting from the additional dimension in 3D. Finally, we leverage linear and devoxelization layers predict the denoised point clouds. Due the scalability of the Transformer, DīT-3D can easily support parameter-efficien fine-tuning with modality and domain transferability, Empirical results demonstrate the state-of-the-a performance of the proposed DiT-3D in high-fidelity and diverse 3D point cloud generation.
References
[1] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022. 1, 2, 3, 4, 5, 6, 7,9
[2] Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffuusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023. 1,
[3] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303 365555,555..... 2023. 1,
[4] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficine fine-tuning. arXiv preprint arXiv:2304. 06648, 2023. 3,
[5] Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages (05-613,220 2017.
[6] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 216-224, 2018.
[7] Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Bong- soo Choy, and Silvio Savarese. Deformnet: Free-form deformation network for 3d shape reconstruction from a single image. Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pages 858 866, 2017.
[8] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning rep- resentations and generative models for 3d point clouds. In Proceedings of the International Conference on Machine Learning (ICML), 2018. 1, 7,8
[9] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with contimuous normalizing flows. In Proceedings of the IFEF/CVF International Conference on Computer Vision (ICCV), pages 4541-4555555 2019. 1, 3, 7,8 8
10

---

[10] Hyeongju Kim, Hyeonseung Lee. Woohyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow on manifolds. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 7,
[11] Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. In Proceedings of the European Conference on Computer Vision (ECCV), page 694-710, 2020. 1,3,7 7,8
[12] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel voxel diffusion. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 5826-5835, 2021. 4,
[13] Xiaohui Zeng, Arash Vahdar, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 3, 8,
[14] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: generative model of high quality 3d textured shapes learned from images. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS), 2022. 2, 3, 7.8
[15] Zhen Liu, Yao Feng, Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In Proceedings of International Conference on Learning Representations (ICLR), 2023. 7,8
[16] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 206-215, 2018.
[17] Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[18] Jinwoo Kim, Jachoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composition for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021.
3, 7,
[19] Diego Valsesia, Giulia Fracastoro, and Enrico Magli. Learning localized generative models for 3d point clouds via graph convolution. In Proceedings of International Conference on Learning Representations (ICLR), 2019.
[20] Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversar- ial network based on tree structured graph convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3859-3868, 2019. 3
[21] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 3
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS), pages 6840- 6851, 2020.
[23] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- score-based generative modeling through stochastic differential equations. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[24] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv, 2021.
[25] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, arXiv preprint 2xXiv:2005.11487, 2022. 3
11

---

[26] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07633 2021. 3
[27] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In Proceedings of International Conference on Learning Representations (ICLR), 2021. 3
[28] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
[29] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2837-2845, 2021. 7,7.8
[30] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842, 2022.
[31] Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2023. 3
[32] Ruihang Chu, Enze Xie, Shentong Mo, Zhenguo Li, Matthias Nießner, Chi- Shi-Wing Fu, and Jiaya Jia. Diffcomplete: Diffusion-basst generative 3d shape completion. arXiv preprint arXiv:2306. 16329, 2023. 3
[33] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel enn for efficient 3d deep learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS),
2019. 3,5
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[35] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations (ICI.R), 2021. 3,7
[36] Yanghao Li, Hanzi Mao, Ross B. Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large- Scale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248,255,2200 2009. 6
[38] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03011 2015. 7
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 8026-8037, 2019.
[40] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014. 7
12


Reviews:
[{"review_id": "lfjorB4lyd", "reviewer": "Reviewer_RjmT", "summary": "The paper introduces DiT-3D, a groundbreaking diffusion transformer for 3D shape generation. It addresses the limitations of previous 3D diffusion methods that mainly relied on the U-Net architecture. DiT-3D leverages the power of Transformers to directly operate the denoising process on voxelized point clouds, resulting in superior scalability and high-quality generations. The authors incorporate 3D positional and patch embeddings to aggregate input from voxelized point clouds and mitigate the computational cost of self-attention by employing 3D window attention in Transformer blocks. The proposed DiT-3D achieves state-of-the-art performance on the ShapeNet dataset, showcasing its ability to generate diverse and high-fidelity 3D point clouds.", "questions": "1. Have the authors considered exploring diffusion in the latent space? This may potentially enhance the overall performance and inference speed.\n2. It would be more valuable to investigate multi-category generation, such as training on entire ShapeNet-13, ShapeNet-55, or even larger datasets like Objaverse [Deitke et al., 2023]. Previous work [Sanghi et al., 2022, 2023] suggests that voxel representation could aid in generalization to some extent.\n\n\n[Deitke et al., 2023] Objaverse: A universe of annotated 3d objects.  In CVPR.\n\n[Sanghi et al., 2022] CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation.  In CVPR.\n\n[Sanghi et al., 2023] CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language.  In CVPR.", "limitations": "N.A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. DiT-3D achieves state-of-the-art performance in single-category point cloud generation, demonstrating its effectiveness in generating high-quality 3D shapes.\n2. The utilization of pre-trained DiT-2D checkpoints from ImageNet to improve DiT-3D on ShapeNet showcases the transferability of 2D diffusion models to the 3D domain, which is an interesting and promising approach.\n3. The model is concise, and the paper is well-written, accompanied by clear and visually appealing illustrations.", "weaknesses": "1. The authors only conducted unconditional generation experiments for single-category and three-category cases, limiting the application fields.\n2. Although the window attention technique is employed to mitigate computational costs, there are concerns regarding the generation speed, when operating on 32 * 32 * 32 (even 64 * 64 * 64) voxel grids.\n3. Most DiT-3D models, except for DiT-3D-S, have parameters exceeding 100 million, considerably higher than other existing 3D generation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer RjmT,\n\n\nThank you for appreciating our approach. We will address your comments below.\n\n\n> Only single-category and three-category cases.\n\nThanks for pointing this out. To validate the generalizability to more categories, we train our DiT-3D on the large-scale ShapeNet-55 dataset with 55 diverse classes covering vehicles, furniture, and daily necessities. We compare the newly trained model with single-category and three-category cases on Chair generation in the Table below. Our models trained on 55 categories achieves competitive results on generating high-fidelity point clouds, and achieves the best performance on diversity.\n\n| Train Class    | Test Class | 1-NNA CD ($\\downarrow$) | COV CD   ($\\uparrow$) |\n| ---- | :----: | :----: | :----: | \n| Chair                | Chair      | **51.99**        | 54.76       |\n| Chair, Car, Airplane | Chair      | 53.35        | 52.81       |\n| All 55 classes       | Chair      | 52.68        | **57.87**       |\n\n> Concerns regarding the generation speed.\n\nThis is a good suggestion! To solve your concerns, we tested our DiT-3D on the Chair generation speed in the Table below, where it is tested on a single V100-32GB GPU with a batch size of 1. When the voxel size is larger, our method with efficient window attention achieves better generation results while maintaining similar inference times.\n\n| Voxel Size | 1-NNA CD ($\\downarrow$) | COV CD   ($\\uparrow$) |  Inference Time ($\\downarrow$) |\n| ---- | :----: | :----: | :----: | \n| 32x32x32   | 51.99        | 54.76       | **2.5s**                |\n| 64x64x64   | **50.32**        | **55.45**       | 3.3s                |\n\n\n> High parameters than other existing 3D generation methods.\n\nSorry for causing the confusion. To clarify this, we compared our DiT-3D-S with PVD and LION on the parameters and performance of Airplane generation in the Table below. Compared to PVD, DiT-3D-S has comparable parameters but achieves significantly improving generation performance. Compared to LION, the recent state-of-the-art method, our DiT-3D-S with 1/3 parameters achieves much better performance.\n\n|  Method  | Params | 1-NNA CD ($\\downarrow$) | COV CD   ($\\uparrow$) |\n| ---- | :----: | :----: | :----: | \n|    PVD   | **27.65M** |     73.82    |    48.88   |\n|   LION   |  110M  |     67.41    |    47.16   |\n| DiT-3D-S | 32.81M |     **62.35**    |    **53.16**   |\n\n\n> Diffusion in the latent space.\n\nThis is a good suggestion! While it is possible to extend our diffusion transformer to the latent space, it will require pre-training a strong 3D encoder-decoder on this data. Meanwhile, our DiT-3D has already achieved advantageous overall performance and maintained comparable generation speed. We will leave this for future work. \n\n> Multi-category generation on ShapeNet-55.\n\nPlease see the first response to the reviewer. \n\n\n"}, {"review_id": "vjAw5LwXMn", "reviewer": "Reviewer_zPmE", "summary": "This paper proposes a Diffusion Transformer for 3D shape generation (point cloud), named DiT-3D, which conducts the denoising process on voxelized point clouds. Technically, it introduces 3D positional and patch embeddings, as well as 3D window attention. The main experiments are done on ShapeNet. In addition, the authors empirically show that the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet.", "questions": "Since the evaluation metric is based on Chamfer Distance or Earth Mover’s Distance, I assume that the authors sample a fixed number of points from the GT mesh (and predicted mesh if the baseline, e.g., GET3D, is a mesh-based), which is 2048. I am not sure whether the metric might favor the proposed method especially when the number of points is small, as details can be missed under such a condition. The authors can try to use a larger number of points to compare baselines with the proposed method.", "limitations": "The limitation is not adequately addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow.\n- The authors extend the 2D window attention operator to 3D.\n- The authors empirically show the benefit of leveraging a  2D transformer pre-trained on natural images for 3D generation.", "weaknesses": "1. It seems that an important baseline or reference is missing: \"Point-E: A System for Generating 3D Point Clouds from Complex Prompts\". In L172-L173, the authors claim that \"We tried to train the diffusion transformer on point coordinates, but it did not work since point clouds are sparsely distributed in the 3D embedding space\". However, given Point-E, it sounds not that convincing. Can the authors explain why Point-E is not mentioned or compared in the paper?\n2. It is hard to tell whether the generated shape is of high fidelity if the number of points is only 2048 (L245). It is more convincing if the number of points is larger than 4096 (Point-E) or 16384 (usually used in high-fidelity point completion). In addition, it will be visually better if the authors can present the 3D shapes in the format of mesh (like Point-E, MeshDiffusion) or colored point clouds. Currently, it is hard to tell whether the point cloud is of high fidelity. Visually, GET3D and Point-E look better than this work.\n3. Only ShapeNet is used. Currently, there are more and more 3D datasets. It will be better if the authors can show results on more categories of ShapeNet, ABO, or (a subset of) Objaverse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zPmE,\n\nThank you for appreciating our approach. We address your comments below.\n\n> Why Point-E is not mentioned or compared in the paper.\n\nThanks for pointing this out. The reason why Point-E [A] is not mentioned in the initial manuscript is that our main focus is exploring a plain diffusion transformer in point clouds generation, instead of text-to-point cloud generation in Point-E. Compared to Point-E, our proposed DiT-3D has conceptual differences. The transformer in Point-E is conditioned on CLIP features from a synthetic rendered view generated by a fine-tuned GLIDE [B] model from a text prompt, while we are simply conditioned on a class. They do not use positional embeddings for the input. However, we need to apply 3D positional embeddings for the voxelized point clouds to maintain the voxel structure locality. We will add this discussion to the revision.\n\n> Evaluation on number of points larger than 2048.\n\nThanks for the suggestion. We initially followed the commonly-used generation setting in PVD and LION to sample 2,048 points on each point cloud in the ShapeNet benchmark for evaluation. Meanwhile, the ShapeNet benchmark does not no colorful point clouds. \n\nTo further demonstrate the effectiveness of our proposed DiT-3D on high-fidelity point cloud generation, we resampled 4,096 points and compared our method with PVD and LION in the Table below. Our DiT-3D achieves the best results in terms of all metrics.\n\n|      Method      | 1-NNA CD ($\\downarrow$) | COV CD   ($\\uparrow$) |\n| ---- | :----: | :----: |\n|        PVD       |     62.76    |    37.25   |\n|       LION       |     57.86    |    52.18   |\n| DiT-3D (ours) |     **51.19**    |    **57.39**   |\n\n\n> More categories of ShapeNet.\n\nTo validate the generalizability to more categories, we train our DiT-3D on the large-scale ShapeNet-55 dataset with 55 diverse classes covering vehicles, furniture, and daily necessities. We compare the newly trained model with the state-of-the-art point cloud generation model, LION, on Mug and Bottle generation in the Table below. Our method achieves the best results in terms of all metrics.\n\n|      Method      | 1-NNA-CD (↓) | COV-CD (↑) | 1-NNA-CD (↓) | COV-CD (↑) |\n| ---- | :----: | :----: | :----: | :----: |\n|       LION       |     70.45    |    31.82   |     61.63    |    39.53   |\n| DiT-3D (ours) |     **57.39**    |    **45.26**   |    **53.26**    |    **51.28**   |\n\n\n> Comparisons on a larger number of points. \n\nPlease see the second response to the reviewer.\n\n> Limitations.\n\nAlthough our plain diffusion transformer on point clouds achieves improved performance in generating high-fidelity and diverse 3D shapes. However, we have not explored the potential of other 3D modalities or text-to-3D generation. We plan to leave this for future work. \n\n**References**\n\n[A] Nichol, et al. \"Point-E: A System for Generating 3D Point Clouds from Complex Prompts\", arXiv preprint arXiv:2212.08751 (2022).\n\n[B] Nichol, et al. \"GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models\", arXiv preprint arXiv:2112.10741 (2022).\n\n"}, {"review_id": "xNnF9NstAy", "reviewer": "Reviewer_QDvm", "summary": "This paper proposes to adapt Diffusion Transformer [1] to class-conditional 3D point cloud generation task. In order to achieve it, the authors propose (1) to apply diffusion/infusion directly to point clouds rather than work in latent space; (2) transform input point clouds to voxel grids to apply transformers to tokens extracted from 3D grids in a straightforward way; (3) to reduce the complexity voxel features are processed in patches to produce patch tokens and self-attention in the transformer is modified to aggregate tokens in a window of a predefined size; (4) final voxel features are devoxelized back into points for per-point noise prediction in the infusion process.\n\nExperimental results show promising results in class-specific point cloud generation on ShapeNet dataset, various ablation studies demonstrating the importance of different components and some additional transferability studies that use selective fine-tuning to adapt models pretrained on a data modality/domain to another one.", "questions": "1. In principle, point to voxel and backwards transitions do not necessarily preserve all the points, since if the resolution is low, several point will collapse into a single voxel and will not be recovered during devoxelization. How is this avoided in this work?\n\n2. Since evaluations in this work only include single-class data setups, the expressivity of the proposed model is left underexamined. LION included some experiment showing that their approach is capable of generating realistic point clouds when pretrained on multi-class data. It would be interesting to compare how your class-conditioned model will perform in that setting.\n\n3. What is a «plain transformer»? How this «plain» property is characterized? I think, it is better to drop this adjective or change it and be more specific.\n\nMinor comments:\n\n* The optimal best possible value of 1-NNA metric is 50%, since it means that the nearest neighbour classifier is incapable of distinguishing true from gerenated samples. The text and tables should be modified accordingly.\n\n* L14: «high computation» -> high computational costs\n\n* L119, L340: «operate» -> perform or implement\n\n* L234: sentence is broken", "limitations": "The authors do not provide any statements about such limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper lies in experimental results beating state of the art with an novel approach, not based on prior 3D point cloud generation works. Although it is an adaptation of Diffusion Transformers working with images to 3D point clouds, such a transition from one data type to another is not trivial, so it is remarkable that the authors made it work in this setting.", "weaknesses": "In my opinion the main weakness of the paper is the quality of the presentation. Text can still be polished to improve readability and correct some mistakes. The authors claim across the paper (e.g. L120) that they apply diffusion for the voxelized point clouds, and I think it is misleading. In fact the diffusion is applied to regular point clouds, it is the infusion network that is designed to operate on voxelized 3D features, but as far as I understand, the denoising is applied on a per-point basis. Clarity of explanation can also be improved, since some details are missing (see questions).\n\nThe authors show a lot of ablation studies, but comparisons to external approaches are limited by the main single-class generation experiment. Qualitative comparisons are moved to supplementary materials, while it is better to show it in the main paper (quality of these comparisons could be improved by decreasing point sizes and sampling more points per shape, so some finer details could be examined).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QDvm,\n\n\nThank you for appreciating our approach. We address your comments below.\n\n> Clarification on the diffusion for the voxelized point clouds.\n\nThanks for pointing this out. We will correct the claim by making it more clear that our diffusion is applied to regular point clouds, then we designed the infusion network to extract point-voxel features and applied the devoxelization layer in the end to denoise the original point clouds. \n\n> Comparison with LION on more categories generation.\n\nThanks for the suggestion. Beyond the single-class comparison, we trained our DiT-3D on the large-scale ShapeNet-55 dataset with 55 diverse classes covering vehicles, furniture, and daily necessities. We compare the newly trained model with the state-of-the-art point cloud generation model, LION, on Mug and Bottle generation in the Table below. Our method achieves the best results in terms of all metrics. We will also decrease the point size and move the qualitative comparisons to the main paper. \n\n|      Method      | 1-NNA-CD (↓) | COV-CD (↑) | 1-NNA-CD (↓) | COV-CD (↑) |\n| ---- | :----: | :----: | :----: | :----: |\n|       LION       |     70.45    |    31.82   |     61.63    |    39.53   |\n| DiT-3D (ours) |     **57.39**    |    **45.26**   |    **53.26**    |    **51.28**   |\n\n\n> How to avoid points collapse into a single voxel?\n\nThis is a good question! In this work, we applied trilinear interpolation to transform the voxel into points to guarantee that the features mapped to each point are distinct. If we assigned the feature of a grid to all points that fall into the grid using the nearest-neighbor interpolation, this will make the points in the same voxel grid always share the same features.\n\n> Evaluation on multi-class settings.\n\nPlease see the second response to the reviewer.\n\n\n> Plain transformer.\n\nThe plain transformer refers to a transformer-based architecture that does not use any U-Net architecture for diffusion. In this work, we explore the plain diffusion transformer on voxelized point clouds, instead of using U-Net architecture for denoising in both PVD and LION. We will add this clarification to the main paper. \n\n\n> Minor comments.\n\nThanks for spotting these. We will fix them accordingly in the revision.\n\n> No limitations.\n\nWe provided the discussion on limitations in L48-52 in the supplementary. Although our plain diffusion transformer on point clouds achieves improved performance in generating high-fidelity and diverse 3D shapes. However, we have not explored the potential of other 3D modalities or text-to-3D generation. We plan to leave this for future work. \n\n"}, {"review_id": "038RGfop8m", "reviewer": "Reviewer_trpZ", "summary": "This paper tackles the task of 3D generation. Inspired by the recent progress of utilizing transformers in 2D image generation with diffusion processes (DiT [1]), this paper proposes to replace the common U-Net in 3D diffusion models with a plain transformer. To adapt the DiT to the 3D scenario, authors make several modifications to the vanilla 2D DiT. Experiments on ShapeNet demonstrate improvements over baselines.", "questions": "1. Authors state\n> 3D shape generation is a challenging and important problem that seeks to synthesize high-fidelity point clouds ... (L27)\n\nI do not think this is a proper statement as there are many 3D representations besides point clouds, e.g., mesh [15] or implicit ones [A, B].\n\n2. L190 \"Q, K, V have the same dimensions\" where the notation \"V\" is the same as the voxel resolution in L176. Please modify it to make readers be able to distinguish the two.\n\n3. L226 \"It’s worth noting that we initialize $\\gamma$ to 1, which is then multiplied with the frozen layers\": $\\gamma$ is not explained.\n\n4. L266 \"trained on point clouds (l-GAN) and latent variables (l-GAN)\": duplicated 1-GAN.\n\n[A] 3D Neural Field Generation using Triplane Diffusion. CVPR 2023\n\n[B] SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation. CVPR 2023.", "limitations": "No limitations are provided by the authors.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper tackles an important task of 3D generations that is important for many downstream tasks.\nThe paper is generally well-written and easy to follow.\nExperiments are thorough and convincing.", "weaknesses": "My main concerns are about the incremental development of the proposed DiT-3D based on DiT [1]. Specifically,\n\na. **Diffusion on voxelized point cloud** has been studied in [12];\n\nb. **3D positional embeddings** is a natural/must-have modification from DiT's 2D positional encodings;\n\nc. **3D window attentions** may not be necessary. Actually, I am quite confused why we need the following procedure: point cloud -> voxel (Sec. 3.2 Voxelized point clouds) -> patches in voxels (Sec. 3.2 patch embeddings) -> reshape patches in voxels into 3D window (Sec. 3.2 3D window). I think these hierarchical steps essentially just change the **actual voxel size**. Then why don't we just have a voxel with a resolution of $(p \\cdot R)^3$ at the very beginning? Here $p$ is the patch size the author used in \"3D Positional and Patch Embeddings\"(Sec. 3.2) and $R$ is the number of patches for \"3D window\" (Sec. 3.2). And this does not prevent authors from applying 3D convolution to exchange information (L179).\n\nIf the above abstraction/simplification is correct, it seems like the major modifications to 3D are just changing from image patches to voxels.\n\nd. Authors state\n> These results indicate that our DiT-3D can support flexible transferability on modality and domain, which is different from previous 3D generation methods [12, 13] based on U-Net as the backbone of DDPMs (L325-327). \n\nCan the authors explain whether there are some experiments to support this statement? As in Tab. 3, we only have results from DiT-3D without any baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer trpZ,\n\nThank you for the detailed review. We will address your concerns below.\n\n> Diffusion on voxelized point cloud has been studied in PVD.\n\nWe agree that diffusion on voxelized point cloud has been used in PVD, but we are the first to use a plain diffusion on voxelized point clouds. Meanwhile, directly applying the transformer to denoising the point-voxel features is not working. Therefore, we designed to use the devoxelization layer in the end to denoise the original point clouds.\n\n> 3D positional embeddings.\n\nAlthough 3D positional embeddings is a nature modification from 2D positional embeddings, we are the first to leverage 3D positional embeddings in a plain diffusion transformer for 3D point clouds generation. Furthermore, in Table 2, we validated the necessity of 3D positional embeddings in our diffusion transformer to generate meaningful 3D point clouds.  \n\n> Clarification on 3D window attentions.\n\nThe 3D window attention designed in the diffusion transformer blocks is to propagate patch-level point-voxel features in efficient memory usage, and we do not change the actual voxel size.\nIf we use the voxel with a resolution of (p·R)^3 at the very beginning, we will lose much semantics from the original input, and performs worse on denoising the original point clouds. \nMeanwhile, we do not want to involve 3D convolution in our transformer as our target is to propose a plain diffusion transformer without convolution layers for 3D point clouds generation.\n\nTo verify the advantage of 3D window attention over 3D convolution, we replaced 3D window attention in our DiT-3D with 3D convolution on the voxel size of (16)^3, and compared the generation results on Airplane category in the Table below. Our method with 3D window attention achieves the best results in terms of both metrics on quality and diversity. \n\n| Method              | 1-NNA-CD (↓) | COV-CD (↑) |\n| ---- | :----: | :----: | \n| 3D convolution      | 71.57        | 42.23      |\n| 3D window attention | **62.35**        | **53.16**      |\n\n\n> Comparison regarding flexible transferability on modality and domain\n\nPrevious methods mainly used diverse and different U-Net architectures as diffusion model to achieve denosing process, and they do not support parameter-efficient fine-tuning from 2D modalities. However, our DiT-3D used a plain transformer blocks architecture for diffusion, and we can efficiently transfer the weights from DiT-2D pre-trained on ImageNet to our DiT-3D transformer blocks.\n\nIn Table 3, we demonstrate the effectiveness of our DiT-3D  in supporting modality transferability of the proposed approach from 2D ImageNet pre-trained weights to 3D generation with parameter-efficient fine-tuning. In addition, by only training 0.09MB parameters of models from the source class to the target class, our DiT-3D can achieve a comparable performance of quality and diversity in terms of all metrics.\n\n\n> Clarification on the proper statement about 3D shape generation.\n\nThanks for pointing this out. We will correct it by making it more clear that 3D point clouds generation is a challenging and important problem that seeks to synthesize high-fidelity point clouds using generative models. We will also add the provided citations for discussion about generation based on other 3D representations.\n\n> notation \"V\" as the voxel resolution.\n\nWe have replaced the notation of voxel resolution $V$ with $v$.\n\n> $\\gamma$ is not explained.\n\n$\\gamma$ refers to learnable scale factors γ in transformer blocks of the diffusion model. We will add this clarification to the revision. \n\n> duplicated 1-GAN.\n\nWe have fixed it. The first one is r-GAN.\n\n> No limitations\n\nWe provided the discussion on limitations in L48-52 in the supplementary. Although our plain diffusion transformer on point clouds achieves improved performance in generating high-fidelity and diverse 3D shapes. However, we have not explored the potential of other 3D modalities or text-to-3D generation. We plan to leave this for future work. \n\n"}, {"review_id": "RD25VDc156", "reviewer": "Reviewer_7dkY", "summary": "This paper proposes DiT-3D, an extension of DiT for point cloud data to achieve point cloud generation. With the design of patch embedding and 3D window attention, the proposed model is able to reduce the computation cost. Also, the proposed model is able to directly leverage pre-trained 2D DiT model by fixing most layers and only finetuning several layers, which can reduce the required training time. The results quantitatively verify the effectiveness of the proposed method.", "questions": "Please refer to the weakness part", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method out-performs existing SOTAs on the unconditional point cloud generation task in different evaluation matrices.\n\n2. The paper is easy to follow and understand. \n", "weaknesses": "1. The proposed DiT-3D looks like the original DiT with several modifications (simply changing existing 2D techniques into 3D version). Specifically, it seems like an ECCV 2022 paper [1] has already addressed a similar design of window attention, this paper is just using such a technique on a different task. Therefore, I think the novelty is limited, and more insights should be explained.\n\n2. Previous point cloud generation works such as PVD and LION conducted experiments on multiple conditional generation tasks to verify their design. However, this paper only shows the results of unconditional generation. I believe more experiments such as point cloud completion or 2D image-to-3D point cloud should be included.\n\n3. The qualitative results shown in the main paper only contain examples generated from DiT-3D and the supplementary material only additionally contains results from PVD, DPM, and SetVAE. As two methods that are quantitatively most comparable to DiT-3D, the visualization of LION and MeshDiffusion are not shown in this paper. So it is not clear how DiT-3D outperforms these methods. \nA minor issue: the computation resources should be detailed in this paper. \n\n[1] SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds (ECCV 2022)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7dkY,\n\nThank you for the detailed review. We will address your concerns below.\n\n> Differences from SWFormer (ECCV’2022) for 3D Object Detection.\n\nAlthough SWFormer used a shifted sparse window operator in each transformer block, our DiT-3D has conceptual differences from their design and implementation. For each SWFormer block, they first repeated multi-head self-attention (MSA) on all valid voxel within the same window by repeating N times, then they performed a shifted sparse window partition to re-generate the sparse windows, and processed the shifted windows with another M self-attention layers. However, our DiT-3D simply reduced the complexity of the self-attention operator in Equation (2) from $O(L^2)$ to $O(L^2/R^3)$ using a single attention layer for each block. \n\nFurthermore, as we discussed in Sec 3.4 in the main paper, we introduced multiple different and efficient designs for 3D shape generation compared with DiT on 2D image generation. We are the first to propose efficient 3D window attention in the transformer blocks for reducing the complexity of the self-attention operator in DiT. We proposed to add a devoxelized operator to the final output of the last linear layer from DiT for denoising the noise prediction in the point cloud space.\n\nOur contribution is not just proposing a new transformer for 3D point clouds generation, but aslo investigating properties of plain diffusion transformer on 3D point clouds generation. We demonstrate that the representations learned on ImageNet have a very positive impact on 3D generation, despite the significant domain gap between 2D images and 3D point clouds. Meanwhile, given a pre-trained DiT-3D model on source classes, we can use the parameter-efficient fine-tuning approach to extend its applicability to new categories.\n\n\n> Experiments on point clouds completion.\n\nThanks for the suggestion! To verify the effectiveness of our DiT-3D on conditional generation, we conducted point cloud completion experiments on ShapeNet and compared with PVD and LION in Table below. Here, we applied Chamfer Distance and Earth Mover’s Distance to evaluate the reconstruction result. As can be seen, our method achieves the best performance in terms of all categories (Airplane, Chair, Car).\n\n\n| Method | Airplane-CD (↓) | Airplane-EMD (↓) | Chair-CD (↓) | Chair-EMD (↓) | Car-CD (↓) | Car-EMD (↓) |\n|--------|:------:|:-------:|:------:|:-------:|:------:|:-------:|\n| PVD    | 0.4415 |  1.030  | 3.211  |  2.939  | 1.774  |  2.146  |\n| LION   | 0.4035 |  0.9732 |  2.725 |  2.863  |  1.405 |  1.982  |\n| DiT-3D (ours) | **0.3521** |  **0.9235** | **2.216** |  **2.385**  |  **1.126** |  **1.513**  |\n\n\n> Clarification on qualitative and quantitative comparisons. \n\nWe provided multiple qualitative comparisons with previous baselines (especially PVD, a U-Net based diffusion model on voxelized point clouds) in Figure 2 in the supplementary, and our DiT-3D generated high-fidelity and diverse point clouds of 3D shapes for each category. \n\nFor quantitative comparison with LION and MeshDiffusion, our DiT-3D outperforms them significantly on all metrics in terms of high-fidelity and diversity, as compared in Table 1 in the main paper. Due to rebuttal time constraints, we will add more visualizations of LION and MeshDiffusion to the supplementary. \n\n> Computation resources.\n\nWe used 8 NVIDIA V100-32GB GPUs for the experiments. \n\n"}]

Author rebuttal:
Dear all reviewers,

We thank each of you for generously dedicating your valuable time and expertise to reviewing our work. We acknowledge and sincerely appreciate the insightful comments and critiques provided by all the reviewers. In response to your invaluable feedback, we have made significant revisions to our manuscript, aiming to address each of your concerns comprehensively and scholarly. Reviewer trpZ and Reviewer 7dkY, we kindly request your reconsideration of your decision, given that we have taken utmost care to address the main comments raised in your reviews thoroughly. 

Once again, we express our sincere appreciation for your valuable contributions to the review process. Your expertise and guidance have been invaluable in improving the quality of our work. We remain committed to continuous discussion and eagerly await your final decision.


Meta-review: The paper received mixed ratings, and there was a rebuttal: Five knowledgeable reviewers recommended: Borderline Accept, Weak Accept, Borderline Accept, Borderline Accept, and Borderline Reject. On balance, it is our recommendation to accept the paper. Authors should attend to the main points in the reviews. when preparing a final version. No basis to overturn the reviews. The method proposed in this paper is novel and the results are indeed good. This paper is valuable and should be shared within the community to advance research on 3D Shape Generation.
Decision: Accept (poster)