Title: DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Models

Transformers

3D Shape Generation
Abstract: Recent Diffusion Transformers (i.e., DiT) have demonstrated their powerful effectiveness in generating high-quality 2D images. However, it is unclear how the Transformer architecture performs equally well in 3D shape generation, as previous 3D diffusion methods mostly adopted the U-Net architecture. 
To bridge this gap, we propose a novel Diffusion Transformer for 3D shape generation, named DiT-3D, which can directly operate the denoising process on voxelized point clouds using plain Transformers. Compared to existing U-Net approaches, our DiT-3D is more scalable in model size and produces much higher quality generations.
Specifically, the DiT-3D adopts the design philosophy of DiT but modifies it by incorporating 3D positional and patch embeddings to aggregate input from voxelized point clouds.
To reduce the computational cost of self-attention in 3D shape generation, we incorporate 3D window attention into Transformer blocks, as the increased 3D token length resulting from the additional dimension of voxels can lead to high computation.
Finally, linear and devoxelization layers are used to predict the denoised point clouds. 
In addition, we empirically observe that the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet.
Experimental results on the ShapeNet dataset demonstrate that the proposed DiT-3D achieves state-of-the-art performance in high-fidelity and diverse 3D point cloud generation.

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

Abstract

Recent Diffusion Transformers ( e.g. DiT [1]) have demonstrated their powerful effectiveness in generating high-quality 2D images. However, it is still being determined whether the Transformer architecture performs equally well in 3D shape generation, as previous 3D diffusion methods mostly adopted the U-Net architecture. To bridge this gap, we propose a novel Diffusion Transformer for 3D shape generation, namely DiT-3D, which can directly operate the denoising process on voxelized point clouds using plain Transformers. Compared to existing U-Net approaches, our DiT-3D is more scalable in model size and produces much higher quality generations. Specifically, the DiT-3D adopts the design philosophy of DiT [1] but modifies it by incorporating 3D positional and patch embeddings to adaptively aggregate input from voxelized point clouds. To reduce the computational cost of self-attention in 3D shape generation, we incorporate 3D window attention into Transformer blocks, as the increased 3D token length resulting from the additional dimension of voxels can lead to high computation. Finally, linear and devoxelization layers are used to predict the denoised point clouds. In addition, our transformer architecture supports efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on the ShapeNet dataset demonstrate that the proposed DiT-3D achieves state-of-the-art performance in high-fidelity and diverse 3D point cloud generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy of the state-of-the-art method by 4.59 and increases the Coverage metric by 3.51 when evaluated on Chamfer Distance.

1 Introduction

In recent times, there has been a growing interest in exploring the potential of diffusion transformers for high-fidelity image generation, as evinced by a series of scholarly works [1, 2, 3, 4]. Notably, a seminal work by Peebles et al. [1] proposed the replacement of the widely-used U-Net backbone with a scalable transformer. Specifically, the proposed method operates on latent patches by training latent 2D diffusion models. However, the efficacy of plain diffusion transformers for 3D shape generation has yet to be explored, as most existing 3D diffusion approaches continue to adopt the U-Net backbone.

Generating high-fidelity point clouds for 3D shape generation is a challenging and significant problem. Early generative methods [5, 6, 7] addressed this problem by directly optimizing heuristic loss objectives, such as Chamfer Distance (CD) and Earth Mover's Distance (EMD). More recent works [8, 9, 10, 11] have explored the usage of the generative adversarial network (GAN)-based

品

and flow-based models to generate 3D point clouds from a probabilistic perspective. Recently, researchers [12, 13, 14, 15] have turned to various denoising diffusion probabilistic models (DDPMs) to generate entire shapes from random noise. For instance, PVD [12] employed the point-voxel representation of 3D shapes as input to DDPMs. They reversed the diffusion process from observed point clouds to Gaussian noise by optimizing a variational lower bound to the likelihood function. Recently, the Diffusion Transformer (DiT) [1, 2] has been shown to surpass the U-Net architecture in 2D image generation, owing to its simple design and superior generative performance. Consequently, we investigate the potential of the Diffusion Transformer for 3D generation. However, extending the 2D DiT to 3D poses two significant challenges: (1) Point clouds are intrinsically unordered, unlike images where pixels are ordered; and (2) The tokens in 3D space have an additional dimension compared to 2D images, resulting in a substantial increase in computational cost.

This work introduces DiT-3D, a novel diffusion transformer architecture designed for 3D shape generation that leverages the denoising process of DDPM on 3D point clouds. The proposed model inherits the simple design of the modules in DiT-2D, with only minor adaptations to enable it to generalize to 3D generation tasks. To tackle the challenge posed by the unordered data structure of point clouds, we convert the point cloud into a voxel representation. DiT-3D employs 3D positional embedding and 3D patch embedding on the voxelized point clouds to extract point-voxel features and effectively process the unordered data. Furthermore, to address the computational cost associated with a large number of tokens in 3D space, we introduce a 3D window attention operator instead of the vanilla global attention in DiT-2D. This operator significantly reduces training time and memory usage, making DiT-3D feasible for large-scale 3D generation tasks. Finally, we utilize linear and devoxelization layers to predict the noised point clouds in the reversed process to generate final 3D shapes.

In order to address the computational cost associated with a large number of tokens in 3D space, we also introduce a parameter-efficient tuning method to utilize the pre-trained DiT-2D model on ImageNet as initialization for DiT-3D (window attention shares the same parameters with vanilla attention). Benefiting from the substantial similarity between the network structure and parameters of DiT-3D and DiT-2D, the representations learned on ImageNet significantly improve 3D generation, despite the significant domain disparity between 2D images and 3D point clouds. To our knowledge, we are the first to achieve parameter-efficient fine-tuning from 2D ImageNet pre-trained weights for high-fidelity and diverse 3D shape generation. In particular, we highly decrease the training parameters from 32.8MB to only 0.09MB.

We present a comprehensive evaluation of DiT-3D on a diverse set of object classes in the ShapeNet benchmark, where it achieves state-of-the-art performance compared to previous non-DDPM and DDPM-based 3D shape generation methods. Qualitative visualizations further emphasize the efficacy of DiT-3D in generating high-fidelity 3D shapes. Extensive ablation studies confirm the significance of 3D positional embeddings, window attention, and 2D pre-training in 3D shape generation. Moreover, we demonstrate that DiT-3D is easily scalable regarding patch sizes, voxel sizes, and model sizes. Our

findings align with those of DiT-2D, where increasing the model size leads to continuous performance improvements. In addition, our parameter-efficient fine-tuning from DiT-2D ImageNet pre-trained weights highly decreases the training parameters while achieving competitive generation performance. By only training 0.09MB parameters of models from the source class to the target class, we also achieve comparable results of quality and diversity in terms of all metrics.

Our main contributions can be summarized as follows:

We present DiT-3D, the first plain diffusion transformer architecture for point cloud shape generation that can effectively perform denoising operations on voxelized point clouds.

We make several simple yet effective modifications on DiT-3D, including 3D positional and patch embeddings, 3D window attention, and 2D pre-training on ImageNet. These modifications significantly improve the performance of DiT-3D while maintaining efficiency.

Extensive experiments on the ShapeNet dataset demonstrate the state-of-the-art superiority of DiT-3D over previous non-DDPM and DDPM baselines in generating high-fidelity shapes.

2 Related Work

3D Shape Generation. 3D shape generation aims to synthesize high-fidelity point clouds or meshes using generative models, such as variational autoencoders [16, 17, 18], generative adversarial networks [19, 8, 20], and normalized flows [9, 10, 11]. Typically, PointFlow [9] utilized a probabilistic framework based on the continuous normalizing flow to generate 3D point clouds from two-level hierarchical distributions. ShapeGF [21] trained a score-matching energy-based network to learn the distribution of points across gradient fields using Langevin dynamics. More recently, GET3D [14] leveraged a signed distance field (SDF) and a texture field as two latent codes to learn a generative model that directly generates 3D meshes. In this work, we mainly focus on denoising diffusion probabilistic models for generating high-fidelity 3D point clouds starting from random noise, where point and shape distributions are not separated.

Diffusion Models. Diffusion models [22, 23, 24] have been demonstrated to be effective in many generative tasks, such as image generation [25], image restoration [26], speech generation [27], and video generation [28]. Denoising diffusion probabilistic models (DDPMs) [22, 23] utilized a forward noising process that gradually adds Gaussian noise to images and trained a reverse process that inverts the forward process. In recent years, researchers [29, 12, 13, 30, 15, 31, 32] have tried to explore diverse pipelines based on diffusion probabilistic models to achieve 3D shape generation. For example, PVD [12] applied DDPM based on PVCNNs [33] on the point-voxel representation of 3D shapes with structured locality into point clouds. To improve the generation quality, LION [13] used two DDPMs to learn a hierarchical latent space based on a global shape latent representation and a point-structured latent space separately. Different from them, we will solve the 3D shape generation problem in our approach by designing a plain transformer-based architecture backbone to replace the U-Net backbone for reversing the diffusion process from observed point clouds to Gaussian noise. Meanwhile, our 3D plain diffusion transformer supports multi-class training with learnable class embeddings as the condition and parameter-efficient fine-tuning with modality and domain transferability differ from DDPM-based 3D generation approaches discussed above.

Transformers in Diffusion Generation. Diffusion Transformers [1, 2, 3, 4] have recently shown their impressive capacity to generate high-fidelity images. For instance, Diffusion Transformer (DiT) [1] proposed a plain diffusion Transformer architecture to learn the denoising diffusion process on latent patches from a pre-trained pre-trained variational autoencoder model in Stable Diffusion [34]. U-ViT [2] incorporated all the time, condition, and noisy image patches as tokens and utilized a Vision transformer(ViT) [35]-based architecture with long skip connections between shallow and deep layers. More recently, UniDiffuser [3] designed a unified transformer for diffusion models to handle input types of different modalities by learning all distributions simultaneously. While those diffusion transformer approaches achieve promising performance in 2D image generation, how a plain diffusion transformer performs on 3D shape generation is still being determined. In contrast, we develop a novel plain diffusion transformer for 3D shape generation that can effectively perform denoising operations on voxelized point clouds. Furthermore, the proposed DiT-3D can support parameter-efficient fine-tuning with transferability across modality and domain.

3D Voxel Diffusion Transformer

2D Latent Diffusion Transformer

+2D PE

Latent

Noise

32x32x4

DiT-2D Block

N x

Layer Norm

Layer Norm

Scale, Shift

Global

Attention

Scale

Scale, Shift

FFN

Scale

𝛾1 , 𝛽

1

𝛾2 , 𝛽

2

𝛼1

𝛼2

Layer Norm

Condition

MLP

Label

Embed

Timestep

+3D PE

DiT-3D Block

N x

Layer Norm

Linear and Devoxelize

Voxel

Noise

32x32x32x4

Embed

Label

Timestep

Layer Norm

Scale, Shift

Scale

Scale, Shift

FFN

Scale

𝛾1 , 𝛽

1

𝛾2 , 𝛽

2

𝛼1

𝛼2

Layer Norm

Condition

MLP

3D Window

Attention

3 Method

Given a set of 3D point clouds, we aim to learn a plain diffusion transformer for synthesizing new high-fidelity point clouds. We propose a novel diffusion transformer that operates the denoising process of DDPM on voxelized point clouds, namely DiT-3D, which consists of two main modules: Design DiT for 3D Point Cloud Generation in Section 3.2 and Efficient Modality/Domain Transfer with Parameter-efficient Fine-tuning in Section 3.3.

3.1 Preliminaries

In this section, we first describe the problem setup and notations and then revisit denoising diffusion probabilistic models (DDPMs) for 3D shape generation and diffusion transformers on 2D images.

Problem Setup and Notations. Given a set S = { p i } S i =1 of 3D shapes with M classes, our goal is to train a plain diffusion transformer from these point clouds for generating high-fidelity point clouds. For each point cloud p i , we have N points for x, y, z coordinates, that is p i ∈ R N × 3 . Note that we have a class label for the 3D shape p i , which is denoted as { y i } i M =1 with y i for the ground-truth category entry i as 1. During the training, we take the class label as input to achieve classifier-free guidance in conditional diffusion models, following the prior diffusion transformer ( i.e. , DiT [1]) on images.

Revisit DDPMs on 3D Shape Generation. To solve the 3D shape generation problem, previous work [12] based on denoising diffusion probabilistic models (DDPMs) define a forward noising process that gradually applies noise to real data x 0 as q ( x t | x t -1 ) = N ( x t ; √ 1 -β t x t -1 , β t I ) , where β t is a Gaussian noise value between 0 and 1 . In particular, the denoising process produces a series of shape variables with decreasing levels of noise, denoted as x T , x T -1 , ..., x 0 , where x T is sampled from a Gaussian prior and x 0 is the final output. With the reparameterization trick, we can have x t = √ ¯ α t x 0 + √ 1 -¯ α t ϵ , where ϵ ∼ N ( 0 , I ) , α t = 1 -β t , and ¯ α t = ∏ t i =1 α i .

For the reverse process, diffusion models are trained to learn a denoising network θ for inverting forward process corruption as p θ ( x t -1 | x t ) = N ( x t -1 ; µ θ ( x t , t ) , σ 2 t I ) . The training objective is to maximize a variational lower bound of the negative log data likelihood that involves all of x 0 , ..., x T

where D KL ( ·||· ) denotes the KL divergence measuring the distance between two distributions. Since both p θ ( x t -1 | x t )) and q ( x t -1 | x t , x 0 ) are Gaussians, we can reparameterize µ θ ( x t , t ) to predict the noise ϵ θ ( x t , t ) . In the end, the training objective can be reduced to a simple mean-squared loss between the model output ϵ θ ( x t , t ) and the ground truth Gaussian noise ϵ as: L simple = ∥ ϵ -ϵ θ ( x t , t ) ∥ 2 . After p θ ( x t -1 | x t )) is trained, new point clouds can be generated by progressively sampling x t -1 ∼ p θ ( x t -1 | x t )) by using the reparameterization trick with initialization of x T ∼ N ( 0 , I ) .

Revisit Diffusion Transformer (DiT) on 2D Image Generation. To generate high-fidelity 2D images, DiT proposed to train latent diffusion models (LDMs) with Transformers as the backbone, consisting of two training models. They first extract the latent code z from an image sample x using an autoencoder with an encoder f enc ( · ) and a decoder f dec ( · ) , that is, z = f enc ( x ) . The decoder is used to reconstruct the image sample ˆ x from the latent code z , i.e. , ˆ x = f dec ( z ) . Based on latent codes z , a latent diffusion transformer with multiple designed blocks is trained with time embedding t and class embedding c , where a self-attention and a feed-forward module are involved in each block. Note that they apply patchification on latent code z to extract a sequence of patch embeddings and depatchification operators are used to predict the denoised latent code z .

Although DDPMs achieved promising performance on 3D shape generation, they can only handle single-class training based on PVCNNs [33] as the encoder to extract 3D representations, and they cannot learn explicit class-conditional embeddings. Furthermore, we are not able to directly transfer their single-class pre-trained model to new classes with parameter-efficient fine-tuning. Meanwhile, we empirically observe that the direct extension of DiT [1] on point clouds does not work. To address this problem, we propose a novel plain diffusion transformer for 3D shape generation that can effectively achieve the denoising processes on voxelized point clouds, as illustrated in Figure 2.

3.2 Diffusion Transformer for 3D Point Cloud Generation

To enable denoising operations using a plain diffusion transformer, we propose several adaptations to 3D point cloud generation in Figure 2 within the framework of DiT [1]. Specifically, our DiT-3D model accepts voxelized point clouds as input and employs a patchification operator to generate token-level patch embeddings. We add 3D positional embeddings to these embeddings and extract point-voxel representations from all input tokens using multiple transformer blocks based on 3D window attention. Finally, we apply a devoxelized linear layer to the unpatchified voxel output, allowing us to predict the noise in the point cloud space.

Denoising on Voxelized Point Clouds. Point clouds are inherently unordered, unlike images where pixels follow a specific order. We encountered difficulty in our attempt to train a diffusion transformer on point coordinates due to the sparse distribution of points in the 3D embedding space. To address this issue, we decided to voxelize the point clouds into dense representations, allowing the diffusion transformers to extract point-voxel features. Our approach differs from DiT [1], which utilizes latent codes z to train the latent diffusion transformer. Instead, we directly train the denoising process on voxelized point clouds using the diffusion transformer. For each point cloud p i ∈ R N × 3 with N points for x, y, z coordinates, we first voxelize it as input v i ∈ R V × V × V × 3 .

3D Positional and Patch Embeddings. With the voxel input v i ∈ R V × V × V × 3 , we introduce patchification operator with a patch size p × p × p to generate a sequence of patch tokens t ∈ R L × 3 . L = ( V/p ) 3 denotes the total number of patchified tokens. A 3D convolution layer is applied on patch tokens to extract patch embeddings e ∈ R L × D , where D is the dimension of embeddings. To adapt to our voxelized point clouds, we add frequency-based sine-cosine 3D positional embeddings instead of the 2D version in DiT [1] to all input tokens. Based on these patch-level tokens, we introduce time embeddings t and class embeddings c as input to achieve multi-class training with learnable class embeddings as the condition, which differs from existing 3D generation approaches with U-Net as the backbone.

3D Window Attention. Due to the increased token length resulting from the additional dimension in 3D space, the computational cost of 3D Transformers can be significantly high. To address this issue, we introduce efficient 3D window attention into Transformer blocks blocks to propagate point-voxel

features in efficient memory usage. For the original multi-head self-attention process with each of the heads Q,K,V have the same dimensions L × D , where L = ( V/p ) 3 is the length of input tokens, we can have the attention operator as:

where D h is the dimension size of each head. The computational complexity of this process is O ( L 2 ) , which will be largely expensive for high voxel resolutions. Inspired by [36], we extend the 2D window attention operator to a 3D one for 3D input tokens instead of vanilla global attention. This process uses a window size of R to reduce the length of total input tokens as

where K is the input tokens to be reduced. Reshape ( L R 3 , D · R 3 ) denotes to reshape K to the one with shape of L R 3 × ( D · R 3 ) , and Linear ( C in , C out )( · ) denotes to a linear layer with a C in -dimensional tensor as input and a C out -dimensional tensor as output. Therefore, the new K has the shape of L R 3 × D . As a result, the complexity of the self-attention operator in Equation (2) is reduced from O ( L 2 ) to O ( L 2 R 3 ) . In our experiments, we set R to 4 in the default setting.

Devoxelized Prediction. Since the transformers blocks are implemented on voxelized point clouds, we can not directly use a standard linear decoder to predict the output noise ϵ θ ( x t , t ) from point clouds. In order to generate the output noise, we devoxelize output tokens from the linear decoder. We first apply the final layer norm and linearly decode each token into a p × p × p × L × 3 tensor, where L is the total number of input tokens. Then we unpatchify the decoded token into a voxel tensor with the shape of V × V × V × 3 . Finally, the unpatchified voxel tensor is devoxelized into a N × 3 tensor as the output noise ϵ θ ( x t , t ) , matching with the ground truth Gaussian noise ϵ in the point cloud space.

Model Scaling. Our DiT-3D is designed to be scalable, adapting to varying voxel sizes, patch sizes, and model sizes. Specifically, it can flexibly accommodate voxel dimensions of 16, 32, 64, patch dimensions of 2, 4, 8, and model complexity ranging from Small, Base, Large and Extra Large, as demonstrated in DiT [1]. For instance, a model designated as DiT-3D-S/4 refers that it utilizes the Small configuration of the DiT model [1], with a patch size p of 4.

3.3 Efficient Modality/Domain Transfer with Parameter-efficient Fine-tuning

Leveraging the scalability of the plain diffusion transformer, we investigate parameter-efficient finetuning for achieving modality and domain transferability. To facilitate modality transfer from 2D to 3D, we can leverage the knowledge pre-trained on large-scale 2D images using DiT [1]. For domain transfer from a source class to target classes, we train DiT-3D on a single class ( e.g. chair) and transfer the model's parameters to other classes ( e.g. airplane, car).

Modality Transfer: 2D (ImageNet) → 3D (ShapeNet) . As large-scale pre-trained DiT checkpoints 2 are readily available, we can skip training our diffusion transformer from scratch. Instead, we can load most of the weights from the DiT [1] pre-trained on ImageNet [37] into our DiT-3D and continue with fine-tuning. To further optimize training efficiency, we adopt the parameter-efficient fine-tuning approach described in recent work, DiffFit [4], which involves freezing the majority of parameters and only training the newly-added scale factors, bias term, normalization, and class condition modules. It's worth noting that we initialize γ to 1, which is then multiplied with the frozen layers.

Domain Transfer: Source Class → Target Class. Given a pre-trained DiT-3D model on chair data, we can use the parameter-efficient fine-tuning approach to extend its applicability to new categories. Specifically, following the same methodology as described above, we leverage the fine-tuning strategy of DiffFit and obtain satisfactory generation results.

3.4 Relationship to DiT [1]

Our DiT-3D contains multiple different and efficient designs for 3D shape generation compared with DiT [1] on 2D image generation:

We effectively achieve the diffusion space on voxelized point clouds, while DiT needs the latent codes from a pre-trained variational autoencoder as the denoising target.

Our plain diffusion transformer first incorporates frequency-based sine-cosine 3D positional embeddings with patch embeddings for voxel structure locality.

We are the first to propose efficient 3D window attention in the transformer blocks for reducing the complexity of the self-attention operator in DiT.

We add a devoxelized operator to the final output of the last linear layer from DiT for denoising the noise prediction in the point cloud space.

4 Experiments

4.1 Experimental Setup

Datasets. Following most previous works [12, 13], we use ShapeNet [38] Chair, Airplane, and Car as our primary datasets for 3D shape generation. For each 3D shape, we sample 2,048 points from 5,000 provided points in [38] for training and testing. We also use the same dataset splits and pre-processing in PointFlow [9], which normalizes the data globally across the whole dataset.

Evaluation Metrics. For comprehensive comparisons, we follow prior work [12, 13] and use Chamfer Distance (CD) and Earth Mover's Distance (EMD) as our distance metrics in computing 1-Nearest Neighbor Accuracy (1-NNA) and Coverage (COV) as main metrics to measure generative quality. 1-NNA calculates the leave-one-out accuracy of the 1-NN classifier to quantify point cloud generation performance, which is robust and correlates with generation quality and diversity. A lower 1-NNA score is better. COV measures the number of reference point clouds matched to at least one generated shape, correlating with generation diversity. Note that a higher COV score is better but does not measure the quality of the generated point clouds since low-quality but diverse generated point clouds can achieve high COV scores.

Implementation. Our implementation is based on the PyTorch [39] framework. The input voxel size is 32 × 32 × 32 × 3 , i.e. , V = 32 . The final linear layer is initialized with zeros, and other weights initialization follows standard techniques in ViT [35]. The models were trained for 10,000 epochs using the Adam optimizer [40] with a learning rate of 1 e -4 and a batch size of 128 . We set T = 1000 for experiments. In the default setting, we use S/4 with patch size p = 4 as the backbone. Note that we utilize 3D window attention in partial blocks ( i.e. , 0,3,6,9) and global attention in other blocks.

4.2 Comparison to State-of-the-art Works

In this work, we propose a novel and effective diffusion transformer for 3D shape generation. In order to validate the effectiveness of the proposed DiT-3D, we comprehensively compare it to previous non-DDPM and DDPM baselines. 1) r-GAN, 1-GAN [8]: (2018'ICML): generative models based on GANs trained on point clouds (l-GAN) and latent variables (l-GAN); 2) PointFlow [9] (2019'ICCV): a probabilistic framework to generate 3D point clouds from a two-level hierarchy of distributions with the continuous normalizing flow; 3) SoftFlow [10] (2020'NeurIPS): a probabilistic framework for training normalizing flows on manifolds to estimate the distribution of various shapes; 4) SetV AE [18] (2021'CVPR): a hierarchical variational autoencoder for sets to learn latent variables for coarse-to-fine dependency and permutation invariance; 5) DPF-Net [11] (2020'ECCV): a discrete latent variable network that builds on normalizing flows with affine coupling layers; 6) DPM [29] (2021'ICCV): the first DDPM approach to learn the reverse diffusion process for point clouds as a Markov chain conditioned on shape latent; 7) PVD [12] (2021'ICCV): a strong DDPM baseline based on the point-voxel representation of 3D shapes; 8) LION [13] (2022'NeurIPS): a recent method based on two hierarchical DDPMs in global latent and latent points spaces; 9) GET3D [14] (2022'NeurIPS): a generative model that directly generates explicit textured 3D meshes based on two latent codes (a 3D SDF and a texture field); 10) MeshDiffusion [15] (2023'ICLR): a very recent DDPM method using graph structure of meshes and deformable tetrahedral grid parametrization of 3D mesh shapes.

For chair generation, we report the quantitative comparison results in Table 1. As can be seen, we achieved the best performance in terms of all metrics compared to previous non-DDPM and DDPM baselines. In particular, the proposed DiT-3D significantly outperforms DPF-Net [11], the current state-of-the-art normalizing flows baseline, decreasing by 12.89 1-NNA@CD & 7.80 1-NNA@EMD, and increasing by 7.74 COV@CD & 3.8 COV@EMD. Moreover, we achieve superior performance gains compared to MeshDiffusion [15], the current state-of-the-art DDPM baseline on meshes, which implies the importance of replacing the U-Net with a plain diffusion transformer from observed point clouds for generating high-fidelity 3D shapes. Meanwhile, our DiT-3D outperforms LION [15] by a large margin, where we achieve the performance gains of 4.59 1-NNA@CD & 1.61 1-NNA@EMD, and 3.51 COV@CD & 2.21 COV@EMD. These significant improvements demonstrate the superiority of our method in 3D shape generation. In addition, significant gains in airplane and car generations can be observed in Table 1. These qualitative results also showcase the effectiveness of applying a plain diffusion transformer to operate the denoising process from point clouds for generating high-fidelity and diverse shapes, as shown in Figure 3.

4.3 Experimental Analysis

In this section, we performed ablation studies to demonstrate the benefit of introducing three main 3D design components (voxel diffusion, 3D positional embeddings, and 3D window attention) in 3D shape generation. We also conducted extensive experiments to explore the efficiency of 3D window attention, modality and domain transferability, and scalability.

(a) Modality transfer.

(b) Domain transfer.

Ablation on 3D Design Components. In order to validate the effectiveness of the introduced 3D adaptation components (voxel diffusion, 3D positional embeddings, and 3D window attention), we ablate the necessity of each module and report the quantitative results in Table 2. Note that no voxel diffusion means we directly perform the denoising process on point coordinates without voxelized point clouds and devoxelization prediction. We can observe that adding bearable voxel diffusion to the vanilla baseline highly decreases the results of 1-NNA (by 32.40 @CD and 30.46 @AUC) and increase the performance of COV (by 31.202 @CD and 37.087 @EMD), which demonstrates the benefit of voxelized point clouds and devoxelization prediction in denoising process for 3D shape generation. Meanwhile, introducing 3D positional embedding in the baseline with voxel diffusion also increases the shape generation performance in terms of all metrics. More importantly, incorporating 3D window attention and two previous modules together into the baseline significantly decreases the training cost by 44.86 hours and results of 1-NNA by 50.75 @CD and 49.2 @EMD, and raises the performance of COV by 44.682 @CD and 49.667 @EMD. These improving results validate the importance of the proposed 3D adaptation components in the plain diffusion transformer to operate the denoising process from observed point clouds for 3D shape generation.

Influence of 2D Pretrain (ImageNet). In order to show the modality transferability of the proposed approach from 2D ImageNet pre-trained weights to 3D generation with parameter-efficient fine-tuning, we report the ablation results of ImageNet pre-train and efficient fine-tuning on chair generation in Table 3a. From comparisons, two main observations can be derived: 1) With the initialization with 2D ImageNet pre-trained weights, the proposed DiT-3D improves the quality of shape generation by decreasing 1-NNA by 2.92@CD and 0.18@EMD. 2) Incorporating parameter-efficient fine-tuning into 2D ImageNet pre-trained weights highly decreases the training parameters while achieving competitive generation performance.

Transferability in Domain. In addition, we explore the parameter-efficient fine-tuning for domain transferability in Table 3b. By only training 0.09MB parameters of models from the source class to the target class, we can achieve a comparable performance of quality and diversity in terms of all metrics. These results indicate that our DiT-3D can support flexible transferability on modality and domain, which differs from previous 3D generation methods [12, 13] based on U-Net as the backbone of DDPMs.

Scaling Patch size, Voxel size and Model Size. To explore the scalability of our plain diffusion transformer to flexible designs, we ablate the patch size from { 2 , 4 , 8 } , voxel size from { 16 , 32 , 64 } , and the model size from { S/4, B/4, L/4, XL/4 } . As seen in Table 4a, when the patch size is 2, the proposed DiT-3D achieves the best performance. This trend is also observed in the original DiT [1] work for 2D image generation. In addition, increasing the voxel size from 16 to 64 for the input of the diffusion denoising process raises the performance in terms of all metrics, as shown in Table 4b. More importantly, we can still observe performance gains by scaling up the proposed plain diffusion transformer to XL/4 when the model is trained for 2,000 epochs. These promising results further demonstrate the strong scalability of our DiT-3D to flexible patch size, voxel size, and model sizes for generating high-fidelity 3D shapes.

(a) Patch size.

5 Conclusion

In this work, we present DiT-3D, a novel plain diffusion transformer for 3D shape generation, which can directly operate the denoising process on voxelized point clouds. Compared to existing U-Net approaches, our DiT-3D is more scalable in model size and produces much higher quality generations. Specifically, we incorporate 3D positional and patch embeddings to aggregate input from voxelized point clouds. We then incorporate 3D window attention into Transformer blocks to reduce the computational cost of 3D Transformers, which can be significantly high due to the increased token length resulting from the additional dimension in 3D. Finally, we leverage linear and devoxelization layers to predict the denoised point clouds. Due to the scalability of the Transformer, DiT-3D can easily support parameter-efficient fine-tuning with modality and domain transferability. Empirical results demonstrate the state-of-the-art performance of the proposed DiT-3D in high-fidelity and diverse 3D point cloud generation.

References

William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748 , 2022. 1, 2, 3, 4, 5, 6, 7, 9

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 1, 2, 3

Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555 , 2023. 1, 3

Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. arXiv preprint arXiv:2304.06648 , 2023. 1, 3, 6

Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 605-613, 2017. 1

Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A papier-mâché approach to learning 3d surface generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 216-224, 2018. 1

Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Bongsoo Choy, and Silvio Savarese. Deformnet: Free-form deformation network for 3d shape reconstruction from a single image. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 858-866, 2017. 1

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In Proceedings of the International Conference on Machine Learning (ICML) , 2018. 1, 3, 7, 8

Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4541-4550, 2019. 1, 3, 7, 8

(b) Voxel size.

(c) Model size.

Hyeongju Kim, Hyeonseung Lee, Woohyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow on manifolds. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2020. 1, 3, 7, 8

Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. In Proceedings of the European Conference on Computer Vision (ECCV) , page 694-710, 2020. 1, 3, 7, 8

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5826-5835, 2021. 2, 3, 4, 7, 8, 9

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2022. 2, 3, 7, 8, 9

Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS) , 2022. 2, 3, 7, 8

Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In Proceedings of International Conference on Learning Representations (ICLR) , 2023. 2, 3, 7, 8

Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 206-215, 2018. 3

Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV) , 2018. 3

Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composition for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15059-15068, 2021. 3, 7, 8

Diego Valsesia, Giulia Fracastoro, and Enrico Magli. Learning localized generative models for 3d point clouds via graph convolution. In Proceedings of International Conference on Learning Representations (ICLR) , 2019. 3

Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3859-3868, 2019. 3

Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020. 3

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS) , pages 68406851, 2020. 3

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of International Conference on Learning Representations (ICLR) , 2021. 3

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv , 2021. 3

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022. 3

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636 , 2021. 3

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In Proceedings of International Conference on Learning Representations (ICLR) , 2021. 3

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022. 3

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2837-2845, 2021. 3, 7, 8

Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 , 2022. 3

Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

Ruihang Chu, Enze Xie, Shentong Mo, Zhenguo Li, Matthias Nießner, Chi-Wing Fu, and Jiaya Jia. Diffcomplete: Diffusion-based generative 3d shape completion. arXiv preprint arXiv:2306.16329 , 2023. 3

Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2019. 3, 5

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022. 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations (ICLR) , 2021. 3, 7

Yanghao Li, Hanzi Mao, Ross B. Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 6

Jia Deng, Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A LargeScale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 248-255, 2009. 6

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015. 7

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , pages 8026-8037, 2019. 7

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DiT-3D, an extension of DiT for point cloud data to achieve point cloud generation. With the design of patch embedding and 3D window attention, the proposed model is able to reduce the computation cost. Also, the proposed model is able to directly leverage pre-trained 2D DiT model by fixing most layers and only finetuning several layers, which can reduce the required training time. The results quantitatively verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The proposed method out-performs existing SOTAs on the unconditional point cloud generation task in different evaluation matrices.\n\n2. The paper is easy to follow and understand.\n\nWEAKNESSES:\n1. The proposed DiT-3D looks like the original DiT with several modifications (simply changing existing 2D techniques into 3D version). Specifically, it seems like an ECCV 2022 paper [1] has already addressed a similar design of window attention, this paper is just using such a technique on a different task. Therefore, I think the novelty is limited, and more insights should be explained.\n\n2. Previous point cloud generation works such as PVD and LION conducted experiments on multiple conditional generation tasks to verify their design. However, this paper only shows the results of unconditional generation. I believe more experiments such as point cloud completion or 2D image-to-3D point cloud should be included.\n\n3. The qualitative results shown in the main paper only contain examples generated from DiT-3D and the supplementary material only additionally contains results from PVD, DPM, and SetVAE. As two methods that are quantitatively most comparable to DiT-3D, the visualization of LION and MeshDiffusion are not shown in this paper. So it is not clear how DiT-3D outperforms these methods. \nA minor issue: the computation resources should be detailed in this paper. \n\n[1] SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds (ECCV 2022)\n\nQUESTIONS:\nPlease refer to the weakness part\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of 3D generation. Inspired by the recent progress of utilizing transformers in 2D image generation with diffusion processes (DiT [1]), this paper proposes to replace the common U-Net in 3D diffusion models with a plain transformer. To adapt the DiT to the 3D scenario, authors make several modifications to the vanilla 2D DiT. Experiments on ShapeNet demonstrate improvements over baselines.\n\nSTRENGTHS:\nThis paper tackles an important task of 3D generations that is important for many downstream tasks.\nThe paper is generally well-written and easy to follow.\nExperiments are thorough and convincing.\n\nWEAKNESSES:\nMy main concerns are about the incremental development of the proposed DiT-3D based on DiT [1]. Specifically,\n\na. **Diffusion on voxelized point cloud** has been studied in [12];\n\nb. **3D positional embeddings** is a natural/must-have modification from DiT's 2D positional encodings;\n\nc. **3D window attentions** may not be necessary. Actually, I am quite confused why we need the following procedure: point cloud -> voxel (Sec. 3.2 Voxelized point clouds) -> patches in voxels (Sec. 3.2 patch embeddings) -> reshape patches in voxels into 3D window (Sec. 3.2 3D window). I think these hierarchical steps essentially just change the **actual voxel size**. Then why don't we just have a voxel with a resolution of $(p \\cdot R)^3$ at the very beginning? Here $p$ is the patch size the author used in \"3D Positional and Patch Embeddings\"(Sec. 3.2) and $R$ is the number of patches for \"3D window\" (Sec. 3.2). And this does not prevent authors from applying 3D convolution to exchange information (L179).\n\nIf the above abstraction/simplification is correct, it seems like the major modifications to 3D are just changing from image patches to voxels.\n\nd. Authors state\n> These results indicate that our DiT-3D can support flexible transferability on modality and domain, which is different from previous 3D generation methods [12, 13] based on U-Net as the backbone of DDPMs (L325-327). \n\nCan the authors explain whether there are some experiments to support this statement? As in Tab. 3, we only have results from DiT-3D without any baselines.\n\nQUESTIONS:\n1. Authors state\n> 3D shape generation is a challenging and important problem that seeks to synthesize high-fidelity point clouds ... (L27)\n\nI do not think this is a proper statement as there are many 3D representations besides point clouds, e.g., mesh [15] or implicit ones [A, B].\n\n2. L190 \"Q, K, V have the same dimensions\" where the notation \"V\" is the same as the voxel resolution in L176. Please modify it to make readers be able to distinguish the two.\n\n3. L226 \"It’s worth noting that we initialize $\\gamma$ to 1, which is then multiplied with the frozen layers\": $\\gamma$ is not explained.\n\n4. L266 \"trained on point clouds (l-GAN) and latent variables (l-GAN)\": duplicated 1-GAN.\n\n[A] 3D Neural Field Generation using Triplane Diffusion. CVPR 2023\n\n[B] SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation. CVPR 2023.\n\nLIMITATIONS:\nNo limitations are provided by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to adapt Diffusion Transformer [1] to class-conditional 3D point cloud generation task. In order to achieve it, the authors propose (1) to apply diffusion/infusion directly to point clouds rather than work in latent space; (2) transform input point clouds to voxel grids to apply transformers to tokens extracted from 3D grids in a straightforward way; (3) to reduce the complexity voxel features are processed in patches to produce patch tokens and self-attention in the transformer is modified to aggregate tokens in a window of a predefined size; (4) final voxel features are devoxelized back into points for per-point noise prediction in the infusion process.\n\nExperimental results show promising results in class-specific point cloud generation on ShapeNet dataset, various ablation studies demonstrating the importance of different components and some additional transferability studies that use selective fine-tuning to adapt models pretrained on a data modality/domain to another one.\n\nSTRENGTHS:\nThe main strength of the paper lies in experimental results beating state of the art with an novel approach, not based on prior 3D point cloud generation works. Although it is an adaptation of Diffusion Transformers working with images to 3D point clouds, such a transition from one data type to another is not trivial, so it is remarkable that the authors made it work in this setting.\n\nWEAKNESSES:\nIn my opinion the main weakness of the paper is the quality of the presentation. Text can still be polished to improve readability and correct some mistakes. The authors claim across the paper (e.g. L120) that they apply diffusion for the voxelized point clouds, and I think it is misleading. In fact the diffusion is applied to regular point clouds, it is the infusion network that is designed to operate on voxelized 3D features, but as far as I understand, the denoising is applied on a per-point basis. Clarity of explanation can also be improved, since some details are missing (see questions).\n\nThe authors show a lot of ablation studies, but comparisons to external approaches are limited by the main single-class generation experiment. Qualitative comparisons are moved to supplementary materials, while it is better to show it in the main paper (quality of these comparisons could be improved by decreasing point sizes and sampling more points per shape, so some finer details could be examined).\n\nQUESTIONS:\n1. In principle, point to voxel and backwards transitions do not necessarily preserve all the points, since if the resolution is low, several point will collapse into a single voxel and will not be recovered during devoxelization. How is this avoided in this work?\n\n2. Since evaluations in this work only include single-class data setups, the expressivity of the proposed model is left underexamined. LION included some experiment showing that their approach is capable of generating realistic point clouds when pretrained on multi-class data. It would be interesting to compare how your class-conditioned model will perform in that setting.\n\n3. What is a «plain transformer»? How this «plain» property is characterized? I think, it is better to drop this adjective or change it and be more specific.\n\nMinor comments:\n\n* The optimal best possible value of 1-NNA metric is 50%, since it means that the nearest neighbour classifier is incapable of distinguishing true from gerenated samples. The text and tables should be modified accordingly.\n\n* L14: «high computation» -> high computational costs\n\n* L119, L340: «operate» -> perform or implement\n\n* L234: sentence is broken\n\nLIMITATIONS:\nThe authors do not provide any statements about such limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Diffusion Transformer for 3D shape generation (point cloud), named DiT-3D, which conducts the denoising process on voxelized point clouds. Technically, it introduces 3D positional and patch embeddings, as well as 3D window attention. The main experiments are done on ShapeNet. In addition, the authors empirically show that the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet.\n\nSTRENGTHS:\n- The paper is clearly written and easy to follow.\n- The authors extend the 2D window attention operator to 3D.\n- The authors empirically show the benefit of leveraging a  2D transformer pre-trained on natural images for 3D generation.\n\nWEAKNESSES:\n1. It seems that an important baseline or reference is missing: \"Point-E: A System for Generating 3D Point Clouds from Complex Prompts\". In L172-L173, the authors claim that \"We tried to train the diffusion transformer on point coordinates, but it did not work since point clouds are sparsely distributed in the 3D embedding space\". However, given Point-E, it sounds not that convincing. Can the authors explain why Point-E is not mentioned or compared in the paper?\n2. It is hard to tell whether the generated shape is of high fidelity if the number of points is only 2048 (L245). It is more convincing if the number of points is larger than 4096 (Point-E) or 16384 (usually used in high-fidelity point completion). In addition, it will be visually better if the authors can present the 3D shapes in the format of mesh (like Point-E, MeshDiffusion) or colored point clouds. Currently, it is hard to tell whether the point cloud is of high fidelity. Visually, GET3D and Point-E look better than this work.\n3. Only ShapeNet is used. Currently, there are more and more 3D datasets. It will be better if the authors can show results on more categories of ShapeNet, ABO, or (a subset of) Objaverse.\n\nQUESTIONS:\nSince the evaluation metric is based on Chamfer Distance or Earth Mover’s Distance, I assume that the authors sample a fixed number of points from the GT mesh (and predicted mesh if the baseline, e.g., GET3D, is a mesh-based), which is 2048. I am not sure whether the metric might favor the proposed method especially when the number of points is small, as details can be missed under such a condition. The authors can try to use a larger number of points to compare baselines with the proposed method.\n\nLIMITATIONS:\nThe limitation is not adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DiT-3D, a groundbreaking diffusion transformer for 3D shape generation. It addresses the limitations of previous 3D diffusion methods that mainly relied on the U-Net architecture. DiT-3D leverages the power of Transformers to directly operate the denoising process on voxelized point clouds, resulting in superior scalability and high-quality generations. The authors incorporate 3D positional and patch embeddings to aggregate input from voxelized point clouds and mitigate the computational cost of self-attention by employing 3D window attention in Transformer blocks. The proposed DiT-3D achieves state-of-the-art performance on the ShapeNet dataset, showcasing its ability to generate diverse and high-fidelity 3D point clouds.\n\nSTRENGTHS:\n1. DiT-3D achieves state-of-the-art performance in single-category point cloud generation, demonstrating its effectiveness in generating high-quality 3D shapes.\n2. The utilization of pre-trained DiT-2D checkpoints from ImageNet to improve DiT-3D on ShapeNet showcases the transferability of 2D diffusion models to the 3D domain, which is an interesting and promising approach.\n3. The model is concise, and the paper is well-written, accompanied by clear and visually appealing illustrations.\n\nWEAKNESSES:\n1. The authors only conducted unconditional generation experiments for single-category and three-category cases, limiting the application fields.\n2. Although the window attention technique is employed to mitigate computational costs, there are concerns regarding the generation speed, when operating on 32 * 32 * 32 (even 64 * 64 * 64) voxel grids.\n3. Most DiT-3D models, except for DiT-3D-S, have parameters exceeding 100 million, considerably higher than other existing 3D generation methods.\n\nQUESTIONS:\n1. Have the authors considered exploring diffusion in the latent space? This may potentially enhance the overall performance and inference speed.\n2. It would be more valuable to investigate multi-category generation, such as training on entire ShapeNet-13, ShapeNet-55, or even larger datasets like Objaverse [Deitke et al., 2023]. Previous work [Sanghi et al., 2022, 2023] suggests that voxel representation could aid in generalization to some extent.\n\n\n[Deitke et al., 2023] Objaverse: A universe of annotated 3d objects.  In CVPR.\n\n[Sanghi et al., 2022] CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation.  In CVPR.\n\n[Sanghi et al., 2023] CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language.  In CVPR.\n\nLIMITATIONS:\nN.A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}