SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Modality Transfer: 2D (ImageNet)  $\rightarrow$  3D (ShapeNet). As large-scale pre-trained DiT checkpoints $^2$  are readily available, we can skip training our diffusion transformer from scratch. Instead, we can load most of the weights from the DiT [1] pre-trained on ImageNet [37] into our DiT-3D and continue with fine-tuning. To further optimize training efficiency, we adopt the parameter-efficient fine-tuning approach described in recent work, DiffFit [4], which involves freezing the majority of parameters and only training the newly-added scale factors, bias term, normalization, and class condition modules. It's worth noting that we initialize  $\gamma$  to 1, which is then multiplied with the frozen layers.
Domain Transfer: Source Class  $\rightarrow$  Target Class. Given a pre-trained DiT-3D model on chair data, we can use the parameter-efficient fine-tuning approach to extend its applicability to new categories. Specifically, following the same methodology as described above, we leverage the fine-tuning strategy of DiffFit and obtain satisfactory generation results.
$^{2}$ https://github.com/facebookresearch/DiT/tree/main/diffusion
Table 1: Comparison results (%) on shape metrics of our DiT-3D and baseline models.
Method	Chair	Airplane	Car
1-NNA (↓)	COV (↑)	1-NNA (↓)	COV (↑)	1-NNA (↓)	COV (↑)
CD	EMD	CD	EMD	CD	EMD	CD	EMD	CD	EMD	CD	EMD
r-GAN [8]	83.69	99.70	24.27	15.13	98.40	96.79	30.12	14.32	94.46	99.01	19.03	6.539
l-GAN (CD) [8]	68.58	83.84	41.99	29.31	87.30	93.95	38.52	21.23	66.49	88.78	38.92	23.58
l-GAN (EMD) [8]	71.90	64.65	38.07	44.86	89.49	76.91	38.27	38.52	71.16	66.19	37.78	45.17
PointFlow [9]	62.84	60.57	42.90	50.00	75.68	70.74	47.90	46.41	58.10	56.25	46.88	50.00
SoftFlow [10]	59.21	60.05	41.39	47.43	76.05	65.80	46.91	47.90	64.77	60.09	42.90	44.60
SetVAE [18]	58.84	60.57	46.83	44.26	76.54	67.65	43.70	48.40	59.94	59.94	49.15	46.59
DPF-Net [11]	62.00	58.53	44.71	48.79	75.18	65.55	46.17	48.89	62.35	54.48	45.74	49.43
DPM [29]	60.05	74.77	44.86	35.50	76.42	86.91	48.64	33.83	68.89	79.97	44.03	34.94
PVD [12]	57.09	60.87	36.68	49.24	73.82	64.81	48.88	52.09	54.55	53.83	41.19	50.56
LION [13]	53.70	52.34	48.94	52.11	67.41	61.23	47.16	49.63	53.41	51.14	50.00	56.53
GET3D [14]	75.26	72.49	43.36	42.77	-	-	-	-	75.26	72.49	15.04	18.38
MeshDiffusion [15]	53.69	57.63	46.00	46.71	66.44	76.26	47.34	42.15	81.43	87.84	34.07	25.85
DiT-3D (ours)	49.11	50.73	52.45	54.32	62.35	58.67	53.16	54.39	48.24	49.35	50.00	56.38
3.4 Relationship to DiT [1]
Our DiT-3D contains multiple different and efficient designs for 3D shape generation compared with DiT [1] on 2D image generation:
- We effectively achieve the diffusion space on voxelized point clouds, while DiT needs the latent codes from a pre-trained variational autoencoder as the denoising target.
- Our plain diffusion transformer first incorporates frequency-based sine-cosine 3D positional embeddings with patch embeddings for voxel structure locality.
- We are the first to propose efficient 3D window attention in the transformer blocks for reducing the complexity of the self-attention operator in DiT.
- We add a devoxelized operator to the final output of the last linear layer from DiT for denoising the noise prediction in the point cloud space.
Datasets. Following most previous works [12, 13], we use ShapeNet [38] Chair, Airplane, and Car as our primary datasets for 3D shape generation. For each 3D shape, we sample 2,048 points from 5,000 provided points in [38] for training and testing. We also use the same dataset splits and pre-processing in PointFlow [9], which normalizes the data globally across the whole dataset.
Evaluation Metrics. For comprehensive comparisons, we follow prior work [12, 13] and use Chamfer Distance (CD) and Earth Mover's Distance (EMD) as our distance metrics in computing 1-Nearest Neighbor Accuracy (1-NNA) and Coverage (COV) as main metrics to measure generative quality. 1-NNA calculates the leave-one-out accuracy of the 1-NN classifier to quantify point cloud generation performance, which is robust and correlates with generation quality and diversity. A lower 1-NNA score is better. COV measures the number of reference point clouds matched to at least one generated shape, correlating with generation diversity. Note that a higher COV score is better but does not measure the quality of the generated point clouds since low-quality but diverse generated point clouds can achieve high COV scores.
Implementation. Our implementation is based on the PyTorch [39] framework. The input voxel size is  $32 \times 32 \times 32 \times 3$ , i.e.,  $V = 32$ . The final linear layer is initialized with zeros, and other weights initialization follows standard techniques in ViT [35]. The models were trained for 10,000 epochs using the Adam optimizer [40] with a learning rate of  $1e - 4$  and a batch size of 128. We set  $T = 1000$  for experiments. In the default setting, we use S/4 with patch size  $p = 4$  as the backbone. Note that we utilize 3D window attention in partial blocks (i.e., 0,3,6,9) and global attention in other blocks.
Figure 3: Qualitative visualizations of high-fidelity and diverse 3D point cloud generation.
4.2 Comparison to State-of-the-art Works
In this work, we propose a novel and effective diffusion transformer for 3D shape generation. In order to validate the effectiveness of the proposed DiT-3D, we comprehensively compare it to previous non-DDPM and DDPM baselines. 1) r-GAN, 1-GAN [8]: (2018'ICML): generative models based on GANs trained on point clouds (l-GAN) and latent variables (l-GAN); 2) PointFlow [9] (2019'ICCV): a probabilistic framework to generate 3D point clouds from a two-level hierarchy of distributions with the continuous normalizing flow; 3) SoftFlow [10] (2020'NeurIPS): a probabilistic framework for training normalizing flows on manifolds to estimate the distribution of various shapes; 4) SetVAE [18] (2021'CVPR): a hierarchical variational autoencoder for sets to learn latent variables for coarse-to-fine dependency and permutation invariance; 5) DPF-Net [11] (2020'ECCV): a discrete latent variable network that builds on normalizing flows with affine coupling layers; 6) DPM [29] (2021'ICCV): the first DDPM approach to learn the reverse diffusion process for point clouds as a Markov chain conditioned on shape latent; 7) PVD [12] (2021'ICCV): a strong DDPM baseline based on the point-voxel representation of 3D shapes; 8) LION [13] (2022'NeurIPS): a recent method based on two hierarchical DDPMs in global latent and latent points spaces; 9) GET3D [14] (2022'NeurIPS): a generative model that directly generates explicit textured 3D meshes based on two latent codes (a 3D SDF and a texture field); 10) MeshDiffusion [15] (2023'ICLR): a very recent DDPM method using graph structure of meshes and deformable tetrahedral grid parametrization of 3D mesh shapes.
For chair generation, we report the quantitative comparison results in Table 1. As can be seen, we achieved the best performance in terms of all metrics compared to previous non-DDPM and DDPM baselines. In particular, the proposed DiT-3D significantly outperforms DPF-Net [11], the current state-of-the-art normalizing flows baseline, decreasing by 12.89 1-NNA@CD & 7.80 1-NNA@EMD, and increasing by 7.74 COV@CD & 3.8 COV@EMD. Moreover, we achieve superior performance gains compared to MeshDiffusion [15], the current state-of-the-art DDPM baseline on meshes, which implies the importance of replacing the U-Net with a plain diffusion transformer from observed point clouds for generating high-fidelity 3D shapes. Meanwhile, our DiT-3D outperforms LION [15] by a large margin, where we achieve the performance gains of 4.59 1-NNA@CD & 1.61 1-NNA@EMD, and 3.51 COV@CD & 2.21 COV@EMD. These significant improvements demonstrate the superiority of our method in 3D shape generation. In addition, significant gains in airplane and car generations can be observed in Table 1. These qualitative results also showcase the effectiveness of applying a plain diffusion transformer to operate the denoising process from point clouds for generating high-fidelity and diverse shapes, as shown in Figure 3.
4.3 Experimental Analysis
In this section, we performed ablation studies to demonstrate the benefit of introducing three main 3D design components (voxel diffusion, 3D positional embeddings, and 3D window attention) in 3D shape generation. We also conducted extensive experiments to explore the efficiency of 3D window attention, modality and domain transferability, and scalability.
Table 2: Ablation studies on 3D adaptation components of our DiT-3D.
Voxel Diffusion	3D Pos Embed	3D Window Attention	Training Cost (hours)	1-NNA (↓)	COV (↑)
CD	EMD	CD	EMD
X	X	X	86.53	99.86	99.93	7.768	4.653
✓	X	X	91.85	67.46	69.47	38.97	41.74
✓	✓	X	91.85	51.99	49.94	54.76	57.37
✓	✓	✓	41.67	49.11	50.73	52.45	54.32
Table 3: Transferability studies on modality and domain with parameter-efficient fine-tuning.
ImageNet
Pre-train	Efficient
Fine-tuning	Params
(MB)	1-NNA (↓)	COV (↑)	Source
Domain	Target
Domain	Params
(MB)	1-NNA (↓)	COV (↑)
CD	EMD	CD	EMD	CD	EMD	CD	EMD
X	X	32.8	51.99	49.94	54.76	57.37	Chair	Chair	32.8	51.99	49.94	54.76	57.37
✓	X	32.8	49.07	49.76	53.26	55.75	Airplane	Chair	0.09	52.56	50.75	53.71	56.32
✓	✓	0.09	50.87	50.23	52.59	55.36	Airplane	Airplane	32.8	62.81	58.31	55.04	54.58
Chair	Airplane	0.09	63.58	59.17	53.25	53.68
(a) Modality transfer.
(b) Domain transfer.
Ablation on 3D Design Components. In order to validate the effectiveness of the introduced 3D adaptation components (voxel diffusion, 3D positional embeddings, and 3D window attention), we ablate the necessity of each module and report the quantitative results in Table 2. Note that no voxel diffusion means we directly perform the denoising process on point coordinates without voxelized point clouds and devoxelization prediction. We can observe that adding bearable voxel diffusion to the vanilla baseline highly decreases the results of 1-NNA (by 32.40 @CD and 30.46 @AUC) and increase the performance of COV (by 31.202 @CD and 37.087 @EMD), which demonstrates the benefit of voxelized point clouds and devoxelization prediction in denoising process for 3D shape generation. Meanwhile, introducing 3D positional embedding in the baseline with voxel diffusion also increases the shape generation performance in terms of all metrics. More importantly, incorporating 3D window attention and two previous modules together into the baseline significantly decreases the training cost by 44.86 hours and results of 1-NNA by 50.75 @CD and 49.2 @EMD, and raises the performance of COV by 44.682 @CD and 49.667 @EMD. These improving results validate the importance of the proposed 3D adaptation components in the plain diffusion transformer to operate the denoising process from observed point clouds for 3D shape generation.
Influence of 2D Pretrain (ImageNet). In order to show the modality transferability of the proposed approach from 2D ImageNet pre-trained weights to 3D generation with parameter-efficient fine-tuning, we report the ablation results of ImageNet pre-train and efficient fine-tuning on chair generation in Table 3a. From comparisons, two main observations can be derived: 1) With the initialization with 2D ImageNet pre-trained weights, the proposed DiT-3D improves the quality of shape generation by decreasing 1-NNA by  $2.92@CD$  and  $0.18@EMD$ . 2) Incorporating parameter-efficient fine-tuning into 2D ImageNet pre-trained weights highly decreases the training parameters while achieving competitive generation performance.
Transferability in Domain. In addition, we explore the parameter-efficient fine-tuning for domain transferability in Table 3b. By only training 0.09MB parameters of models from the source class to the target class, we can achieve a comparable performance of quality and diversity in terms of all metrics. These results indicate that our DiT-3D can support flexible transferability on modality and domain, which differs from previous 3D generation methods [12, 13] based on U-Net as the backbone of DDPMs.
Scaling Patch size, Voxel size and Model Size. To explore the scalability of our plain diffusion transformer to flexible designs, we ablate the patch size from  $\{2,4,8\}$ , voxel size from  $\{16,32,64\}$ , and the model size from  $\{\mathrm{S} / 4,\mathrm{B} / 4,\mathrm{L} / 4,\mathrm{XL} / 4\}$ . As seen in Table 4a, when the patch size is 2, the proposed DiT-3D achieves the best performance. This trend is also observed in the original DiT [1] work for 2D image generation. In addition, increasing the voxel size from 16 to 64 for the input of the diffusion denoising process raises the performance in terms of all metrics, as shown in Table 4b. More importantly, we can still observe performance gains by scaling up the proposed plain diffusion transformer to XL/4 when the model is trained for 2,000 epochs. These promising results further demonstrate the strong scalability of our DiT-3D to flexible patch size, voxel size, and model sizes for generating high-fidelity 3D shapes.
Table 4: Scalability studies on flexible patch, voxel, and model sizes.
Patch Size	1-NNA (↓)	COV (↑)	Voxel Size	1-NNA (↓)	COV (↑)	Model Size	Params (MB)	1-NNA (↓)	COV (↑)
CD	EMD	CD	EMD	CD	EMD	CD	EMD	CD	EMD	CD	EMD
8	53.84	51.20	50.01	52.49	16	54.00	50.60	50.73	52.26	S/4	32.8	56.31	55.82	47.21	50.75
4	51.99	49.94	54.76	57.37	32	51.99	49.94	54.76	57.37	B/4	130.2	55.59	54.91	50.09	52.80
2	51.78	49.69	54.54	55.94	64	50.32	49.73	55.45	57.32	L/4	579.0	52.96	53.57	51.88	54.41
XL/4	674.7	51.95	52.50	52.71	54.31
(a) Patch size.
(b) Voxel size.
(c) Model size.
In this work, we present DiT-3D, a novel plain diffusion transformer for 3D shape generation, which can directly operate the denoising process on voxelized point clouds. Compared to existing U-Net approaches, our DiT-3D is more scalable in model size and produces much higher quality generations. Specifically, we incorporate 3D positional and patch embeddings to aggregate input from voxelized point clouds. We then incorporate 3D window attention into Transformer blocks to reduce the computational cost of 3D Transformers, which can be significantly high due to the increased token length resulting from the additional dimension in 3D. Finally, we leverage linear and devoxelization layers to predict the denoised point clouds. Due to the scalability of the Transformer, DiT-3D can easily support parameter-efficient fine-tuning with modality and domain transferability. Empirical results demonstrate the state-of-the-art performance of the proposed DiT-3D in high-fidelity and diverse 3D point cloud generation.
[1] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022. 1, 2, 3, 4, 5, 6, 7, 9
[2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 2, 3
[3] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555, 2023. 1, 3
[4] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. arXiv preprint arXiv:2304.06648, 2023. 1, 3, 6
[5] Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 605-613, 2017. 1
[6] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A papier-mâché approach to learning 3d surface generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 216-224, 2018. 1
[7] Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Bongsoo Choy, and Silvio Savarese. Deformnet: Free-form deformation network for 3d shape reconstruction from a single image. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pages 858-866, 2017. 1
[8] Panos Achlioptas, Olga Diamanti, Ioannis Mitlagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In Proceedings of the International Conference on Machine Learning (ICML), 2018. 1, 3, 7, 8
[9] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4541-4550, 2019. 1, 3, 7, 8