Title: VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Text to shape generation

3D shape generation

Efficient inference

Representation Learning
Abstract: Conditional 3D generation is undergoing a significant advancement, enabling the free creation of 3D content from inputs such as text or 2D images. However, previous approaches have suffered from low inference efficiency, limited generation categories, and restricted downstream applications. In this work, we revisit the impact of different 3D representations on generation quality and efficiency. We propose a progressive generation method through Voxel-Point Progressive Representation (VPP). VPP leverages structured voxel representation in the proposed Voxel Semantic Generator and the sparsity of unstructured point representation in the Point Upsampler, enabling efficient generation of multi-category objects. VPP can generate high-quality 8K point clouds within 0.2 seconds. Additionally, the masked generation Transformer allows for various 3D downstream tasks, such as generation, editing, completion, and pre-training. Extensive experiments demonstrate that VPP efficiently generates high-fidelity and diverse 3D shapes across different categories, while also exhibiting excellent representation transfer performance. Codes will be released at https://github.com/qizekun/VPP.

VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation

Abstract

Conditional 3D generation is undergoing a significant advancement, enabling the free creation of 3D content from inputs such as text or 2D images. However, previous approaches have suffered from low inference efficiency, limited generation categories, and restricted downstream applications. In this work, we revisit the impact of different 3D representations on generation quality and efficiency. We propose a progressive generation method through Voxel-Point Progressive Representation (VPP). VPP leverages structured voxel representation in the proposed Voxel Semantic Generator and the sparsity of unstructured point representation in the Point Upsampler, enabling efficient generation of multi-category objects. VPP can generate high-quality 8K point clouds within 0.2 seconds . Additionally, the masked generation Transformer allows for various 3D downstream tasks, such as generation, editing, completion, and pre-training. Extensive experiments demonstrate that VPP efficiently generates high-fidelity and diverse 3D shapes across different categories, while also exhibiting excellent representation transfer performance. Codes will be released at https://github.com/qizekun/VPP .

1 Introduction

In the last few years, text-conditional image generation [62, 71, 72, 74] has made significant progress in terms of generation quality and diversity and has been successfully applied in games [7], content creation [47, 48] and human-AI collaborative design [36]. The success of text-to-image generation is largely owed to the large-scale image-text paired datasets [78]. Recently, there has been an increasing trend to boost conditional generation from 2D images to 3D shapes. Many works attempt to use existing text-3D shape paired datasets [7, 9, 24, 50] to train the models, but the insufficient pairs limit these methods to generate multi-category, diverse, and high-res 3D shapes.

To tackle this challenge, some works [37, 65, 82, 94] use NeRF [56] and weak supervision of largescale vision-language models [70] to generate 3D shapes. Although creative quality is achieved, the optimization costs of differentiable rendering are quite expensive and impractical. Some methods use images as bridges [38, 59] for the text-conditional 3D generation, which improves the quality-time trade-off. However, the complex multi-stage model leads to long inference latency and cumulative bias. Moreover, most methods [58, 76] are task-limited due to specific model design that can not be implemented to more downstream tasks, e.g. , editing, leading to narrow application scope. To sum up, current methods are faced with three challenges: low inference efficiency , limited generation category , and restricted downstream tasks , which have not been solved by a unified method.

One of the most crucial components in 3D generation is the geometry representation , and mainstream methods are typically built with voxel, 3D point cloud, or Signed Distance Function (SDF) [11, 61]. However, the aforementioned issues in 3D generation may come from the different properties of these representations, which are summarized as follows:

Voxel is a structured and explicit representation. It shares a similar form with 2D pixels, facilitating the adoption of various image generation methods. However, the dense nature of voxels results in increased computational resources and time requirements when generating high-resolution shapes.

3D Point Cloud is the unordered collection of points that explicitly represents positional information with sparse semantics, enabling a flexible and efficient shape representation. However, 3D point clouds typically dedicatedly designed architectures, e.g. , PointNet [66, 67].

SDF describes shapes through an implicit distance function. It is a continuous representation method and is capable of generating shapes with more intricate details than points or voxels. But it requires substantial computation for high-resolution sampling and is sensitive to initialization.

Due to significant shape differences across various categories, the structured and explicit positional information provides spatial cues, thereby aiding the generalization of multiple categories. It coincides with previous multi-category generation methods [76, 77]. In contrast, single-category generation may require less variation. Hence an implicit and continuous representation would contribute to the precise expression of local details [9, 25].

Motivated by the analysis above, we aim to leverage the advantage of different forms of representation. Therefore, a novel V oxelP oint P rogressive ( VPP ) Representation method is proposed to achieve efficient and universal 3D generation. We use voxel and point as representations for the coarse to fine stages to adapt to the characteristics of 3D. To further enhance the inference speed, we employ mask generative Transformers for parallel inference [5] in both stages, which can concurrently obtain broad applicability scope [42]. Based on voxel VQGAN [21], we utilize the discrete semantic codebooks as the reconstruction targets to obtain authentic and rich semantics. Given the substantial computational resources required by high-resolution voxels, we employ sparse points as representations in the

smoothed points

grid points

smoothed points

grid points

rec voxel

input voxel

(a) Reconstruction Results of 3D VQGAN

rec voxel

(with          )

second stage. Inspired by the masked point modeling approaches based on position cues [60, 98], we utilize the voxels generated in the first stage as the positional encoding for generative modeling. We reconstruct the points and predict semantic tokens to obtain sparse representations. Table 1 compares the efficiency and universality of text-conditional 3D generation methods.

In summary, our contributions are: (1) We propose a novel voxel-point progressive generation method that shares the merits of different geometric representations, enabling efficient and multi-category 3D generation. Notably, VPP is capable of generating high-quality 8K point clouds within 0.2 seconds on a single RTX 2080Ti. (2) To accommodate 3D generation, we implement unique module designs, e.g. 3D VQGAN, and propose a Central Radiative Temperature Schedule strategy in inference. (3) As a universal method, VPP can complete multiple tasks such as editing, completion, and even pre-training. To the best of our knowledge, VPP is the first work that unifies various 3D tasks.

2 VPP

Our goal is to design a 3D generation method by sharing the merits of both geometric representations, which is capable of addressing the three major issues encountered in previous generation methods. In this work, we propose VPP, a Voxel-Point Progressive Representation method for two-stage 3D generation. To tackle the problem of limited generation categories , VPP employs explicit voxels and discrete VQGAN for semantics representation in the first stage. To address the issue of low inference efficiency , VPP employs efficient point clouds as representations in the second stage and leverages the parallel generation capability of the mask Transformer. Given the wide range of applications of generative modeling, VPP resolves the problem of restricted downstream tasks . The training overview of VPP is shown in Fig. 3.

2.1 How Does 2D VQGAN Adapt to 3D Voxels?

Due to the formal similarity between voxels and pixels, we can readily draw upon various image generation methods. VQGAN [21] is capable of generating realistic images through discrete semantic codebooks and seamlessly integrating with mask generative Transformers [5, 6]. We utilize 3D volumetric SE-ResNet [35] as the backbone of VQGAN encoder E , decoder G and discriminator D .

Given one voxel observation v ∈ R 3 whose occupancy is defined as function o : R 3 → [0 , 1] . The original VQGAN is trained by optimizing a vectorized quantized codebook with the loss L VQ during autoencoding while training a discriminator between real and reconstructed voxels with loss L GAN. Denoting the reconstructed voxel result as ˆ v , the overall loss of vanilla VQGAN can be written as:

However, unlike 2D images, 3D voxels only have non-zero values in the region that encompasses the object in grid centroids. This may lead to a short-cut solution that degrades by outputting voxels with values as close to zero as possible, i.e. , all empty occupancy. To address this issue, we propose to

(b) Point Upsampler

Decoder

Reconstructed

Grid

Smoother

Seq-Invariance

CD Loss

Tokenizer

mask

L1 Loss

Reconstructed

Tokens

Mask Voxel

Transformer

3D VQGAN

Encoder

(a) Voxel Semantic Generator

Voxelization

mask

Prompt

Embedding

Cross Entropy Loss

Mask Point

Transformer

Voxelization

Input Point

Clouds

Input Point

Clouds

EP

U

2P

minimize the occupancy disparity between the reconstructed and ground truth voxels. Formally, let ζ ◦ o ( v ) be the averaged occupancy in SE(3) space, the overall loss L 3D-VQGAN is:

where L OCC is proposed to minimize the occupancy rate for better geometry preservation.

2.2 How to Bridge the Representation Gap Between Voxels and Points?

We aim to utilize the voxel representation as the intermediate coarse generation before fine-grained point cloud results. However, voxels are uniform and discrete grid data, while point clouds are unstructured with unlimited resolutions, which allows for higher density in complex structures. Besides, point clouds typically model the surfaces, while voxel representation is solid in nature. This inevitably leads to a representation gap issue, and a smooth transformation that bridges the representation gap is needed. We employ the Lewiner algorithm [40] to extract the surface of the object based on voxel boundary values and utilize a lightweight sequence-invariant Transformer [84] network T η parametrized with η as Grid Smoother to address the issue of grid discontinuity. Specifically, given generated point clouds P = { p i ∈ R 3 } N i =1 with N points. The model is trained by minimizing the geometry distance between generated and ground truth point clouds G = { g i ∈ R 3 } N i =1 that is downsampled to N with farthest point sampling (FPS). To make the generated surface points more uniformly distributed, we propose L uniform that utilizes the Kullback-Leibler (KL) divergence D KL between the averaged distance of every centroid p i and its K-Nearest Neighborhood (KNN) N i . Then we optimize the parameter η of Grid Smoother by minimizing loss L SMO:

where L CD is ℓ 1 Chamfer Distance [23] geometry disparity loss, U is the uniform distribution.

2.3 Voxel-Point Progressive Generation Trough Mask Generative Transformer

We employ mask generative Transformers [5] for rich semantic learning in both the voxel and point stages. Mask generative Transformer can be seen as a special form of denoising autoencoder [34, 85]. It forces the model to reconstruct masked input to learn the statistical correlations between local patches. Furthermore, generating reconstruction targets through a powerful tokenizer can further enhance performance [2, 19, 91, 104]. From the perspective of discrete variational autoencoder

Position

Embedding

Masked

Reconstructed

Generated

Prompt

Embedding

Mask Voxel

Transformer

3D VQGAN

Decoder

CLIP

Grid

Smoother

Decoder

Mask Point

Transformer

Masked

Reconstructed

Generated

Text /

Image

(dVAE) [2, 26, 39, 71], the overall optimization is to maximize the evidence lower bound (ELBO) [17, 32, 73] of the log-likelihood P( x i | ˜ x i ) . Let x denote the original data, ˜ x the masked data, and z the semantic tokens, the generative modeling can be described as:

where (1) Q ϕ ( z | x ) denotes the discrete semantic tokenizer; (2) P ψ ( x | z ) is the tokenizer decoder to recover origin data; (3) ˜ z = Q ϕ ( z | ˜ x ) denotes the masked semantic tokens from masked data; (4) P θ ( z | ˜ z ) reconstructs masked semantic tokens in an autoencoding way. In the following sections, we extend the mask generative Transformer to voxel and point representations, respectively. The semantic tokens of voxel and point are described as z v and z p in Figs. 3 and 4.

Voxel Semantic Generator We generate semantic voxels with low resolution in the first stage to balance efficiency and fidelity. We utilize Mask Voxel Transformer for masked voxel inputs, where the 3D VQGAN proposed in Section 2.1 is used as the tokenizer to generate discrete semantic tokens. Following prior arts [5, 6], we adopt Cosine Scheduling to simulate the different stages of generation. The masking ratio is sampled by the truncated arccos distribution with density function p ( r ) = 2 π (1 -r 2 ) -1 2 . The prompt embedding comes from the CLIP [70] features of cross-modal information. We render multi-view images from the 3D object [68] and utilize BLIP [41] to get the text descriptions of the rendered images. Furthermore, in order to mitigate the issue of excessive dependence on prompts caused by the high masking ratio ( e.g. , the same results will be generated by one prompt), we introduce Classifier Free Guidance [33] (CFG) to strike a balance between diversity and quality. Following Sanghi et al. [77], the prompt embeddings are added with Gaussian noise to reduce the degree of confidence. We use ϵ ∼ N (0 , 1) and γ ∈ (0 , 1) as the level of noise perturbation for the trade-off control, which will later be studied.

Point Upsampler Point clouds are unordered and cannot be divided into regular grids like pixels or voxels. Typically, previous masked generative methods [19, 60] utilize FPS to determine the center of local patches, followed by KNN to obtain neighboring points as the geometric structure. Finally, a lightweight PointNet [66, 67] or DGCNN [89] is utilized for patch embedding. However, these methods rely on pre-generated Positional Encodings (PE), rendering the use of fixed positional cues infeasible in conditional 3D generation since only conditions like texts are given. To tackle this issue, we adopt the first-stage generated voxels-smoothed point clouds as the PEs. A pretrained Point-MAE [60] encoder is used as the tokenizer ( i.e. , teacher [19]). The Mask Point Transformer learns to reconstruct tokenizer features, which will be used for the pretrained decoder reconstruction.

2.4 Inference Stage

Parallel Decoding An essential factor contributing to the efficient generation of VPP is the parallel generation during the inference stage. The model takes fully masked voxel tokens as input and generates semantic voxels conditioned on the CLIP [70] features of text or image prompts. Following [5, 6], a cosine schedule is employed to select the fixed fraction of the highest confidence masked tokens for

a table lamp

a machine gun

CLIP-Forge

CLIP-Sculptor

VPP (Ours)

a truck

VPP (Ours)

a jet

a rectangular table

a round table

an office chair

X

prediction at each step. Furthermore, we pass the generated voxels through Grid Smoother to obtain smoothed point clouds, which will serve as the positional encoding for the Point Upsampler in the upsampling process. Note that all the stages are performed in parallel. The inference overview of VPP is illustrated in Fig. 4.

Central Radiative Temperature Schedule Due to the fact that 3D shape voxels only have non-zero values in the region that encompasses the object in the center, we hope to encourage diversity in central voxels while suppressing diversity in edge vacant voxels. Consequently, we propose the Central Radiative Temperature Schedule to accommodate the characteristics of 3D generation. As for the voxel with a size of R × R × R , we calculate the distance r to the voxel center for each predicted grid. Then, we set the temperature for each grid as T = 1 -( r/R ) 2 , where T represents the temperature. Thus, we can achieve the grid that is closer to the center will have a higher temperature to encourage more diversity and vice versa.

3 Experiments

3.1 Conditional 3D Generation

Text-conditioned Generation Following [59, 76, 77], we use Accuracy (ACC), Fréchet Inception Distance (FID) [31], and Inception Score (IS) [75] as metrics to assess the generation quality and diversity. We follow [76] to define 234 predetermined text queries and apply a classifier for evaluation. For a fair comparison, all methods use 'a/an' as the prefix for text prompts. Similar to [59], we employ a double-width PointNet++ model [67] to extract features and predict class probabilities for point clouds in ShapetNet [4]. Table 2 shows the comparison between VPP and other methods. It can be observed that our VPP outperforms all other methods substantially in all metrics. Notably, the FID metric of ours is much lower than other methods, demonstrating that our method generates higher quality and richer diversity of 3D shapes.

By employing a Shape As Points [52, 63] model pre-trained on multi-category ShapeNet data, VPP is capable of performing smooth surface reconstruction on generated point clouds. In Blender rendering, we employ varnish and automatic smoothing techniques to improve the display quality of the generated mesh. We qualitatively show in Fig. 5 that our method can generate higher-quality shapes and remain faithful to text-shape correspondence. It is noteworthy that VPP can generate smoother 3D shapes, e.g. , the truck. Besides, our generation results are visually pleasing across different object categories with complex text descriptions. Therefore, the above advantages enable our method to be much more useful in practical applications.

77

！

A

Table 2: Text conditioned 3D generation results on ShapeNetCore13 [4] dataset. Point-E [59] is pre-trained on a large-scale private point-text paired dataset containing millions of samples, and we report its zero-shot transfer results on ShapeNet. Accuracy (ACC), Fréchet Inception Distance (FID), and Inception Score (IS) are reported.

Image-Conditioned Generation By leveraging the CLIP features of 2D images as conditions in the Voxel Semantic Generator, VPP also possesses the capability of single-view reconstruction. Fig. 6 illustrates image-conditional generated samples of VPP. It can be observed that the generated point clouds are of detailed quality and visually consistent with the input image. For example, the flat wings of an airplane and the curved backrest of a chair.

3.2 Text-Conditioned 3D Editing and Upsampling

Text-conditioned 3D editing and upsample completion tasks are more challenging than most conditional 3D generation tasks, which most of previous methods can not realize due to the inflexible model nature. In contrast, our VPP can be used for a variety of shape editing and upsample completion without extra training or model fine-tuning.

Text-Conditioned 3D Editing Since we use the masked generative Transformer in VPP, our model can be conditioned on any set of point tokens: we first obtain a set of input point tokens, then mask the tokens corresponding to a local area, and decode the masked tokens with unmasked tokens and text prompts as conditions. We show the examples in Fig. 7 (a). The figure provides examples of VPP being capable of changing part attributes (view) and modifying local details (view) of the shape, which correctly corresponds to the text prompts.

Point Cloud Upsampling Except for editing, we also present our conditional generative model for the point cloud upsample completion, where the sparse point cloud inputs are completed as a dense output. We use the second stage Point Upsampler to achieve this, and the results are illustrated in Fig. 7 (b). It is observed that VPP generates completed and rich shapes of high fidelity while being consistent with the input sparse geometry.

3.3 Transfer Learning on Downstream Tasks

After training, the Mask Transformer has learned powerful geometric knowledge through generative reconstruction [30], which enables VPP to serve as a self-supervised learning method for downstream representation transferring. Following previous works [19, 60, 68], we fine-tune the Mask Point Transformer encoder, i.e. , the second-stage Point Upsampler for 3D shape recognition.

(b) Point cloud Upsampling

(a) Text-Conditional Editing

with two legs

square chair

with curved sail

plane

ʼ

s wings with engines

ScanObjectNN [83] and ModelNet [92] are currently the two most challenging 3D object datasets, which are obtained through real-world sampling and synthesis, respectively. We show the evaluation of 3D shape classification in Table 3. It can be seen that: (i) Compared to any supervised or selfsupervised method that only uses point clouds for training, VPP achieves the best generalization performance. (ii) Notably, VPP outperforms Point-MAE by +4.0% and +0.3% accuracy on the most challenging PB_T50_RS and ModelNet40 benchmark.

3.4 Diversity & Specific Text-Conditional Generation

We show the diverse qualitative results of VPP on text-conditional 3D generation in Fig. 8 (a). It can be observed that VPP can generate a broad category of shapes with rich diversity while remaining faithful to the provided text descriptions. Meanwhile, we present the qualitative results of VPP on more specific text-conditional 3D generation in Fig. 8 (b). Notably, VPP can generate high-fidelity shapes that react well to very specific text descriptions, like a cup chair , a round table with four legs and an aircraft during World War II , etc .

an ak47

a cup chair

a dressing table

a boat

a round table

with four legs

a round table

with five legs

a square chair

an aircraft during

World War II

a flat jet

a cargo van

an airplane

a taxi car

a bench

(a) Diversity Text-Conditional Generation

(b) Specific Text-Conditional Generation

a sail boat

a porsche 911

origin points

partial points

generated points

origin points

partial points

generated points

3.5 Partial Completion

We present the point cloud partial completion experiments in Fig. 9. By employing a block mask on the original point clouds, VPP can generate partial samples, which illustrates the partial completion capability of VPP. Besides, the generated samples exhibit diversity, further demonstrating the diverse performance of VPP.

a round table

an office chair

3.6 ShapeNet Retrieval Experiment

We conduct the retrieval evaluation on samples generated by VPP from the ShapeNet dataset. The results are shown in Fig. 10. It can be observed that there are no completely identical samples, proving the great generation ability of VPP is not caused by overfitting the ShapeNet dataset. The VPP results more likely come from the understanding and integration of the learned shape knowledge.

4 Related Works

Conditional 3D generation with 2D images or text has witnessed rapid development in recent years. One line of works focuses on solving the ill-posed image-conditioned 3D generation ( i.e. , single-view reconstruction), where impressive results have been achieved [10, 23, 57, 88]. By introducing a text-shape paired dataset based on ShapeNet [4] that includes chair and table categories, [7] pioneered another line of single-category text-conditioned shape generation. Facilitated with this human-crawled 3D data, ShapeCrafter [24] further enables text-conditioned shape edition in a recursive fashion. To leverage easily rendered 2D images rather than human-crafted languages, several works propose to train rendered image-to-shape generation [9, 51, 76, 77] which enables text-conditioned generation through pretrained large vision-language (VL) models, i.e. , CLIP [70]. Besides rendered images, Point-E [59] and Shape-E [38] propose to use real-world text-image-shape triplets from a large-scale in-house dataset, where images are used as the representation bridge.

However, 3D data is significantly lacking and expensive to collect [19]. DreamFields [37] first achieves text-only training by using NeRF [56], where rendered 2D images are weakly supervised to align with text inputs using CLIP. Following this direction, DreamFusion [65] incorporates the distillation loss of a diffusion model [74]. Dream3D [94] improves the knowledge prior by providing the prompt shapes initialization. Though impressive results are obtained, these approaches also introduce significant time costs because of the case-by-case NeRF training during inference. More recently, TAPS3D aligns rendered images and text prompts based on DMTet [79] to enable text-conditional generation, but its generalizability is limited to only four categories. 3DGen [28] achieves high-quality generation through the utilization of Triplane-based diffusion models [3, 80] and the Objaverse dataset [14]. Zero-1-to-3 [46] employs relative camera viewpoints as conditions for diffusion modeling. Based on this, One-2-3-45 [45] achieves rapid open-vocabulary mesh generation through multi-view synthesis. Fantasia3D [8] leverages DMTet to employ differentiable rendering of SDF as a substitute for NeRF, enabling a more direct mesh generation. SJC [87] and ProlificDreamer [90] achieve astonishing results by proposing the variational diffusion distillation.

5 Conclusions

In this paper, we present VPP, a novel Voxel-Point Progressive Representation method that achieves efficient and universal conditional 3D generation. We explore the suitability of different 3D geometric representations and design a VQGAN module tailored for 3D generation. Notably, VPP is capable of generating high-quality 8K point clouds in less than 0.2 seconds using a single RTX 2080Ti GPU. Extensive experiments demonstrate that VPP exhibits excellent performance in conditional generation, editing, completion, and pretraining.

Acknowledgments

This research was supported by the National Key R&D Program of China (2022YFB2804103), the Key Research and Development Program of Shaanxi (2021ZDLGY01-05), the National Natural Science Foundation of China (31970972), and the Institute for Interdisciplinary Information Core Technology (IIISCT).

References

Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2016. 17

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In Int. Conf. Learn. Represent. (ICLR) . OpenReview.net, 2022. 4, 5

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 10

Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR , abs/1512.03012, 2015. 6, 7, 10, 17

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 2, 3, 4, 5

Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Int. Conf. Mach. Learn. (ICML) , 2023. 3, 5

Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian Conf. Comput. Vis. (ACCV) , 2019. 2, 10

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Int. Conf. Comput. Vis. (ICCV) , 2023. 10

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G. Schwing, and Liangyan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 2, 10

Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Eur. Conf. Comput. Vis. (ECCV) , 2016. 10

Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, New Orleans, LA, USA, August 4-9, 1996 , pp. 303-312. ACM, 1996. 2

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2017. 17

Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. CoRR , abs/2307.05663, 2023. 20

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 10, 20

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI Conf. Artif. Intell. (AAAI) , 2021. 17

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language-driven open-vocabulary 3d scene understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 17

Runpei Dong, Zhanhong Tan, Mengdi Wu, Linfeng Zhang, and Kaisheng Ma. Finding the task-optimal low-bit sub-distribution in deep neural networks. In Int. Conf. Mach. Learn. (ICML) , 2022. 5

Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal comprehension and creation. CoRR , abs/2309.11499, 2023. 20

Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? In Int. Conf. Learn. Represent. (ICLR) , 2023. 4, 5, 7, 8, 10, 17, 19

Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer. Point transformer. IEEE Access , 9:134826134840, 2021. 17

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2021. 2, 3, 18

Guofan Fan, Zekun Qi, Wenkai Shi, and Kaisheng Ma. Point-gcc: Universal self-supervised 3d scene pre-training via geometry-color contrast. CoRR , abs/2305.19623, 2023. 17

Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2017. 4, 10

Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. Shapecrafter: A recursive textconditioned 3d shape generation model. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2022. 2, 10

Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2022. 2

Alex Graves. Practical variational inference for neural networks. In John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger (eds.), Adv. Neural Inform. Process. Syst. (NIPS) , 2011. 5

Meng-Hao Guo, Junxiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. PCT: point cloud transformer. Comput. Vis. Media , 7(2):187-199, 2021. 8

Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. CoRR , abs/2303.05371, 2023. 10

Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. MVTN: multi-view transformation network for 3d shape recognition. In Int. Conf. Comput. Vis. (ICCV) , 2021. 17

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 7

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst. (NIPS) , 2017. 6

Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In ACM Conf. Comput. Learn. Theory (COLT) , 1993. 5

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021. 5

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2020. 4

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2018. 3

Rowan T Hughes, Liming Zhu, and Tomasz Bednarz. Generative adversarial networks-enabled humanartificial intelligence collaborative applications for creative and design industries: A systematic review of current approaches and trends. Frontiers in artificial intelligence , 4:604234, 2021. 2

Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 2, 10, 20

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. CoRR , abs/2305.02463, 2023. 2, 10

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Int. Conf. Learn. Represent. (ICLR) , 2014. 5

Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira, and Geovan Tavares. Efficient implementation of marching cubes' cases with topological guarantees. Journal of graphics tools , 8(2):1-15, 2003. 4

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In Int. Conf. Mach. Learn. (ICML) , 2022. 5, 17

Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 2

Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2018. 8

Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning on point clouds. In Eur. Conf. Comput. Vis. (ECCV) , 2022. 19

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-345: Any single image to 3d mesh in 45 seconds without per-shape optimization. CoRR , abs/2306.16928, 2023. 10

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Int. Conf. Comput. Vis. (ICCV) , 2023. 10

Vivian Liu, Han Qiao, and Lydia Chilton. Opal: Multimodal image generation for news illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology , pp. 1-17, 2022. 2

Vivian Liu, Jo Vermeulen, George W. Fitzmaurice, and Justin Matejka. 3dall-e: Integrating text-to-image AI in 3d design workflows. CoRR , abs/2210.11603, 2022. 2

Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Int. Conf. Comput. Vis. (ICCV) , 2021. 17

Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 2

Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. Iss: Image as stetting stone for text-guided 3d shape generation. In Int. Conf. Learn. Represent. (ICLR) , 2023. 10

Zhaoyang Lyu, Jinyi Wang, Yuwei An, Ya Zhang, Dahua Lin, and Bo Dai. Controllable mesh generation through sparse latent point diffusion models. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 6

Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual MLP framework. In Int. Conf. Learn. Represent. (ICLR) , 2022. 8

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In Int. Conf. Comput. Vis. (ICCV) , 2021. 17

Daniel Maturana and Sebastian A. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IEEE/RSJ Int. Conf. Intell. Robot. and Syst. (IROS) , 2015. 17

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021. 2, 10

Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 10

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers , pp. 1-8, 2022. 2

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. CoRR , abs/2212.08751, 2022. 2, 6, 7, 10, 17

Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Eur. Conf. Comput. Vis. (ECCV) , 2022. 3, 5, 7, 8, 17, 19

Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2019. 2

Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Int. Conf. Comput. Vis. (ICCV) , 2021. 2

Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2021. 6

Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas A. Funkhouser. Openscene: 3d scene understanding with open vocabularies. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 17

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In Int. Conf. Learn. Represent. (ICLR) , 2023. 2, 10, 20

Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2017. 2, 5, 8, 17

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Adv. Neural Inform. Process. Syst. (NIPS) , 2017. 2, 5, 6, 8, 17

Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In Int. Conf. Mach. Learn. (ICML) , 2023. 5, 7, 8, 17, 19

Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Abed Al Kader Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2022. 8, 17

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn. (ICML) , 2021. 2, 5, 10, 17

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Int. Conf. Mach. Learn. (ICML) , 2021. 2, 5

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR , abs/2204.06125, 2022. 2

Jorma Rissanen. Modeling by shortest data description. Autom. , 14(5):465-471, 1978. 5

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 2, 10

Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Adv. Neural Inform. Process. Syst. (NIPS) , 2016. 6

Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 2, 6, 7, 10

Aditya Sanghi, Rao Fu, Vivian Liu, Karl Willis, Hooman Shayani, Amir Hosein Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 2, 5, 6, 7, 10

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR , abs/2111.02114, 2021. 2

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2021. 10

J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 10

Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Int. Conf. Comput. Vis. (ICCV) , 2015. 17

Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from A single image with diffusion prior. In Int. Conf. Comput. Vis. (ICCV) , 2023. 2

Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2019. 8, 17

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst. (NIPS) , 2017. 4, 8, 17, 19

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Int. Conf. Mach. Learn. (ICML) , 2008. 4

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Int. Conf. Comput. Vis. (ICCV) , 2021. 19

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 10, 20

Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single RGB images. In Eur. Conf. Comput. Vis. (ECCV) , 2018. 10

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. , 38(5):146:1-146:12, 2019. 5, 8, 19

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2023. 10

Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. MVP: multimodality-guided visual pre-training. In Eur. Conf. Comput. Vis. (ECCV) , 2022. 4

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2015. 8, 17, 19

Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Eur. Conf. Comput. Vis. (ECCV) , 2020. 17

Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 2, 10

Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. ULIP: learning unified representation of language, image and point cloud for 3d understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 17

Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph. , 35(6):1-12, 2016. 17

Li Yi, Hao Su, Xingwen Guo, and Leonidas J. Guibas. Syncspeccnn: Synchronized spectral CNN for 3d shape segmentation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2017. 17

Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022. 3, 8, 17, 19

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: latent point diffusion models for 3d shape generation. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2022. 2

Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pp. 2048-2059, October 2023. 17

Junbo Zhang, Guofan Fan, Guanghan Wang, Zhengyuan Su, Kaisheng Ma, and Li Yi. Language-assisted 3d feature learning for semantic scene understanding. In AAAI Conf. Artif. Intell. (AAAI) , 2023. 17

Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2AE: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In Adv. Neural Inform. Process. Syst. (NeurIPS) , 2022. 8, 19

Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR) , 2023. 8, 19

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan L. Yuille, and Tao Kong. ibot: Image BERT pre-training with online tokenizer. In Int. Conf. Learn. Represent. (ICLR) , 2022. 4

A Additional Related Work

3D Representation Learning includes point-based [66, 67], voxel-based [55], and multiview-based methods [29, 81], etc . Due to the sparse but geometry-informative representation, point-based methods [20, 69] have become mainstream approaches in object classification [83, 92]. Voxel-based CNN methods [15, 97] provide dense representation and translation invariance, achieving outstanding performance in object detection [12] and segmentation [1, 96]. Furthermore, due to the vigorous development of attention mechanisms [84], 3D Transformers [20, 49, 54] have also brought about effective representations for downstream tasks. Recently, 3D self-supervised representation learning has been widely studied. PointContrast [93] leverages contrastive learning across different views to acquire discriminative 3D scene representations. Point-BERT [98] and Point-MAE [60] first introduce masked modeling pretraining into 3D. ACT [19] pioneers cross-modal geometry understanding via 2D/language foundation models. RECON [68] further proposes to unify generative and contrastive learning. Facilitated by foundation vision-language models like CLIP [70], another line of works are proposed towards open-world 3D representation learning [16, 22, 64, 95, 100, 101].

B Implementation Details

B.1 Experimental Details

Training Details We use ShapeNetCore from ShapeNet [4] as the training dataset. ShapeNet is a clean set of 3D CAD models with rich annotations, including 51K unique 3D models from 55 common object categories. For the acquisition of multi-modal data, we follow ReCon [68] for multi-view rendering and utilize BLIP [41] based on rendered images to obtain textual data. Table 4 shows the training hyperparameters and model architecture information of each part of VPP.

Downstream Tasks Details Following Point-E [59], we use pre-trained PointNet++ as a classifier in all ACC, FID, and IS evaluations to extract the features and calculate the accuracy of generated point clouds. In point cloud generation and editing, we employ 8 or 4 steps for a parallel generation. The generation task utilizes initial voxel codebooks composed entirely of [MASK] tokens. In editing, we extract VQGAN features from the original point cloud to initialize the voxel codebooks. As for the transfer classification task on ScanObjectNN [83] and ModelNet40 [92], we fully follow the previous work [19, 60] configuration and trained 300 epochs with the AdamW optimizer, and used the voting strategy in testing.

3D VQGAN

Encoder

Input Point

Clouds

3D VQGAN

Decoder

3D VQGAN

Discriminator

real / fake

Voxelization

..

..

Codebook

1

2

N-1

N

Quantization

L1 Reconstructed Loss

Occupancy Rate Loss

Reconstructed

Voxel

Input

Voxel

Za

2

source

rebuild

source

rebuild

source

rebuild

source

rebuild

source

rebuild

source

rebuild

B.2 Implementation Details of 3D VQGAN

We show the detailed training overview of 3D VQGAN in Fig. 11. Following the training recipe of VQGAN [21], we use L1 loss to supervise the reconstruction of the voxel and feed the reconstructed voxel into the discriminator to ensure the generated authenticity by GAN loss. Besides the L1 loss and GAN loss, we also introduce the occupancy rate loss to make the occupancy rate of the reconstructed voxel grid similar to the ground truth so as to obtain a better reconstruction of the voxel. Fig. 12 illustrates the reconstructed results of 3D VQGAN, showcasing its impressive capabilities in the domain of 3D voxel reconstruction. As depicted in the figure, the model exhibits remarkable noise rectification capabilities. It not only reconstructs the voxels faithfully but also manages to rectify certain imperfections and artifacts present in the input data.

C Additional Experiments

We conduct more experiments to further demonstrate the generation quality and universality of VPP. Including diversity & specific text-conditional generation, few-shot transfer classification, and ablation studies.

C.1 Novel Categories Point Clouds Generation

Besides the generation of common categories, we also conduct novel categories generation experiments to show the generalization performance of VPP in Fig. 13. Based on the learning of common categories, VPP can generalize to generate more novel category shapes, which are visually reasonable, such as a car boat .

a car boat

an airplane boat

a car

a boat

an airplane

(a) common categories

(b) novel categories

C.2 Few-Shot Transfer Classification

We conduct few-shot experiments on the ModelNet40 [92] dataset, and the results are shown in Table 5. In the self-supervised benchmark without the use of additional modality data, VPP achieves excellent performance compared to previous works.

C.3 Ablation Study

Training Hyper Parameter Fig. 15 (a-b) shows the ablation study of the image-text features ratio and Gaussian noise strength. It can be observed that either too large or too small text-image feature ratio and noise are not conducive to the quality and diversity of 3D generation.

Inference Parameters To explore the parameter dependencies of the Mask Voxel Transformer in the inference process, we study the effects of temperature and iteration steps. The results are shown in Fig. 15 (c-d). It can be seen that moderate temperature achieves optimal generation classification accuracy. Higher temperature promotes model diversity, while with the increase of iteration steps, the model's FID initially decreases and then increases.

Backbone Choice Fig. 14 shows the selection of the CLIP model backbones during VPP training. Both ViT-B/32 and ViT/L14 achieve excellent accuracy and diversity.

86.92

88.04

88.54

81.25

31.34

29.62

30.23

38.89

ViT-B/16

ViT-B/32

ViT-L/14

ResNet50

CLIP Backbone Choice

ACC

↑

FID

↓

Ratio

25

30

35

40

45

50

80

81

82

83

84

85

86

87

88

89

0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8

(a) Image-Text Feature Ratio

FID

↓

ACC

↑

FID

↓

Noise

27

28

29

30

31

32

33

34

80

81

82

83

84

85

86

87

88

89

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

ACC

↑

Temperature

28

29

30

31

32

33

34

80

81

82

83

84

85

86

87

88

89

0

0.8

1

1.2

1.5

1.8

(c) Central Radiative Temperature

ACC

↑

FID

↓

Number of Steps

30

31

32

33

34

80

81

82

83

84

85

86

87

88

89

0

2

4

8

12

(d) Iteration Steps

FID

↓

ACC

↑

D Limitations & Future Works

Despite the substantial advantages of our model in terms of generation speed and applicability, there exists a considerable gap in generation quality compared to NeRF-based models [37, 65, 87]. To address this limitation, we intend to explore the utilization of larger models and more extensive datasets for training, e.g. , Objaverse [13, 14]. Furthermore, multi-modal large language models with combined comprehension and generation abilities [18] have advanced rapidly. Incorporating VPP's 3D generation capabilities into such models represents a potential avenue for future research.

Broader Impact

VPP enables the rapid generation of high-quality 3D models in a fraction of a second. This has wide-ranging applications in fields such as gaming, virtual reality, augmented reality, and digital media production. These advances can lead to more immersive and visually stunning user experiences in entertainment and educational content. Furthermore, as a generative model, VPP may produce deceptive and malicious content and potentially impact associated employment opportunities.

Peer reviews:
comments: SUMMARY:
The paper proposes an efficient conditional 3D generation via voxel-point progressive representation. More specifically, a voxel semantic generator and a point upsampler have been created to achieve efficient generation on multi-category objects. To verify the effectiveness of the method, extensive experiments with SOTA results are achieved.

STRENGTHS:
1. The paper is well-written and well-organized. 
2. Extensive experiments are conducted, and impressive results are obtained.

WEAKNESSES:
1. It seems that the paper can address point upsample task instead of upsample completion task, since upsample refers to generate dense points while completion refers to synthesize new points given partial scans.  
2. It would be better to show some ablation studies on the network architectures such as removing or replacing specific components to see how they affect the performances.  
3. It would be clearer to indicate Tab .3 in line 234 for the corresponding quantitative results.

QUESTIONS:
1. To reduce the degree of confidence when generating voxels because of the high masking rate, Gaussian noise is added to the prompt embedding.The question is instead of adding Gaussian noise, how about directly sampling smaller masking rate? 
2. How is Eq. (2) obtained? It would be better to provide some hints on the calculation.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to use a voxel-point progressive representation for efficient 3D generation, and it proposes a few architectures for different applications, including generation, editing, upsampling, and pre-training. Based on the reported results, the proposed method could generate various 3D shapes and could achieve competitive classification results.\n\nSTRENGTHS:\n1. The work proposes a voxel-point progressive representation, which could provide good 3D generation results on the ShapeNet dataset as shown in the reported experiments. \n\n2. Many modules, such as 3D VQGAN, Voxel Semantic Generator, Grid Smoother, and Point Upsampler, have been proposed.\n\nWEAKNESSES:\n1. The proposed method could only generate 3D shapes belonging to the categories of the ShapeNet dataset.  It does not show any 3D shape results from unseen categories even with the help of CLIP.\n\n2. Point-Voxel representation has been broadly studied in previous methods, such as [A-C], which has already been proven to be an efficient representation for 3D point cloud analysis.\n\n3. The classification results for both the ScanObjectNN and ModelNet40 datasets are very saturated.  Also, the improvements seem to be very incremental.\n\n[A] Liu, Z., Tang, H., Lin, Y., & Han, S. (2019). Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32.\n\n[B] Zhang, C., Wan, H., Shen, X., & Wu, Z. (2022). PVT: Point‐voxel transformer for point cloud learning. International Journal of Intelligent Systems, 37(12), 11985-12008.\n\n[C] Liu, Z., Tang, H., Zhao, S., Shao, K., & Han, S. (2021). Pvnas: 3d neural architecture search with point-voxel convolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8552-8568.\n\nQUESTIONS:\n1. Is the proposed method capable of generating 3D shapes of novel categories?\n\n2. Is the proposed method capable of generating complete point clouds given a partial point cloud?\n\n3. Does the proposed method demonstrate robustness when presented with noisy point clouds as input?\n\n4. Is the generated 3D shapes caused by overfitting the ShapeNet dataset? Maybe the authors could try to retrieve the most similar shape in the ShapeNet dataset to prove its generation ability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose an approach to generate 3D point clouds of objects with an image or text description as input. Authors use a pre-trained CLIP model to generate text/image embeddings and use this to first generate features in voxel space (Voxel Semantic Generator). These voxel features are then decoded into a coarse voxel grid. Authors then convert the voxels to point clouds and use a smoothing network to obtain a uniform point cloud. Authors use a transformer to then convert this point cloud into a detailed 3D shape (also point cloud).\nAuthors show qualitative and quantitative comparison with relevant baselines (CLIP-Sculptor, CLIP-Forge and Point-E).\nAuthors also show several applications like text and image conditioned 3D generation, shape editing and completion.\n\nSTRENGTHS:\n+ Ideas presented in the work are technically sound.\n+ Presented results outperform competing baselines.\n+ Authors submitted code. Although I did not run it but this is still appreciated. Code will also be released upon acceptance.\n+ Paper is mostly well written. See comments below for minor improvements.\n\nWEAKNESSES:\n[Technical]\n1. How is the proposed Voxel Semantic Generator different than “CLIP-Conditioned Coarse Transformer” from CLIP-Sculptor? They are essentially doing the same thing, 1. Learn a voxel based encoding of 3D shapes and 2. Use a transformer to learn how to unmask the 3D features conditioned on a prompt.\nCan authors clarify this better?\n\n2. L175-178: Since the GT for Grid Smoother is generated using furthest point sampling, why can’t we use FSP at inference time to smoothen the coarse voxel grid from the 3D VQGAN decoder?\nWhy do we need to learn a neural network?\nHow useful is adding KL loss over Chamfer loss?\n\n3. L180: The output of the grid smoother is a point cloud (I assume this because it is trained with a Chamfer loss). L169 mentions that “point tokens” are masked which are then unmasked by the transformer. What exactly are the “point tokens”? Is it just the positional encoding of points?\nFig.2 (b) mentions “semantic tokens” what does this mean? It is not explained in Sec 3.3 (Point Upsampler).\n\n4. Eq. 1: How useful is the occupancy loss over MSE? Is this critical? Please add an ablation study in supp. mat. to support the proposal.\n\n[Minor]\n- Fig. 2: Please add symbols used in the text to Fig.2. This makes it easier to follow the text and map various components of the pipeline.\n- Add dimensionality of each embedding/latent code along with the symbols, eg: C_{pr} \\in \\mathbb{R}^{<whatever>\\times<whatever>}. This significantly improves clarity and readability.\n- L120: How is the mapping performed? How is the codebook generated? If an existing work is used, please add citation here? This is present in the supp. mat. to an extent. Add a pointer so that the reader knowns where to look.\n- L140: Just curious, how important is the scheduling of the mask fraction?\nL147: Is there significant performance difference at inference time between direct and multi-step prediction?\n- L234: Table number is missing.\n- Please avoid violent objects like guns for showing qualitative results. It is understandable that it is sometimes necessary to demonstrate performance but whenever possible, let us try to avoid this.\n\nQUESTIONS:\nOverall I'm positive about the work. There are some concerns about the novelty/necessity of some components (see above) and it would be great if authors can clarify these.\n\nLIMITATIONS:\nDiscussion on Limitations and Broader Impact is not included in the main paper but it is present in the supp. mat. Authors are encouraged to add some discussion on limitations and future work in the main paper as it allows the community to better use and build upon your research.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe VPP proposes a model for 3D generation. It utilizes both point-based and voxel-based representations. Voxel-based representations are used to generate the coarse tokens, and the point-based one further improves the result. Both of which are pretrained with MAE-like self-supervised method.\n\nThe proposed method applies to image-to-point, text-to-point, and point completion. The propsed method is novel and effective. However, some parts are not presented clearly.\n\nSTRENGTHS:\n1. The limitations of existing methods are well analyzed.\n2. The proposed method is carefully designed and performs well.\n3. The proposed method applies to more downstream tasks than exsiting methods.\n4. The experiments show the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The inference efficiency in Table 1 only presents hours or seconds, it is recommended to provide the exact time.\n\nTotally, the paper is not well presented. Some details are not shown clearly. To be specific,\n2. The tokenizer for points upsampler is not described. What is it structure and how to train it?\n3. It would be better to refer to some important symbols in Figure2. For example, prompt embedding in Figure 2 should also be marked as C_pr.\n4. The GAN's structure and loss in 3D VQGAN are not introduced.\n5. As described in Section 3.2, the mask voxel transformer will be forwarded multiple times for better quality. It should also be commented in Figure 2 or 3 for better presence.\n\nSome typos:\n6. In about line 151: \"when the inputs of the mask voxel transformer are grid tokens with high masking fraction and prompt embedding we find that this will lead to the semantics of the predicted grid tokens being much more biased toward the prompt embedding\". I think the authors may miss punctuations.\n\nQUESTIONS:\n1. I am not sure what parallel decoding in Figure 3 means.\n2. When training the voxel semantic transformer, part of voxel tokens are masked, as shown in Figure 2a, and the masking ratio is about 64%. While all tokens are masked voxel tokens during inference. Is there a gap between training and inference?\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a text-driven point cloud generation model that can be used for various downstream tasks such as generation, editing, \n completion and pretraining, while being very efficient. The method largely follows Muse [2], but adapted it to 3D point clouds. The model consists of multiple components, which are trained individually. First, a VQGAN is trained on the voxelized shapes, which embed a shape to a latent grid. Next, a masked transformer is trained on the latent grid to model its distribution, using CLIP feature as semantic conditioning. For the decoding stage, a grid smoother network moves the voxel centers to more homogeneous locations, while finally another set of point cloud VQVAE and masked transformer is trained to upsample the coarse point cloud to high resolution point cloud. The paper also demonstrated that the proposed components can be used in whole or in part to tackle a wide range of tasks.\n\nSTRENGTHS:\n* The proposed method is able to handle a wide range of tasks such as generation, editing, completion and pretraining.\n* VPP significantly outperforms baseline methods on the main task of shape generation.\n* The proposed method is computationally efficient compared to optimization-based methods such as DreamFusion.\n\nWEAKNESSES:\n* The components of the proposed method are not new -- they are mostly borrowed from previous works such as CLIP, MUSE and Point-MAE.\n* The method contained a large number of stages, making it potentially difficult to implement or improve upon.\n* Unlike diffusion-based methods such as DreamFusion, Dream3D, Magic3D, the proposed method does not generate surface or texture. It is also not clear if the proposed method is able to do zero-shot generation like these methods.\n* The proposed method is only trained on ShapeNet, which is relatively small and less diverse compared to larger datasets such as Objaverse. It is not sure if the method is able to scale to larger datasets.\n\nQUESTIONS:\n* What is the benefit of using voxel as intermediate representation compared to directly encoding the point cloud?\n* Missing table reference on L234.\n\nLIMITATIONS:\nLimitations and societal impacts are adequately addressed in the supplemental material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}