VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation
Authors: Zekun Qi
Muzhou Yu
Runpei Dong
Kaisheng Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Text to shape generation
3D shape generation
Efficient inference
Representation Learning
TL;DR: We achieve rapid, multi-category 3D conditional generation by sharing the merits of different representations. VPP can generate 3D shapes less than 0.2s using a single RTX 2080Ti.

Abstract: Conditional 3D generation is undergoing a significant advancement, enabling the free creation of 3D content from inputs such as text or 2D images. However, previous approaches have suffered from low inference efficiency, limited generation categories, and restricted downstream applications. In this work, we revisit the impact of different 3D representations on generation quality and efficiency. We propose a progressive generation method through Voxel-Point Progressive Representation (VPP). VPP leverages structured voxel representation in the proposed Voxel Semantic Generator and the sparsity of unstructured point representation in the Point Upsampler, enabling efficient generation of multi-category objects. VPP can generate high-quality 8K point clouds within 0.2 seconds. Additionally, the masked generation Transformer allows for various 3D downstream tasks, such as generation, editing, completion, and pre-training. Extensive experiments demonstrate that VPP efficiently generates high-fidelity and diverse 3D shapes across different categories, while also exhibiting excellent representation transfer performance. Codes will be released at https://github.com/qizekun/VPP.

VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation
Zekun Qi f
Muzhou Yu
Runpei Dong
Xi Jiaotong University
{q2k139318, muzhou9999, runpei dong) @stu xjtu. edu, en
Kaisheng Ma 9
Tsinghua University
kaisheng@mail.ts tsinghua edu. cn
Bs Sphous
restangie sable
rourd tabie
5 dench
ašrp
square chair
transport
Amaccl car
20
ScanDbjectNN
ModelNet
+0.3%
+4.1%
offics chair
ththost
VPP PainMMA
VPP
PointMAE
(a) Test -onnditional Generation
(b) Pretraining
lega
Image -Conditional Generation
(d) Text Condiitinnal Editing
(e) Upsampling 2
Figure
Overview of VPP's applications through generative modeling.
Abstract
Conditional 3D generation undergoing significant advancement, enabling the free creation of 3D content from inputs such as text or 2D images. However, pre- vious approaches have suffered from low inference efficiency, limited generation categories, and restricted downstream applications. In this work, revisit the impact of different 3D representations on generation quality and efficiency. We propose progressive generation method through Voxel-Point Progressive Repre- sentation (VPP). VPP leverages structured voxel representation in the proposed Voxel Semantic Generator and the sparsity of unstructured point representation in the Point Upsampler, enabling efficient generation of multi-catee catlii- objects. VPP generate high-quality 8K point clouds within 0.2 seconds. Addition- ally, the masked generation Transformer allows for various 3D downstream tasks, such generation, editing, completion, and pre-training. Extensive experiments demonstrate that VPP efficiently generates high-fidelity and diverse 3D shapes across different categories, while also exhibiting excellent representation transfer performance. Codes will be released at https: //gi //gi thub. com/ http//iizeknnnnne
Equal contribution. Prooect lead. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Comparison of text-conditioned 3D generation methods efficiency and applications.
Method
Latency
Device
Generation Category
Downstream Task
DrcamFields [37]
1.2h
8xTPUv4
Multi-Category
Generation
DreamPusion [65]
1.5h
4XTPUv4
Multi- Category
Generation
CLIP-Mesh [58]
30min
V100
Multi-Cutee Culti-Category
Generation
CLIP-Sculpter [77]
0.9s
V100
Dluti-Categoor
Generation
Point-E (40M) [59]
25s
V100
Multi-Category
Generation
ShupeCrafter [24]
Generation, Editing
LION [99]
27s
V100
Single-Category
Generation, Completion
SDFusion [9]
Single-Category
Generation, Completion
VPP (Ours)
0.2s
RTX 20801i
Multi-Category
Generation, Editing, Completion, Pretraining
Introduction
In the last few years, text-conditional image generation [62, 71, 72, 74] has made significant progress in terms of generation quality and diversity and has been successfully applied in games [7], content creation [47, 48] and human-AI collaborative design [36]. The success of text-to-image generation is largely owed to the large-scale image-text paired datasets [78]. Recently, there has been an increasing trend boost conditional generation from 2D images to 3D shapes. Many works attempt to use existing text-3D shape paired datasets [7, 9, 24, 50] to train the models, but the insufficient pairs limit these methods to generate multi-category, diverse, and high-res 3D shapes.
To tackle this challenge, some works [37, 65, 82, 94] use NeRF [56] and weak supervision of large- scale vision-language models [70] generate 3D shapes. Although creative quality is achieved, the optimization costs of differentiable rendering are quite expensive and impractical. Some methods use images as bridges [38, 59] for the text-conditional 3D generation, which improves the quality-time trade-off. However, the complex multi-stage model leads to long inference latency and cumulative bias. Moreover, most methods [58, 76] are task-limited due to specific model design that can not be implemented to more downstream tasks, e.g., editing, leading to narrow application scope. To sum up, current methods are faced with three challenges: low inference efficiency, limited generation category, and restricted downstreum lasks, which have not been solved by unified method.
One of the most crucial components in 3D generation is the geometry representation, and mainstream methods are typically built with voxel, 3D point cloud, or Signed Distance Function (SDF) [11, 61). However, the aforementioned issues in 3D generation may come from the different properties of these representations, which are summarized as follows:
Voxel a structured and explicit representation. It shares a similar form with 2D pixels, facilitating the adoption of various image generation methods. However, the dense nature of voxels results in increased computational resources and time requirements when generating high-resolution shapes.
3D Point Cloud is the unordered collection of points that explicitly represents positional informa- tion with sparse semantics, enabling flexible and efficient shape representation. However, 3D point clouds typically dedicatedly designed architectures, e.g., PointNet [66, 67].
SDF describes shapes through an implicit distance function. is continuous representation method and is capable of generating shapes with more intricate details than points or voxels. But it requires substantial computation for reghuresolution sampling and is sensitive to initialization.
Due to significant shape differences across various categories, the structured and explicit positional information provides spatial cues, thereby aiding the generalization of multiple categories. It coincides with previous multi-category generation methods [76, 77]. In contrast, single-category generation may require less variation. Hence an implicit and continuous representation would contribute to the precise expression of local details [9, 25].
Motivated by the analysis above, we aim to leverage the advantage of different forms of representation. Therefore, a novel Voxel-Point Progressive (VPP Representation method is proposed to achieve efficient and universal 3D generation. We use voxel and point as representations for the coarse fine stages to adapt to the characteristics of 3D. To further enhance the inference speed, we employ mask generative Transformers for parallel inference [5] in both stages, which can concurrently obtain broad applicability scope [42]. Based on voxel VQGAN [21], we utilize the discrete semantic codebooks as the reconstruction targets to obtain authentic and rich semantics. Given the substantial computational resources required by high-resolution voxels, we employ sparse points representations in the

---

input voxel
rec voxel
rec voxel (with foce)
grid points
smoothed points
grid points
smoothed points
(a) Reconstruction Results of 3D VQGAN
(b) Smoothing performance of Grid Smoother
Figure 2: Qualitative examples. (a) 3D VQGAN reconstruction results with without Locc (Section 2.1). (b) 3D point clouds result with or without proposed Grid Smoother (Section 2.2).
second stage. Inspired by the masked point modeling approaches based on position cues [60, 98], we utilize the voxels generated in the first stage the positional encoding for generative modeling. We reconstruct the points and predict semantic tokens to obtain sparse representations. Table compares the efficiency and universality of text-conditional 3D generation methods.
In summary, our contributions are: (I) We propose novel voxel-point progressive generation method that shares the merits of different geometric representations, enabling efficient and multi-category 3D generation. Notably, VPP capable of generating high-quality 8K point clouds within 0.2 seconds on single RTX 2080Ti. (2) To accommodate 3D generation, we implement unique module designs, e.g. 3D VQGAN, and propose Central Radiative Temperature Schedule strategy in inference. (3) As universal method, VPP complete multiple tasks such editing, completion, and even pre- pre-training. To the best of our knowledge, VPP is the first work that unifies various 3D tasks.
2
VPP
Our goal is to design a 3D generation method by sharing the merits of both geometric representations, which is capable of addressing the three major issues encountered in previous generation methods. In this work, we propose VPP, a Voxel-Point Progressive Representation method for two-stage 3D) generation. To tackle the problem of limited generation categories, VPP employs explicit voxels and discrete VQGAN for semantics representation in the first stage. To address the issue of low inference efficiency, VPP employs efficient point clouds representations in the second stage and leverages the parallel generation capability of the mask Transformer. Given the wide range of applications of generative modeling, VPP resolves the problem of restricted downstream tasks. The training overview of VPP is shown in Fig. 3.
2.1 How Does 2D VQGAN Adapt to 3D Voxels?
Due to the formal similarity between voxels and pixels, we can readily draw upon various image generation methods. VQGAN [21] is capable of generating realistic images through discrete semantic codebooks and seamlessly integrating with mask generative Transformers [5, 6]. We utilize 3D volumetric SE-ResNet [35] the backbone of VQGAN encoder E, decoder G and discriminator D.
Given one voxel observation E RR3 whose occupancy defined as function o TR3 [0, 1). The original VQGAN is trained by optimizing a vectorized quantized codebook with the loss Lvo during autoencoding while training discriminator between real and reconstructed voxels with loss LCAN- Denoting the reconstructed voxel result as v, the overall loss of vanilla VQGAN can be written as:
1|| ||
[|sg[F[(v] [v))
=q =1 [|sg[=q] F[()]13 + [log D(v) + log(1 D(V))
Lux((1..G.2).D)
However, unlike 2D images, 3D voxels only have non-zero values in the region that encompasses the object in grid centroids. This may lead a short-cut solution that degrades by outputting voxels with values as close to zero as possible, i.e., all empty occupancy. To address this issue, propose to

---

Seq-Invariance Loes
3D VOGAN Encoder
Cross Entropy Loes
Grid Smoolter
Mask Point Transformer
oooo
Decoder
Prompt Embedding
Voreliztionn
mask
mask
CD Loss
Loss
Mesk Voxel Transforme
Tokenizer
Input
Reconstructed Tokens
Point Clouds
Reconstructed
Clouds
(a) Voxel Semantic Generator
Point Upsampler
Figure 3: Training process overview of VPP. (a) In the coarse stage, the VQ codebook is reconstructed via Mask Voxel Transformer conditioned on prompt embeddings. (b) In the fine stage, sparse semantic tokens are reconstructed via the Mask Point Transformer conditioned on positional embeddings.
minimize the occupancy disparity between the reconstructed and ground truth voxels. Formally, let Ç (v) the averaged occupancy SE(3) space, the overall loss L3D VOGAN is:
LID-VQGAN Lvo + LGAN + o(v) - -
Loce
where Locc. is proposed to minimize the occupancy rate for better geometry preservation.
2.2 How to Bridge the Representation Gap Between Voxels and Points?
We aim to utilize the voxel representation the intermediate coarse generation before fine-grained point cloud results. However, voxels are uniform and discrete grid data, while point clouds are unstructured with unlimited resolutions, which allows for higher density in complex structures. Besides, point clouds typically model the surfaces, while voxel representation is solid nature. This inevitably leads to representation gap issue, and smooth transformation that bridges the representation gap is needed. We employ the Lewiner algorithm [40] to extract the surface of the object based voxel boundary values and utilize lightweight requence-invariant Transformer [84] network Tr parametrized with as Grid Smoother to address the issue of grid discontinuity. Specifically, given generated point clouds p = {p: E R with N points. The model trained by minimizing the geometry distance between generated and ground truth point clouds & = {gi E )i=1 that is downsampled N with farthest point sampling (FPS). To make the generated surface points more uniformly distributed, we propose Luniform that utilizes the Kullback-Leibe between the averaged distance of every centroid p, and its K-Nearest Neighborhood (KNN) N,. Then we optimize the parameter of Grid Smoother by minimizing loss LsMO:
(KL) divergence DKL
Lsmo
Σ
min p
min gg Pll DKL pET,(P)
Σ llp. JEN;
Py 12, u
(P)(P)
SEg
(ET,(P)
geg
Cen
Lustoon
(2)
where LCD is & Chamfer Distance [23] geometry disparity loss, 24 is the uniform distribution.
2.3 Voxel-Point Progressive Generation Trough Mask Generative Transformer
We employ mask generative Transformers [5] for rich semantic learning in both the voxel and point stages. Mask generative Transformer can seen as special form of denoising autoencoder [34, 85). It forces the model to reconstruct masked input learn the statistical correlations between local patches. Furthermore, generating reconstruction targets through powerful tokenizer can further enhance performance [2, 19, 91, 104]. From the perspective of discrete variational autoencoder

---

Prompl
Embedding
CLIP
Position Embedding
Image
OOO0 %
Mask Voxel Transformes
30 VOGAN Decoder
Grid Smoolher
Mask Paint
Transfomeer
Decordr
Masked
Reconstructed
Generated Coarse Voxel
Masked
Reconstructed
Generated Clouds
Voxel Tokens
Voxel Tokens
Point
Parallel Decoding
Parallel Decoding
Figure 4: Inference process overview of VPP. Point clouds are generated conditioned the input text or images. Parallel decoding is adopted for efficient Mask Transformer token prediction.
(dVAE) [2, 26, 39, 71], the overall optimization is to maximize the evidence lower bound (ELBO) [17, 32, 73] of the log-likelihood P( (ā;)) Let æ denote the original data, I the masked data, and the semantic tokens, the generative modeling can be described
Σ
log P(o+(T;)
(Bar (EerQu [lo P * [log (R...jjD
Σ
DRL. (Ro.lloal)s)))
(3)
(ED
where (1) (=/2z) denotes the discrete semantic tokenizer, (2) P P (z|z) is the tokenizer decoder to recover origin data; (3) 2 (=(=)) denotes the masked semantic tokens from masked data; (4) Po(2|²) reconstructs masked semantic tokens in autoencoding way. In the following sections, we extend the mask generative Transformer to voxel and point representations, respectively. The semantic tokens of voxel and point are described 2" and z in Figs. 3 and 4.
Voxel Semantic Generator We generate semantic voxels with low resolution in the first stage to balance efficiency and fidelity. We utilize Mask Voxel Transformer for masked voxel inputs, where the 3D VQGAN proposed in Section 2.1 is used as the tokenizer to generate discrete semantic tokens. Following prior arts [5, 6], adopt Cosine Scheduling to simulate the different stages of generation. The masking ratio is sampled by the truncated arccos distribution with density function
p(r) 21 - ( 1. The prompt embedding comes from the CLIP [70] features of cross-modal information. We render multi-view images from the 3D object [68] and utilize BLIP [41] get the text descriptions of the rendered images. Furthermore, in order to mitigate the issue of excessive dependence on prompts caused by the high masking ratio (e.g., the same results will be generated by one prompt), we introduce Classifier Free Guidance [33] (CFG) to strike balance between diversity and quality. Following Sanghi al. [77], the prompt embeddings are added with Gaussian noise to reduce the degree of confidence. We use ~ N0. 1) and E (0, 1) as the level of noise perturbation for the trade-off control, which will later be studied.
Point Upsampler Point clouds are unordered and cannot be divided into regular grids like pixels or voxels. Typically, previous masked generative methods [19, 60) utilize FPS to determine the center of local patches, followed by KNN to obtain neighboring points as the geometric structure. Finally, a lightweight PointNet [66, 67] or DGCNN [89] is utilized for patch embedding. However, these methods rely pre-generated Positional Encodings (PE), rendering the use of fixed positional cues infeasīble conditional 3D generation since only conditions like texts are given. To tackle this issue, adopt the first-stage generated voxels-smoothed point clouds the PEs. A pretrained Point-MAE [60] encoder is used as the tokenizer (i.e., teacher [19]). The Mask Point Transformer learns to reconstruct tokenizer features, which will be used for the pretrained decoder reconstruction.
2.4 Inference Stage
Parallel Decoding An essential factor contributing to the efficient generation of VPP the parallel generation during the inference stage. The model takes fully masked voxel tokens as input and gener- ates semantic voxels conditioned on the CL.PP [70] features of text or image prompts. Following [5. 6), cosine schedule is employed to select the fixed fraction of the highest confidence masked tokens for
5

---

CLIP-Forge
CLIP- CLI--Sculplor
VPP (Ours)
VPP (Ours)
Lorp
reclongslor table
jeb
on office
machimee gus
Figure 5: Qualitative comparison of VPP. CLIP-Sculppi [77] and CLiP-Forge [76] on text condi- tioned 3D generation. VPP generates shapes of higher fidelity with smoother surfaces while being consistent with the input text. More diverse results are shown Appendix
prediction each step. Furthermore, we pass the generated voxels through Grid Smoother to obtain smoothed point clouds, which will serve as the positional encoding for the Point Upsampler in the upsampling process. Note that all the stages are performed in parallel. The inference overview of VPP is illustrated in Fig. 4.
Central Radiative Temperature Schedule Due the fact that 3D shape voxels only have non-zero values in the region that encompasses the object in the center, we hope to encourage diversity in central voxels while suppressing diversity in edge vacant voxels. Consequently, we propose the Central Radiative Temperature Schedule to accommodate the characteristics of 3D generation. As for the voxel with a size of R x x R, we calculate the distance r to the voxel center for each predicted grid. Then, we set the temperature for each grid as T = 1 (r/R}² where T represents the temperature. Thus, we can achieve the grid that is closer to the center will have a higher temperature to encourage more diversity and vice versa.
Experiments
3.1 Conditional 3D Generation
Following [59, 76, 77], we use Accuracy (ACC), Fréchet Inception Distance (FID) [31], and Inception Score (IS) [75] as metries to assess the generation quality and diversity. We follow [76] to define 234 predetermined text queries and apply classifier for evaluation. For fair comparison, all methods use "a/an" as the prefix for text prompts. Similar to [59], we employ double-width PointNet++ model [67] to extract features and predict class probabilities for point clouds in ShapetNet [4]. Table 2 shows the comparison between VPP and other methods. can be observed that our VPP outperforms all other methods substantially in all metrics. Notably, the FID metric of ours is much lower than other methods, demonstrating that our method generates higher quality and richer diversity of 3D shapes.
Text-conditioned Generation
By employing Shape As Points [52, 63] model pre-trained on multi-category ShapeNet data, VPP is capable of performing smooth surface reconstruction on generated point clouds. In Blender rendering, we employ varnish and automatic smoothing techniques improve the display quality of the generated mesh. We qualitatively show in Fig. 5 that our method can generate higher-quality shapes and remain faithful to text-shape correspondence. It is noteworthy that VPP can generate smoother 3D shapes, egg., the truck. Besides, our generation results are visually pleasing across different object categories with complex text descriptions. Therefore, the above advantages enable method to be much more useful in practical applications.

---

Figure 6: Qualitative examples of single image conditioned 3D generation with VPP.
Table 2: Text conditioned 3D generation results ShapeNerCCree33 [4] dataset. Point-E [59] is pre-trained on a large-scale private point-texi paired dataset containing millions of samples, and we report its zero-shot transfer results ShapeNet. Accuracy (ACC), Fréchet Inception Distance (FID), and Inception Score (IS) are reported.
Method
ACC1
FID
1St
CLIP-Forge [76]
83.33%
2425.25
CLIP-Sculptor [77]
87.08%
1480. 11
Point-E (40M text-only) [59]
62.56%
85.37
7.93
VPP (ours)
88.04%
29.82
10.64
Image-Conditioned Generation By leveraging the CLIP features of 2D images as conditions in the Voxel Semantic Generator, VPP also possesses the capability of single-vier reconstruction. Fig. illustrates condeicondiiional generated samples of VPP. It can be observed that the generated point clouds are of detailed quality and visually consistent with the input image. For example, the flat wings of an airplane and the curved backrest of chair.
3.2 Text-Conditioned 3D Editing and Upsampling
Text-conditioned 3D editing and upsample completion tasks are more challenging than most condi- tional 3D generation tasks, which most of previous methods can not realize due to the inflexible model nature. In contrast, our VPP can be used for variety of shape editing and upsample completion without extra training or model fine-tuning.
Text-Conditioned 3D Editing Since we use the masked generative Transformer VPP, our model be conditioned any set of point tokens: first obtain of input point tokens, then mask the tokens corresponding to a local area, and decode the masked tokens with unmasked tokens and text prompts conditions. We show the examples in Fig. (a). The figure provides examples of VPP being capable of changing part attributes (view) and modifying local details (view) of the shape, which correctly corresponds the text prompts.
Point Cloud Upsampling Except for editing, we also present our conditional generative model for the point cloud upsample completion, where the sparse point cloud inputs are completed as dense output. We the second stage Point Upsampler achieve this, and the results are illustrated in Fig. 7 (b). It is observed that VPP generates completed and rich shapes of high fidelity while being consistent with the input sparse geometry.
3.3 Transfer Learning on Downstream Tasks
After training, the Mask Transformer has learned powerful geometric knowledge through generative reconstruction [30], which enables VPP to serve as a self-supervised learning method for downstream representation transferring. Following previous works [19, 60, 68), we fine-tune the Mask Point Transformer encoder, i.e., the second-stage Point Upsampler for 3D shape recognition.

---

with tuo legs
square chavr
cursed
viega with engines
(a) Text-Conditlonal Editing
(b) Point cloud Upsampling
Figure 7:
Qualitative examples of text conditioned 3D editing and upsample completion with VPP.
Table 3: Downstream 3D object classification results on the ScanObjectNN and ModelNet40 datasets.
The inference model parameters (M), FLOPS #F (G), and overall accuracy (%) are reported.
ScanObjectNN
ModelNet40
Method
#P
#F
OBJ BG
OBJ ONLY
PB TS0 RS
1k P
8k P
Supervised Learning Only
PointNet [66]
3.5
0.5
73.3
79.2
68.0
89.2
90.8
PointNet++ [67]
1.5
1.7
82.3
84.3
77.9
90.7
91.9
DGCNN [89]
1.8
2.4
82.8
86.2
78.1
92.9
PointCNN [43]
0.6
86.1
85.5
78.5
92.2
PCT [27]
2.88
2.3
93.2
PointMLP [53]
12.6
31.4
85.4+#0.3
94.5
PointNeXt [69]
1.4
3.6
87.7=0.4
94.0
with reff-Supervieed Representation Learning
Transformer [84]
22.1
4.8
83.04
84.06
79.11
91.4
91.8
PoinI-BERT [98]
22.1
4.8
87.43
88.12
83.07
93.2
93.8
Point-MAF [60]
22.1
4.8
90.02
88.29
85.18
93.8
94.0
Point-M2AE [102]
14.8
3.6
91.22
88.81
86.43
94.0
VPP w/o vot.
22.1
4.8
92.77
91.56
88.65
93.8
94.0
VPP w/ vot.
22.1
4.8
93.11
91.91
89.28
94.1
94.3
with Pretrained Cross Modss Teacher Representation Learning
ACT [19]
22.1
4.8
93.29
91.91
88.21
93.7
94.0
12P-MAE [103]
12.9
3.6
94.15
91.57
90.11
94.1
ReCon [68]
43.6
5.3
95.18
93.29
90.63
94.5
94.7
ScanObjectNN [83] and ModelNet [92] are currently the most challenging 3D object datasets, which are obtained through real-world sampling and synthesis, respectively. We show the evaluation of 3D shape classification in Table It can seen that: (i) Compared to any supervised or self- supervised method that only uses point clouds for training, VPP achieves the best generalization performance. (ii) Notably, VPP outperforms Point-MAE by +4.0% and +0.3% accuracy on the most challenging PB_T50 R50_55_RRRA _S and ModelNet40 benchmark.
3.4 Diversity & Specific Text-Conditional Generation
We show the diverse qualitative results of VPP on text-conditional 3D generation in Fig. (a). It can be observed that VPP can generate broad category of shapes with rich diversity while remaining faithful to the provided text descriptions. Meanwhile, we present the qualitative results of VPP on more specific text-conditional 3D generation in Fig. 8 (b). Notably, VPP can generate high-fidelity shapes that react well to very specific text descriptions, like cup chair, a round table with four legs and aircraft during World War II, etc.

---

ak47
cup chair
dressing table
s square chair round tad le wi th four legs
sas boat
a round table ws five legs
cargo van
aircraft during World Far II
atrp
porsche 911
a bench
tams
car
flat jet
(a) Diversity Text-Conditional Generation
(b) Specific Text-Conditional Generation
Figure 8: (a) Diversity qualitative results VPP on text conditioned 3D generation. (b) Qualitative results of VPP on more specific text-conditioned 3D generation. VPP can generate a broad category of shapes with rich diversity and high fidelity while remaining faithful the provided text descriptions. Besides, VPP can react to very specific text descriptions, like a cup chair.
origin points
partial points
generated points
origin points
partial points
generated points
Figure 9: Partial inputs point clouds completion results of VPP. Our model is capable of generating diverse completion samples.
3.5 Partial Completion
present the point cloud partial completion experiments in Fig. 9. By employing a block mask on the original point clouds, VPP can generate partial samples, which illustrates the partial completion capability of VPP, Besides, the generated samples exhibit diversity, further demonstrating the diverse performance of VPP.

---

round table
Retrieved Samples from ShapeNet
office cheir
Retrieved Samples from ShapeNet
Figure 10: ShapeNet training data retrieval ablation experiment.
3.6 ShapeNet Retrieval Experiment
We conduct the retrieval evaluation samples generated by VPP from the ShapeNet dataset. The results are shown in Fig. 10. can be observed that there are no completely identical samples, proving the great generation ability of VPP is not caused by overfitting the ShapeNet dataset. The VPP results more likely come from the understanding and integration of the learned shape knowledge.
4 Related Works
Conditional 3D generation with 2D images or text has witnessed rapid development in recent years. One line of works focuses on solving the ill-posed image-conditioned 3D generation (i.e., single-view reconstruction), where impressive results have been achieved [10, 23, 57, 88]. By introducing text-shape paired dataset based on ShapeNet [4] that includes chair and table categories, [7] pioneered another line of category text-conditioned shape generation. Facilitated with this numan-crawled 3D data, ShapeCrafter [24] further enables exx-coonitiooed shape edition in a recursive fashion. To leverage easily rendered 2D images rather than human- crrafted languages, several works propose to train rendered image-to-shape generation [9, 51, 76, 77] which enables text-conditioned generation through pretrained large vision-language (VI.) models, i.e., CLIP [70]. Besides rendered images, Point-E [59] and Shape-F [38] propose to use real-world text-image-shape triplets from a large-scale in-house dataset, where images are used as the representation bridge.
However, 3D data is significantly lacking and expensive to collect [19]. DreamFields [37] first achieves text-only training by using NeRF [56], where rendered 2D images are weakly supervised to align with text inputs using CLIP. Following this direction, DreamFusion [65] incorporates the distillation loss of diffusion model [74]. Dream3D [94] improves the knowledge prior by providing the prompt shapes initialization. Though impressive results are obtained, these approaches also introduce significant time costs because of the case-by-case NeRF training during inference. More recently, TAPS3D aligns rendered images and text prompts based on DM'er [79] to enable text-conditional generation, but its generalizability is limited to only four categories. 3DGen [28] achieves high-quality generation through the utilization of Triplane-basee diffusion models [3, 80] and the Objaverse dataset [14]. Zero- --o-3 [46] employs relative camera viewpoints as conditions for diffusion modeling. Based on this, One-2-3-45 [45] achieves rapid open- neen-voaabulary mesh generation through mue synthesis. Fantasia3D [8] leverages DMTet to employ differentiable rendering of SDF substitute for NeRF, enabling more direct mesh generation. SJC [87] and PromificDreamer [90] achieve astonishing results by proposing the variational diffusion distillation.
Conclusions
In this paper, we present VPP, novel Voxel-Poine Progressive Representation method that achieves efficient and universal conditional 3D generation. We explore the suitability of different 3D geometric representations and design VQGAN module tailored for generation. Notably, VPP capable of generating high-quality 8K point clouds in less than 0.2 seconds using single RTX 2080Ti GPU. Extensive experiments demonstrate that VPP exhibits excellent performance conditional generation, editing, completion, and pretraining.
10

---

Acknowledgments
This research was supported by the National Key R&D Program of China (2022YFB2844003) the Key Research and Development Program of Shaanxi (2021ZDLGY01--0)) the National Natural Science Foundation of China (31970972), and the Institute for Interdisciplinary Information Core Technology (IIISCT).
References
[1] Tro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Toannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In /EEF/CVF Conf. Compua. Vis. Pattern Recog. (CVPR), 2016.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In Int. Conf. Learn. Reprexent, (ICLR). OpenReview.net, 2022. 5,5
[3] Eric R. Chan, Connor Z. Lin, Mathew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay. Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-awe aware 3d generative adversarial networks. In IEEE/CVF Conf. Comput. Vis. Pattern
Recog. (CVPR), 2022. 10
[4] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR, abs/1512.0300
2015. 6, 7, 10, 17
[5] Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In /EEF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022. 4,5
[6] Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Int. Conf. Mach. Learn. (ICML), 2023.
3,5 5
[7] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang. Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian Conf. Comput. Vis. (ACCV), 2, 10
[8] Rui Chen, Yongwei Chen, Ningxin Jiao, Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Int. Conf. Comput. Vis. (ICCV), 2023. 10
[9] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G. Schwing. and Liangyan Gui. Sdfusion: Multimodal 3d shape completion. reconstruction, and generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 2, 10
[10] Christopher B. Choy, Danfei Xu, Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Eur. Conf. Comput. Vis. (ECCV), 2016. 10
[11] Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, New Orleans, LA, USA, August 4-9, 1996, pp. 303-312, ACM, 1996. 2
[12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2017.
[13] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. CoRR, abs/2307. 05666665663.... 2023. 20
[14] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt. Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In IEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), 2023. 10, 20
[15] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-can: Towards high performance voxel-basee 3d object detection. In AAAI Conf. Artif. Intell. (AAAI), 2021. 17

---

[16] Runyu Ding, Jihan Yang, Chuhui Xue. Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language-driven open-vocabulary 3d scene understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), 2023.
17
[17] Runpei Dong Zhanhong Tan, Mengdi Wu, Linfeng Zhang, and Kaisheng Ma. Finding the task aak-ooptimal low-bit sub-distribution deep neural networks. In Int. Conf. Mach. Learn. (ICMI.), 2022.
[18] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal comprehension and creation. CoRR, abs/2309. 11499, 2023. 20
[19] Runpei Dong. Zekun Qi. Linfeng Zhang. Junbo Zhang. Jianjian Sun, Zheng Ge. Li Yi, and Kaisheng Ma. Autoencoders cross-modal teachers: Can pretrained 2d image transformers help 3d representation learnin?? In Int. Conf. Learn. Represent. (ICLR), 2023. 4, 8, 10, 17,
[20] Nico Engel. Vasileios Belagiannis, and Klaus Dietmayer. Point transformer. IEEE Access, 9:134826- 134840, 2021. 17
[21] Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In /EEE/CVF Conf. Comput. Vis. Pottern Recog. (CVPR), 2021. 2,3, 18
[22] Guofan Fan, Zekun Qi. Wenkai Shi, and Kaisheng Ma. Point-gec: Universal self-supervised 3d scene pre-training via geometry-color contrast. CoRR, abs/2305. 19623, 2023. 17
[23] Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object recon- struction from a single image. In IEEE/CVF Conf. Comput. Vis. Puttern Recug. (CVPR), 2017. 4, 10
[24] Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. Shapecrafter: A recursive text- conditioned 3d shape generation model. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 2,
[25] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin, Daiqing Li, Litany, Zan Gojcic, and Sanja Fidler. GeGd: A generative model of high quality 3d textured shapes learned from images. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 2
[26] Alex Graves. Practical variational inference for neural networks. In John Shawe-Taylor, Richard S. 7emel, Peter I.. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger (eds.), Adv. Neural Inform. Process.
Syst. (NIPS), 2011. 5
[27] Meng-Hao Guo, Junxiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. PCT: point cloud transformer. Comput. Vis. Media, 7(2):187 199, 2021. 8
[28] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. CORR, abs/2303.05s 00371, 2023, 10
[29] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. MVTN: multi-view transformation network for
3d shape recognition. In Int. Conf. Comput. Vis. (ICCV), 2021. 17
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR). 2022.
7
[31] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Adv. Neural Inform. Process. Syst, (NIPS), 2017.
[32] Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In ACM Conf. Comput. Learn. Theory (COLT), 1993. 5
[33] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 5
[34] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Adv. Neurul
Inform. Process. Syst. (NeurIPS). 2020. 4
[35] Jie Hu, Li Shen, and Gang Sun. Squeeze-and areezze-and--exee and-excitation networks. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2018. 3
12

---

[36] Rowan T Hughes, Liming Zhu, and Tomasz Bednarz. Generative adversarial setworks-enablee human- artificial intelligence collaborative applications for creative and design industries: A systematic review of current approaches and trends. Frontiers in artificial intelligence, 4:604234, 2021.
[37] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zeno-shot text-guided object generation with dream fields. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022. 2, 10, 20
[38] Heewoo Jun and Alex Nichol.
Shap-e: Generating conditional 3d implicit functions.
CoRR.
abs/2305.02463. 2023. 2, 10
[39] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Int. Conf. Learn. Represent. (ICLR), 2014. 5
[40] Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira, and Geovan Tavares. Efficient implementation of marching cubes' cases with topological guarantees. Journal of graphics tools, 8(2):1115, 2003. 4
[41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping hunguage-imag pre- -r-iniiiing for unified vision- languaee understanding and generation. In Int. Conf. Mach. Learn. (ICML). 2022. 5, 17
[42] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 2
[43] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on X- x-transformed points. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2018. 8
[44] Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning on point clouds. In Eur. Conf. Comput. Vis. (ECCV), 2022. 19
[45] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3- 45: Any single image to 3d mesh in 45 seconds without per-shape optimization. CoRR, abs/2306. 16928, 2023. 10
[46] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-tor Zero-shot one image to 3d object. In Int. Conf. Comput. Vis. (ICCV), 2023, 10
[47] Vivian Liu, Han Qiao, and Lydia Chilton. Opal: Multimodal image generation for news illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1-17, 2022. 2
[48] Vivian Liu, Jo Vermeulen, George W. Fitzmaurice, and Justin Matejka. 3dall-e: Integrating text-to-image Al in 3d design workflows. CoRR, abs/2210.1 1160200,,.1,,,,,, 2022. 2
[49] Ze Liu, Zheng Zhang, Yue Cao. Han Hu, and Xin Tong. Group free object detection via transformers. In Int. Conf. Comput. Vis. (ICCV), 17
[50] Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi- Wing Fu. Towards implicit text- text-guided 3d shape generation. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022. 2
[51] Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi Wing Fu. Iss: Image stetting stone for text-guided 3d shape generation. In Int. Conf. Learn Represent. (ICLR), 2023. 10
[52] Zhaoyang Lyu, Jinyi Wang, Yuwei An, Ya Zhang, Dahua Lin, and Bo Dai. Controllable mesh generation through sparse latent point diffusion models. In IEEF/VVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023.
[53] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual MLP framework. In Int. Conf. Learn. Represent. (ICLR), 2022.
[54] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In Int. Conf. Comput. Vis. (ICCV), 2021. 17
[55] Daniel Maturana and Schastian A. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IEFF/RSJ Int. Conf. Intell. Robot. and Syst. (IROS), 2015. 17
[56] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021. 2, 10
13

---

[57] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In JEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR). 2022.
[58] Nasir Mohammad Khalid, Tianhao Xie, Fugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-8, 2022. 2
[59] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. CoRR, ahs/2212.08851,, 2022. 2, 6, 7, 10, 17
[60] Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning, In Eun. Conf. Comput. Vis. (ECCV), 2022. 3,5,
8, 17, 19
[61] Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE/CVF Conf. Compus. Vis. Pattern Recog. (CVPR), 2019. 2
[62] Or Patashnik. Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Int. Conf. Comput. Vis. (ICCV), 2021.
[63] Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Mare Pollefeys, and Andreas Geiger. Shape points: A differentiable poisson solver. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2021. 6
[64] Songyou Peng, Kyle Genova, Chiyu "Max" " Jiang. Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 17
[65] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In Int. Conf. Learn Represent. (ICLR), 2023. 2, 10, 20
[66] Charles Ruizhongtai Qi. Hao Su, Kaichun Mo. and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In /EEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2017. 2, 2, 5, 17
[67] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Adv. Neural Inform. Process. Syst. (NIPS). 2017. 2, 5, 6, 8, 17
[68] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In Int. Conf. Mach.
Learn. (ICML), 2023. 5, 7, 8, 17,
[69] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Abed Al Kader Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 8, 17
[70] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn. (ICML), 2021.
5, 10, 17
[71] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Int. Conf. Mach. Learn. (ICML), 2021. 2,5
[72] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical exxtcconditinnal image generation with CLIP latents. CoRR, 2ttp/2204.01125... 2022. 2
[73] Jorma Rissanen. Modeling by shortest data description. Autom., 14(5):465-471, 1978.
[74] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), 2022. 2, 10
[75] Tim Salimans, Ian J. Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Adv. Neural Inform. Process. Syst. (NIPS), 2016. 6
[76] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Ye Yi Cheng, Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape shape generation. In /EEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022. 2, 6, 7, 10
14

---

[77] Aditya Sanghi. Rao Fu, Vivian Liu, Karl Willis, Hooman Shayani. Amir Hosein Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 2, 6,
[78] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400:: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/21 11222111.02222 11.0114,, 2021.
[79] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2021. 10
[80] J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 10
[81] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Int. Conf. Comput. Vis. (ICCV), 2015. 17
[82] Junshu Tang. Tengfei Wang. Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it it- High-fidelity 3d creation from A single image with diffusion prior. In Int. Conf. Comput. Vis. (ICCV), 2023. 2
[83] Mikaela Angelina Uy. Quang- Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2019. 8, 17
[84] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst. (NIPS), 2017. 4, 8, 17, 19
[85] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and compos- ing robust features with denoising autocncoders. In Int. Conf. Mach. Learn. (ICMI., 2008. 4
[86] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt Kusner. Unsupervised point cloud pre-training via occlusion completion. In Int. Conf. Comput. Vis. (ICCV), 2021. 19
[87] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffuusio models for 3d generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 10, 20
[88] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating mesh models from single RGB images. Eur. Conf. Comput. Vis. (ECCV), 2018. 10
[89] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph., 38(5):146:1 146:12, 2019, 5, 8. 19
[90] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Ady. Neural Inform. Process. Syst. (NewrIPS), 2023. 10
[91] Longhui Wei, Lingxi Xie. Wengang Zhou, Houqiang Li, and Qi Tian. MVP: multimodality guided visual pre-training. In Eur. Conf. Comput. Vis. (ECCV), 2022. 4
[92] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2015. 8, 17, 19
[93] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Lconidas J. Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Eur. Conf. Comput. Vis. (ECCV), 2020. 17
[94] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 2, 10
[95] Le Xue, Mingfei Gao, Chen Xing, Roberto los Niebles, and Silvio Savarese. ULIP: learning unified representation of language, image and point cloud for 3d understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 17
Jiajun Wu, Caiming Xiong, Ran Xu, Juan Car-
15

---

[96] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang. Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Gruph., 15,6) 2016.
[97] Li Yi, Hao Su, Xingwen Guo, and Leonidas J. Guibas. Synespecenn: Synchronized spectral CNN for 3d shape segmentation. In IEEE/CVE Conf. Comput. Vis. Pattern Recog. (CVPR). 2017.
[98] Xumin Yu, Lalu Tang, Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In IEEE/CVF Conf. Compus. Vis. Pattern Recog. (CVPR), 2022. 3, 8, 17, 19
[99] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic. Litany, Sanja Fidler, and Karsten Kreis. LION: latent point diffussin models for 3d shape generation. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 2
[100] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representa- tions from 2d dense clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 2048-2059, October 2023. 17
[101]
Junbo Zhang, Guofan Fan, Guanghan Wang. Zhengyuan Su, Kaisheng Ma, and Li Yi. Language-assisse assisted 3d feature learning for semantic scene understanding. In AAAI Conf. Artif. Intell. (AAAI), 2023. 17
[102] Renrui Zhang. Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2AE: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 8,
[103] Renrui Zhang. Liuhui Wang. Yu Qiao. Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-polne to-point masked autoencoders. In IEEE/CVF Conf. Comput. Vis. Puttern Recog. (CVPR), 2023. 8,
[104] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Kong. ibot: Image BERT pre training with online tokenizer. In Int. Conf. Learn, Represent, (ICLR), 2022, 4
16

---

A Additional Related Work
3D Representation Learning includes point-based [66, 67], voxel-based [55], and multiview-basee methods [29, 81], etc. Due to the sparse but
representation, point-based methods [20, 69] have become mainstream approaches in object classification [83, 92]. Voxel-based CNN methods [15, 97] provide dense representation and translation invariance, achieving outstanding performance in object detection [12] and segmentation [1, 96]. Furthermore, due to the vigorous development of attention mechanisms [84], 3D Transformers [20, 49, 54] have also brought about effective representations for downstream tasks. Recently, 3D self-supervised representation learning has been widely studied. PointContrast [93] leverages contrastive learning across different views to acquire discriminative 3D scene representations. Point-BERT [98] and Point-MAE [60] first introduce masked modeling pretraining into 3D. ACT [19] pioneers rross-modal geometry understanding via 2D/aagguaee foundation models. RECON [68] further proposes to unify generative and contrastive learning. Facilitated by foundation vision-language models like CLIP [70], another line of works are proposed towards open-world 3D representation learning [16, 22, 64, 95, 100, 101].
B Implementation Details
B.1 Experimental Details
Training Details We use ShapeNerCore from ShapeNet [4] as the training dataset. ShapeNet is a clean set of 3D CAD models with rich annotations, including 51K unique 3D models from 55 common object categories. For the acquisition of multi-modal data, we follow ReCon [68] for multi-view rendering and utilize BLIP [41] based on rendered images obtain textual data. Table shows the training hyperparameters and model architecture information of each part of VPP.
Table Training recipes for 3D VQGAN, Voxel Generator, Grid Smoother and Point Upsampler.
Config
3D VQGAN
Voxel Generator
Grid Smoother
Point Upsampler
Training Parameters
Optimizer
Adam
AdamW
A
A
Learning rate
lc-4
1c-3
1c-3
1e-3
Weight decay
le-4
5e-2
Se-2
5e-2
Training epochs
100
100
100
300
Warmup epochs
10
Learning rate scheduler
cosine
cosine
cosine
cosine
Batch size
32
128
128
128
Drop path rate
0.1
0.1
0.1
Input point size
8192
8192
8192
1024
Model Architecture
Backbone
CNN
Transformer
Transformer 64
Transformer
Layers
12
Hidden size
256
256
384
Heads
Voxel resolution
24/32
24/32
24/32
24/32
Point patch size
32
GPU device
NVIDIA A100
NVIDIA A100
NVIDIA A100
NVIDIA A100
Downstream Tasks Details Following Point-E [59], we use pre-trained PointNet++ classifier in all ACC, FID, and IS evaluations to extract the features and calculate the accuracy of generated point clouds. In point cloud generation and editing, we employ 8 or 4 steps for a parallel generation. The generation task utilizes initial voxel codebooks composed entirely of [MASK] tokens. In editing, we extract VQGAN features from the original point cloud to initialize the voxel codebooks. As for the transfer classification task on ScanObjectNN [83] and ModeINet40 [92], we fully follow the previous work [19, 60) configuration and trained 300 epochs with the AdamW optimizer, and used the voting strategy in testing.
17

---

Codebook
N-1
SD VOGAN Encoder
3D VOGAN Decaderr
3D VOGAN Discrininator
Vowelization
Quantization
2
2q
Input Point Clouds
Reconstructed Vowel
Vosel
L1 Reconstructed Loss
Occupancy
Figure 11: The training overview of 3D VQGAN. We introduce the occupancy rate loss to have better reconstruction of the voxel.
source
rebuild
source
rebuild
source
rebuild
source
rebuild
source
rebuild
source
rebuild
Figure 12: Reconstruction results of 3D VQGAN. Our model is capable of flawlessly reconstructing original voxel, and can even rectify some noise.
B.2 Implementation Details of 3D VQGAN
We show the detailed training overview of 3D VQGAN in Fig. 11. Following the training recipe of VQGAN [21], we use LI loss to supervise the reconstruction of the voxel and feed the reconstructed voxel into the diseriminator ensure the generated authenticity by GAN loss. Besides the L1 loss and GAN loss, we also introduce the occupancy rate loss to make the occupancy rate of the reconstructed voxel grid similar to the ground truth so obtain better reconstruction of the voxel. Fig. 12 illustrates the reconstructed results of 3D VQGAN, showcasing its impressive capabilities in the domain of 3D voxel reconstruction. As depicted in the figure, the model exhibits remarkable noise rectification capabilities. It not only reconstructs the voxels faithfully but also manages to rectify certain imperfections and artifacts present the input data.
C Additional Experiments
We conduct experiments further demonstrate the generation quality and universality of VPP. Including diversity & specific text-conditional generation, few-shot transfer classification, and ablation studies.
C.1 Novel Categories Point Clouds Generation
Besides the generation of common categories, we also conduct novel categories generation experi- ments to show the generalization performance of VPP in Fig. 13. Based on the learning of common categories, VPP can generalize to generate more novel category shapes, which are visually reasonable, such boat.
18

---

aérplane
asrplane boat
(a) common categories
novel categories
Figure 13: Novel categories point clouds generation.
Table 5: Downstream few-shot results on ModelNet40. Overall accuracy (%) w/o voting is reported.
5-way
10-way
Method
10-shot
20-shot
10-shot
20-shot
Supervised Learning Only
DGCNN [89]
31.6 + 2.8
40.8 1 4.6
19.
16.9 1.5
OcCo [86]
90.6 +
92.5
82.9 1.3
86.5 2.2
with Sele-Supervised Representation Learning
Transformer [84]
87.8 + 5.2
93.3 + 4.3
84.6 + 5.5
89.4 + 6.3
OcCo [86]
94.0 1 3.6
2.3
89.4 + 5.1
92.4 4.6
Point-BERT [98]
94.6 3.1
2.7
9..0
92.7 + 5.1
MaskPoint [44]
9.70
1.72
9.0
1
Point-MAE [60]
97.8 2.5
92.
95.0
Point-M2AE [102]
96.8
98.3
92.3 4.5
95.0
3.0
VPP (ours)
96.9 + 1.9
98.3 + 1.5
93.0 + 4.0
95.4 + 3.1
with Pretrained Cross-Madal Teacher Representation Learning
ACT [19]
96.8 1 2.3
98.0 + 1.4
93.3 1 4.0
95.6 +: 2.8
12P-MAE [103]
97.0 + 1.8
98.3 + 1.3
92.6 5.0
95.5 + 3.0
ReCon [68]
97.3 + 1.9
98.9 1 1.2
93.3 1 3.9
95.8 3.0
C.2 Few-Shot Transfer Classification
We conduct few-shot experiments on the ModelNet40 [92] dataset, and the results are shown in Table 5. In the self-supervised benchmark without the use of additional modality data, VPP achieves excellent performance compared previous works.
C.3 Ablation Study
Training Hyper Parameter Fig. 15 (a-b) shows the ablation study of the image-text features ratio and Gaussian noise strength. It can be observed that either too large or too small text-image feature ratio and noise are not conducive to the quality and diversity of 3D generation.
Inference Parameters To explore the parameter depen- dencies the Mask Voxel Transformer in the inference process, study the effects of temperature and iteration steps. The results are shown in Fig. 15 (c-d). can be seen that moderate temperature achieves optimal genera- tion classification accuracy. Higher temperature promotes model diversity, while with the increase of iteration steps, the model's FID initially decreases and then increases.
ACC1
FID
ResNer50
38.89
VIT-8/32
29.62
88.04
WT-8/16
31.34
85.92
Backbone Choice Fig. 14 shows the selection of the CLIP model backbones during VPP training. Both ViT-B/32 and ViT/L14 achieve excellent accuracy and diversity.
CLIP Backbone Choice
Figure 14: Ablation study on CLIP back- bone choices.
19

---

1otd
8:
0.15 05 955 96 645 0.55
3
0. 61 95 05 Noise
0
1.0
Rato
Temperature
Number Steps
Image-Text Festure Rario
Goussian Noise Strength
(c) Central Radistive Temperature
Iteaaioo Steps
Figure 15: Ablation study on image-text feature ratio, Gaussian noise strength, inference temperature, and inference steps.
D Limitations & Future Works
Despite the substantial advantages our model terms of generation speed and applicability, there exists considerable gap in generation quality compared to NeRF-based models [37, 65, 87]. To address this limitation, we intend to explore the utilization of larger models and more extensive datasets for training, e.g., Objaverse [13, 14]. Furthermore, multi-modal large language models with combined comprehension and generation abilities [18] have advanced rapidly, Incorporating VPP's 3D generation capabilities into such models represents potential avenue for future research.
Broader Impact
VPP enables the rapid generation of high-quality 3D models in a fraction of a second. This has wide-ranging applications in fields such gaming, virtual reality, augmented reality, and digital media production. These advances can lead to more immersive and visually stunning user experiences in entertainment and educational content. Furthermore, generative model, VPP may produce deceptive and malicious content and potentially impact associated employment opportunities.
20


Reviews:
[{"review_id": "foJw4PT8pv", "reviewer": "Reviewer_AZ71", "summary": "The paper proposed a text-driven point cloud generation model that can be used for various downstream tasks such as generation, editing, \n completion and pretraining, while being very efficient. The method largely follows Muse [2], but adapted it to 3D point clouds. The model consists of multiple components, which are trained individually. First, a VQGAN is trained on the voxelized shapes, which embed a shape to a latent grid. Next, a masked transformer is trained on the latent grid to model its distribution, using CLIP feature as semantic conditioning. For the decoding stage, a grid smoother network moves the voxel centers to more homogeneous locations, while finally another set of point cloud VQVAE and masked transformer is trained to upsample the coarse point cloud to high resolution point cloud. The paper also demonstrated that the proposed components can be used in whole or in part to tackle a wide range of tasks.", "questions": "* What is the benefit of using voxel as intermediate representation compared to directly encoding the point cloud?\n* Missing table reference on L234.", "limitations": "Limitations and societal impacts are adequately addressed in the supplemental material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed method is able to handle a wide range of tasks such as generation, editing, completion and pretraining.\n* VPP significantly outperforms baseline methods on the main task of shape generation.\n* The proposed method is computationally efficient compared to optimization-based methods such as DreamFusion.", "weaknesses": "* The components of the proposed method are not new -- they are mostly borrowed from previous works such as CLIP, MUSE and Point-MAE.\n* The method contained a large number of stages, making it potentially difficult to implement or improve upon.\n* Unlike diffusion-based methods such as DreamFusion, Dream3D, Magic3D, the proposed method does not generate surface or texture. It is also not clear if the proposed method is able to do zero-shot generation like these methods.\n* The proposed method is only trained on ShapeNet, which is relatively small and less diverse compared to larger datasets such as Objaverse. It is not sure if the method is able to scale to larger datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**W1: Are the components of the proposed method new?**\n\nWe have introduced some novel structures or strategies to adapt to 3D generation. Such as the **3D VQGAN with occ loss**, the **Grid Smoother** to smooth the gap between two representations, and the **Central Radiative Temperature Schedule** strategy during inference. \nOur goal is to share the merits of different representations for efficient multi-category generation. It is natural to draw inspiration from Muse for voxels and Point-MAE for points.\n\n**W2: Is VPP difficult to implement or improve upon?**\n\n- It is a common phenomenon to use three stages in the generative model. For example, Muse needs to train VQGAN, Base Transformer, and SuperRes Transformer before inference. \n- Furthermore, each module of VPP is based on CNN or Transformer, which is easy to implement. Modifying any stage could potentially contribute to improvements in the overall performance.\n\n**W3&4: Can VPP generate surface and do zero-shot generation? How about VPP trained on a larger dataset like Objaverse?**\n\n- First, we present the **surface reconstruction** results based on SAP in **Figure 1** of the PDF we provide in the Global Response. It demonstrates that VPP is capable of generating smooth surfaces. \n- Second, VPP trained on ShapeNet can not achieve open vocabulary zero-shot generation. But it can produce some **novel categories** to some extent. The results are shown in **Figure 6** of the provided PDF.\n- Third, by using a larger dataset **Objaverse**, VPP is capable of generating more common objects. We present the results in **Figure 2** of the provided PDF. Due to computational resource constraints, our model was trained for only 50 epochs on Objaverse. The current results represent preliminary findings.\n\n**Q1:  What is the benefit of using voxel as an intermediate representation compared to directly encoding the point cloud?**\n\n- We analyze the characteristics of different representations in the introduction. Due to significant shape differences across various categories, the structured and explicit positional information provides direct **spatial cues**, thereby aiding the generalization of multiple categories. This observation aligns with previous multi-category generation methods [1-2]. \n- Additionally, point clouds possess continuous and sparse semantic information and heavily rely on positional encoding [3]. The point token represents **both local geometric semantics and positional information**, which significantly reduces the generation performance in complex multi-category scenarios. We show the results for directly encoding the point cloud in Figure 7 of the global response PDF. As observed, models trained without voxel representation exhibit inferior performance in generating object details compared to our current approach, particularly for generating objects with intricate structures, such as \"an airplane\".\n\n**Q2: We have corrected all the typos.**\n\n[1] Sanghi A, Chu H, Lambourne J G, et al. Clip-forge: Towards zero-shot text-to-shape generation. CVPR 2022\n\n[2] Sanghi A, Fu R, Liu V, et al. CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes From Natural Language. CVPR 2023\n\n[3] Pang Y, Wang W, Tay F E H, et al. Masked autoencoders for point cloud self-supervised learning. ECCV 2022"}, {"review_id": "Dm5ZN2DDsu", "reviewer": "Reviewer_A5WF", "summary": "The VPP proposes a model for 3D generation. It utilizes both point-based and voxel-based representations. Voxel-based representations are used to generate the coarse tokens, and the point-based one further improves the result. Both of which are pretrained with MAE-like self-supervised method.\n\nThe proposed method applies to image-to-point, text-to-point, and point completion. The propsed method is novel and effective. However, some parts are not presented clearly.", "questions": "1. I am not sure what parallel decoding in Figure 3 means.\n2. When training the voxel semantic transformer, part of voxel tokens are masked, as shown in Figure 2a, and the masking ratio is about 64%. While all tokens are masked voxel tokens during inference. Is there a gap between training and inference?\n\n", "limitations": "Not applicable", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The limitations of existing methods are well analyzed.\n2. The proposed method is carefully designed and performs well.\n3. The proposed method applies to more downstream tasks than exsiting methods.\n4. The experiments show the effectiveness of the proposed method.", "weaknesses": "1. The inference efficiency in Table 1 only presents hours or seconds, it is recommended to provide the exact time.\n\nTotally, the paper is not well presented. Some details are not shown clearly. To be specific,\n2. The tokenizer for points upsampler is not described. What is it structure and how to train it?\n3. It would be better to refer to some important symbols in Figure2. For example, prompt embedding in Figure 2 should also be marked as C_pr.\n4. The GAN's structure and loss in 3D VQGAN are not introduced.\n5. As described in Section 3.2, the mask voxel transformer will be forwarded multiple times for better quality. It should also be commented in Figure 2 or 3 for better presence.\n\nSome typos:\n6. In about line 151: \"when the inputs of the mask voxel transformer are grid tokens with high masking fraction and prompt embedding we find that this will lead to the semantics of the predicted grid tokens being much more biased toward the prompt embedding\". I think the authors may miss punctuations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**W: Some presentation suggestions. The unclear explanation of tokenizer and 3D VQGAN.** \n\n- We appreciate your suggestions regarding the presentation. In the revised version, we include **the exact time in Table 1**, incorporate **symbols** into the process diagram, and emphasize multi-step decoding in Figure 2&3 to enhance readability. Additionally, we correct all the typos. \n\n- About the tokenizer of the Point Upsampler, we employ a **pre-trained Point-MAE** encoder as the tokenizer to generate semantic point tokens. We will elaborate on it in the subsequent revised version.\n\n- The structural diagram of 3D VQGAN is shown in **Figure 4** of the global response PDF. We have described the proposed $L_{occ}$ loss in the main paper and demonstrated the training pipeline of 3D VQGAN in the supplementary materials. We will further enrich the explanation of the VQGAN loss in the revised version.\n\n\n**Q1: The meaning of parallel decoding.** \n\nParallel decoding is introduced to distinguish it from sequential autoregressive decoding. In each step of parallel decoding, the model concurrently predicts all tokens in parallel, conditioned on the probability distribution from the previous step. Parallel decoding constitutes one of the crucial factors enabling VPP to achieve efficient generation.\n\n**Q2: Is there a gap between training and inference on the mask ratio?** \n\nVPP employs a cosine schedule during parallel decoding. Specifically, at each inference step: \n- The mask transformer predicts the logits of the codebook for all masked grid tokens. \n- Subsequently, we employ the cosine schedule to calculate the mask ratio at the current step and determine which tokens are to be fixed based on the logits scores. \n- The non-fixed tokens are replaced with mask tokens and proceed to the next step. \n\nDue to the cosine schedule of mask ratio during inference, we sample mask ratios from the arccos distribution during training to simulate various inference steps. The mean mask ratio is 0.64, yet it **doesn't mean a fixed mask ratio**. "}, {"review_id": "PtH7iF8WeV", "reviewer": "Reviewer_YMLb", "summary": "Authors propose an approach to generate 3D point clouds of objects with an image or text description as input. Authors use a pre-trained CLIP model to generate text/image embeddings and use this to first generate features in voxel space (Voxel Semantic Generator). These voxel features are then decoded into a coarse voxel grid. Authors then convert the voxels to point clouds and use a smoothing network to obtain a uniform point cloud. Authors use a transformer to then convert this point cloud into a detailed 3D shape (also point cloud).\nAuthors show qualitative and quantitative comparison with relevant baselines (CLIP-Sculptor, CLIP-Forge and Point-E).\nAuthors also show several applications like text and image conditioned 3D generation, shape editing and completion.", "questions": "Overall I'm positive about the work. There are some concerns about the novelty/necessity of some components (see above) and it would be great if authors can clarify these.", "limitations": "Discussion on Limitations and Broader Impact is not included in the main paper but it is present in the supp. mat. Authors are encouraged to add some discussion on limitations and future work in the main paper as it allows the community to better use and build upon your research. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Ideas presented in the work are technically sound.\n+ Presented results outperform competing baselines.\n+ Authors submitted code. Although I did not run it but this is still appreciated. Code will also be released upon acceptance.\n+ Paper is mostly well written. See comments below for minor improvements.", "weaknesses": "[Technical]\n1. How is the proposed Voxel Semantic Generator different than “CLIP-Conditioned Coarse Transformer” from CLIP-Sculptor? They are essentially doing the same thing, 1. Learn a voxel based encoding of 3D shapes and 2. Use a transformer to learn how to unmask the 3D features conditioned on a prompt.\nCan authors clarify this better?\n\n2. L175-178: Since the GT for Grid Smoother is generated using furthest point sampling, why can’t we use FSP at inference time to smoothen the coarse voxel grid from the 3D VQGAN decoder?\nWhy do we need to learn a neural network?\nHow useful is adding KL loss over Chamfer loss?\n\n3. L180: The output of the grid smoother is a point cloud (I assume this because it is trained with a Chamfer loss). L169 mentions that “point tokens” are masked which are then unmasked by the transformer. What exactly are the “point tokens”? Is it just the positional encoding of points?\nFig.2 (b) mentions “semantic tokens” what does this mean? It is not explained in Sec 3.3 (Point Upsampler).\n\n4. Eq. 1: How useful is the occupancy loss over MSE? Is this critical? Please add an ablation study in supp. mat. to support the proposal.\n\n[Minor]\n- Fig. 2: Please add symbols used in the text to Fig.2. This makes it easier to follow the text and map various components of the pipeline.\n- Add dimensionality of each embedding/latent code along with the symbols, eg: C_{pr} \\in \\mathbb{R}^{<whatever>\\times<whatever>}. This significantly improves clarity and readability.\n- L120: How is the mapping performed? How is the codebook generated? If an existing work is used, please add citation here? This is present in the supp. mat. to an extent. Add a pointer so that the reader knowns where to look.\n- L140: Just curious, how important is the scheduling of the mask fraction?\nL147: Is there significant performance difference at inference time between direct and multi-step prediction?\n- L234: Table number is missing.\n- Please avoid violent objects like guns for showing qualitative results. It is understandable that it is sometimes necessary to demonstrate performance but whenever possible, let us try to avoid this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**W1: Difference between Voxel Semantic Generator and CLIP-Sculptor?**\n\n- Our goal is to share the representation advantages of both voxels and points. The objective of the Voxel Semantic Generator is to provide positional encoding for the Point Upsampler. In contrast, CLIP-Sculptor entirely uses voxels as representation. The transformer sequence length is cubically proportional to the voxel resolution, resulting in slower inference speeds at high resolutions. \n- Additionally, VPP employs a **meticulously designed 3D VQGAN** to generate codebooks rather than VQVAE. VPP utilizes a mask ratio based on the **arccos distribution sampling** method, rather than a two-step unrolled training loss.\n- During inference, we use the **Central Radiative Temperature Schedule strategy** to get better performance.\n\n**W2: Why do we need the proposed Grid Smoother instead of FPS?**\n\nFPS cannot serve as a substitute for the proposed Grid Smoother module. Its objective is to **mitigate the representation gap** as possible. This is because the point cloud generated by the Voxel Semantic Generator is **discrete and grid**. Through a neural network, we obtain **continuous and uniformly distributed point clouds**. The effect of grid smoothing is demonstrated in **Figure 3** of the global response PDF. In contrast, the nature of FPS is downsampling, rendering it incapable of producing continuously valued points.\n\nFurthermore, FPS leads to a reduction in point quantity. Typically, our Voxel Generator can only produce between 200 to 500 points. Employing FPS downsampling would result in the loss of certain geometric information.\n\n**W3: The explanation of \"point tokens\" and \"semantic tokens\".**\n\nWe employ a pre-trained Point-MAE encoder as the tokenizer of PointUpsampler to generate semantic point tokens. The point cloud generated by the grid smoother is utilized to provide positional encoding for these semantic tokens. We provide a more comprehensive explanation of this aspect in the revised version.\n\n**W4: How useful is the occupancy loss over MSE?**\n\nThanks for your suggestion! We have incorporated this result into **Figure 3** of the global response PDF. As depicted, the addition of the $L_{occ}$ loss significantly improves the reconstruction performance compared to the vanilla VQGAN. This improvement not only faithfully restores the voxels but also effectively **corrects specific imperfections and noise** in the input data.\n\nFurthermore, the following table presents a quantitative performance comparison regarding the $L_{occ}$ loss.\n\n|              | Acc    | FID   | IS    |\n| ------------ | ------ | ----- | ----- |\n| VPP w Locc   | 88.04% | 29.82 | 10.64 |\n| VPP w/o Locc | 84.19% | 35.75 | 9.98  |\n\n\n\n**Minor**\n\n- We greatly appreciate all the constructive suggestions you have provided. We will incorporate symbols in the figures, add pointers, and correct typos, which will significantly enhance the readability. \n- Concerning the scheduling of the mask ratio, MaskGit[1] has conducted experiments on the scheduling strategy of mask fraction for parallel decoding. In this study, we adopt the optimal strategy. \n- In the ablation study Figure 8 of the main paper, we observed that a moderate number of inference steps (4/8) yields the best generative performance, while single-step generation leads to a decrease in accuracy by up to 6%.\n\n[1] Chang H, Zhang H, Jiang L, et al. Maskgit: Masked generative image transformer. CVPR 2022"}, {"review_id": "V7CQjvYdOp", "reviewer": "Reviewer_THuC", "summary": "This work proposes to use a voxel-point progressive representation for efficient 3D generation, and it proposes a few architectures for different applications, including generation, editing, upsampling, and pre-training. Based on the reported results, the proposed method could generate various 3D shapes and could achieve competitive classification results.", "questions": "1. Is the proposed method capable of generating 3D shapes of novel categories?\n\n2. Is the proposed method capable of generating complete point clouds given a partial point cloud?\n\n3. Does the proposed method demonstrate robustness when presented with noisy point clouds as input?\n\n4. Is the generated 3D shapes caused by overfitting the ShapeNet dataset? Maybe the authors could try to retrieve the most similar shape in the ShapeNet dataset to prove its generation ability.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The work proposes a voxel-point progressive representation, which could provide good 3D generation results on the ShapeNet dataset as shown in the reported experiments. \n\n2. Many modules, such as 3D VQGAN, Voxel Semantic Generator, Grid Smoother, and Point Upsampler, have been proposed.", "weaknesses": "1. The proposed method could only generate 3D shapes belonging to the categories of the ShapeNet dataset.  It does not show any 3D shape results from unseen categories even with the help of CLIP.\n\n2. Point-Voxel representation has been broadly studied in previous methods, such as [A-C], which has already been proven to be an efficient representation for 3D point cloud analysis.\n\n3. The classification results for both the ScanObjectNN and ModelNet40 datasets are very saturated.  Also, the improvements seem to be very incremental.\n\n[A] Liu, Z., Tang, H., Lin, Y., & Han, S. (2019). Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32.\n\n[B] Zhang, C., Wan, H., Shen, X., & Wu, Z. (2022). PVT: Point‐voxel transformer for point cloud learning. International Journal of Intelligent Systems, 37(12), 11985-12008.\n\n[C] Liu, Z., Tang, H., Zhao, S., Shao, K., & Han, S. (2021). Pvnas: 3d neural architecture search with point-voxel convolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8552-8568.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**W1: Unseen categories generation.**\n\n- Differing from NeRF-based approaches like DreamFusion, although they can achieve open vocabulary zero-shot generation, the high computational time and training costs make practical utilization challenging. Our method strikes a balance among **multi-category generation, generation efficiency, and generation quality**, and is able to perform **multiple 3D downstream tasks** including conditional generation, editing, completion, and pre-training. \n- VPP trained on ShapeNet is incapable of achieving open vocabulary zero-shot generation, but it can produce novel categories to some extent. Furthermore, by employing a larger dataset Objaverse[1], VPP demonstrates the capability to generate more common objects. These results are illustrated in **Figure 2&6** of our global response PDF. Due to computational resource constraints, our model was trained for only 50 epochs on Objaverse. The current results represent preliminary findings.\n\n**W2: The correlation between the previous point-voxel representation and VPP.**\n\n- The previous point-voxel methods were predominantly architectural endeavors, aimed at achieving improved classification, detection, or segmentation performance. \n- VPP is the first endeavor that achieves efficient conditional 3D generation through the sharing of distinct representation advantages. We express our gratitude to the preceding point-voxel representation methodologies, and in the appendix, we will incorporate an Additional Related Work section to acknowledge and cite pertinent contributions.\n\n**W3: Improvement of classification results.**\n\n- Our Point Upsampler is implemented based on Point-MAE and achieved a 4.1 % performance improvement on ScanobjectNN. Our model achieves SOTA performance on the benchmark that only uses ShapeNet point clouds as pre-trained data. \n- Some recent work, such as I2P-MAE[2] and ReCon[3], achieved better performance through cross-modal tutors and cross-modal data.\n\n**Q1: Capability of generating 3D shapes of novel categories.**\n\nWe conduct novel-categories generation experiments, which demonstrated the capability to generate novel-categories shapes, such as \"a car boat\". The results are presented in **Figure 6** of the global response PDF.\n\n**Q2: Capability of partial point cloud generation.**\n\nThank you for your suggestion. We generate partial data by employing a block mask on the original point clouds. The results are depicted in **Figure 5** of the global response PDF, illustrating the partial generation capability of VPP. Furthermore, the generated samples exhibit diversity, providing further evidence of the performance of VPP.\n\n**Q3: Robustness of VPP when presented with noisy point clouds as input.**\n\n- As depicted in **Figure 3** of the global response PDF, our model demonstrates the capacity to not only faithfully reconstruct the provided input but also proficiently rectify distinct imperfections and noise inherent within the input data. \n- In addition, we quantitatively compare the robustness of the model to the training data. The table below shows the changes in text-conditional generation metrics after we added random scale, translate, and jitter disturbances to the input point cloud.\n\n|      | disturbance | Acc    | FID   | IS    |\n| ---- | ----------- | ------ | ----- | ----- |\n| VPP  | -           | 88.04% | 29.82 | 10.64 |\n| VPP  | random scale       | 87.63% | 31.20 | 10.15  |\n| VPP  | random translate   |83.76% | 36.92 | 8.71  |\n| VPP  | random jitter      | 87.29% | 30.68 | 10.44  |\n\n\n**Q4: Are the generated 3D shapes caused by overfitting the ShapeNet dataset?**\n\nThank you for your suggestion. We conduct a retrieval evaluation on samples generated by VPP and the ShapeNet dataset. The results are shown in **Figure 8** of the global response PDF. There are no completely identical samples, therefore VPP is not overfitting the ShapeNet dataset. The results generated by VPP are more like an understanding and integration of shape knowledge.\n\n\n\n[1] Deitke M, Schwenk D, Salvador J, et al. Objaverse: A universe of annotated 3d objects. CVPR 2023\n\n[2] Zhang R, Wang L, Qiao Y, et al. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. CVPR 2023\n\n[3] Qi Z, Dong R, Fan G, et al. Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining ICML 2023"}, {"review_id": "D5j1h7LRdW", "reviewer": "Reviewer_U3Fo", "summary": "The paper proposes an efficient conditional 3D generation via voxel-point progressive representation. More specifically, a voxel semantic generator and a point upsampler have been created to achieve efficient generation on multi-category objects. To verify the effectiveness of the method, extensive experiments with SOTA results are achieved. ", "questions": "1. To reduce the degree of confidence when generating voxels because of the high masking rate, Gaussian noise is added to the prompt embedding.The question is instead of adding Gaussian noise, how about directly sampling smaller masking rate? \n2. How is Eq. (2) obtained? It would be better to provide some hints on the calculation.  ", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-organized. \n2. Extensive experiments are conducted, and impressive results are obtained. ", "weaknesses": "1. It seems that the paper can address point upsample task instead of upsample completion task, since upsample refers to generate dense points while completion refers to synthesize new points given partial scans.  \n2. It would be better to show some ablation studies on the network architectures such as removing or replacing specific components to see how they affect the performances.  \n3. It would be clearer to indicate Tab .3 in line 234 for the corresponding quantitative results. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**W1: The capability of VPP for partial completion task.**\n\nThank you for your suggestions, and we will make the necessary revisions to the pertinent statement. Furthermore, in **Figure 5** of the global response PDF, we show the partial generation results, which provide additional evidence to substantiate the efficacy of VPP.\n\n**W2: More ablation studies on the components of the proposed VPP.**\n\n- In **Figure 3** of the global response PDF, we present the visualization results of our proposed 3D VQGAN and Grid Smoother. It is observable that the VQGAN with $L_{occ}$ loss exhibits **enhanced reconstruction performance and adeptness in countering adversarial noise**. The Grid Smoother effectively accomplishes its role in voxel-point smoothing. \n- Furthermore, we quantitatively evaluate the impact of specific components on performance, as delineated in the following table.\n\n|          | Acc    | FID   | IS    |\n| -------- | ------ | ----- | ----- |\n| VPP w Locc   | 88.04% | 29.82 | 10.64 |\n| VPP w/o Locc | 84.19% | 35.75 | 9.98  |\n\n|                   | Acc    | FID   | IS    |\n| ----------------- | ------ | ----- | ----- |\n| VPP w Grid Smoother   | 88.04% | 29.82 | 10.64 |\n| VPP w/o Grid Smoother | 86.32%  | 32.13 | 10.25 |\n\n**W3: We have corrected all the typos.**\n\n**Q1: How about directly sampling a smaller masking rate instead of adding Gaussian noise?**\n\nDue to the cosine schedule generation steps during inference, we simulate various steps that may be encountered during inference by sampling from the arccos distribution. Therefore, the mask ratio during training is **not a constant value**, and it is necessary to sample high mask ratios as well. This is because the inference process starts with a 100% [mask token].\n\n**Q2: Some hints on the calculation of Eq.(2).**\n\nThe approach of infusing adaptive Gaussian noise into cross-modal features originates from [1], wherein noise parameter λ directly influences the generated samples X to **modulate the generative dependence**. The authors derived and demonstrated this by employing the cumulative distribution function of the inner product of random vectors on a sphere, along with the utilization of the Gamma function.\n\n\n\n[1] Towards language-free training for text-to-image generation, CVPR 2022"}]

Author rebuttal:
We sincerely thank all reviewers for your valuable feedback that significantly contributed to our work. VPP achieves efficient, multi-category, high-quality conditional generation through voxel-point progressive representation, and is capable of performing various tasks such as editing, completion, and pre-training.

We present additional experimental results in the following PDF to offer a more intuitive and clear response to the reviewer's relevant comments and questions. Moreover, we further demonstrate the optimized mesh generation performance and the generation results using a larger dataset. Specifically, this PDF includes the following:

Figure 1: Improved surface reconstruction results through SAP.

Figure 2: Visualization of text-conditioned generation results using Objaverse dataset.

Figure 3: Visualization results of 3D VQGAN and Grid Smoother.

Figure 4: Structure diagram of the 3D VQGAN.

Figure 5: Generation results of partial inputs.

Figure 6: Novel categories generation results.

Figure 7: Ablation experiment that only uses point clouds as representation.

Figure 8: Ablation experiment that retrieval ShapeNet samples.

Due to space limitations, we employ a two-column layout.

Meta-review: This paper proposes a conditional 3D generation method via voxel-point progressive representation. It receives two weak accepts, one borderline accept, one accept and one borderline reject. The authors well resolved the reviewers' concerns. Reviewer THuC raises two new concerns during discussion. However, it is pretty late for the author to response. The AC thinks the new issues do not affect the contribution of this paper and recommends acceptance for this paper. The authors are urged to solve the new concerns in the camera ready version.
Decision: Accept (poster)