SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zekun Qi† Muzhou Yu† Runpei Dong‡
{qzk139318，muzhou9999，runpei.dong}@stu.xjtu.edu.cn
Kaisheng Ma
kaisheng@mail.tsinghua.edu.cn
Figure 1: Overview of VPP's applications through generative modeling.
Conditional 3D generation is undergoing a significant advancement, enabling the free creation of 3D content from inputs such as text or 2D images. However, previous approaches have suffered from low inference efficiency, limited generation categories, and restricted downstream applications. In this work, we revisit the impact of different 3D representations on generation quality and efficiency. We propose a progressive generation method through Voxel-Point Progressive Representation (VPP). VPP leverages structured voxel representation in the proposed Voxel Semantic Generator and the sparsity of unstructured point representation in the Point Upsampler, enabling efficient generation of multi-category objects. VPP can generate high-quality 8K point clouds within 0.2 seconds. Additionally, the masked generation Transformer allows for various 3D downstream tasks, such as generation, editing, completion, and pre-training. Extensive experiments demonstrate that VPP efficiently generates high-fidelity and diverse 3D shapes across different categories, while also exhibiting excellent representation transfer performance. Codes will be released at https://github.com/qizekun/VPP.
$\dagger$  Equal contribution.  $\ddagger$ Project lead.  $\ddagger$ Corresponding author.
Table 1: Comparison of text-conditioned 3D generation methods on efficiency and applications.
Method	Latency	Device	Generation Category	Downstream Task
DreamFields [37]	1.2h	8xTPUv4	Multi-Category	Generation
DreamFusion [65]	1.5h	4xTPUv4	Multi-Category	Generation
CLIP-Mesh [58]	30min	V100	Multi-Category	Generation
CLIP-Sculptor [77]	0.9s	V100	Multi-Category	Generation
Point-E (40M) [59]	25s	V100	Multi-Category	Generation
ShapeCrafter [24]	-	-	Single-Category	Generation, Editing
LION [99]	27s	V100	Single-Category	Generation, Completion
SDFusion [9]	-	-	Single-Category	Generation, Completion
VPP (Ours)	0.2s	RTX 2080Ti	Multi-Category	Generation, Editing, Completion, Pretraining
In the last few years, text-conditional image generation [62, 71, 72, 74] has made significant progress in terms of generation quality and diversity and has been successfully applied in games [7], content creation [47, 48] and human-AI collaborative design [36]. The success of text-to-image generation is largely owed to the large-scale image-text paired datasets [78]. Recently, there has been an increasing trend to boost conditional generation from 2D images to 3D shapes. Many works attempt to use existing text-3D shape paired datasets [7, 9, 24, 50] to train the models, but the insufficient pairs limit these methods to generate multi-category, diverse, and high-res 3D shapes.
To tackle this challenge, some works [37, 65, 82, 94] use NeRF [56] and weak supervision of large-scale vision-language models [70] to generate 3D shapes. Although creative quality is achieved, the optimization costs of differentiable rendering are quite expensive and impractical. Some methods use images as bridges [38, 59] for the text-conditional 3D generation, which improves the quality-time trade-off. However, the complex multi-stage model leads to long inference latency and cumulative bias. Moreover, most methods [58, 76] are task-limited due to specific model design that can not be implemented to more downstream tasks, e.g., editing, leading to narrow application scope. To sum up, current methods are faced with three challenges: low inference efficiency, limited generation category, and restricted downstream tasks, which have not been solved by a unified method.
One of the most crucial components in 3D generation is the geometry representation, and mainstream methods are typically built with voxel, 3D point cloud, or Signed Distance Function (SDF) [11, 61]. However, the aforementioned issues in 3D generation may come from the different properties of these representations, which are summarized as follows:
- Voxel is a structured and explicit representation. It shares a similar form with 2D pixels, facilitating the adoption of various image generation methods. However, the dense nature of voxels results in increased computational resources and time requirements when generating high-resolution shapes.
- 3D Point Cloud is the unordered collection of points that explicitly represents positional information with sparse semantics, enabling a flexible and efficient shape representation. However, 3D point clouds typically dedicatedly designed architectures, e.g., PointNet [66, 67].
- SDF describes shapes through an implicit distance function. It is a continuous representation method and is capable of generating shapes with more intricate details than points or voxels. But it requires substantial computation for high-resolution sampling and is sensitive to initialization.
Due to significant shape differences across various categories, the structured and explicit positional information provides spatial cues, thereby aiding the generalization of multiple categories. It coincides with previous multi-category generation methods [76, 77]. In contrast, single-category generation may require less variation. Hence an implicit and continuous representation would contribute to the precise expression of local details [9, 25].
Motivated by the analysis above, we aim to leverage the advantage of different forms of representation. Therefore, a novel Voxel-Point Progressive (VPP) Representation method is proposed to achieve efficient and universal 3D generation. We use voxel and point as representations for the coarse to fine stages to adapt to the characteristics of 3D. To further enhance the inference speed, we employ mask generative Transformers for parallel inference [5] in both stages, which can concurrently obtain broad applicability scope [42]. Based on voxel VQGAN [21], we utilize the discrete semantic codebooks as the reconstruction targets to obtain authentic and rich semantics. Given the substantial computational resources required by high-resolution voxels, we employ sparse points as representations in the
input voxel
rec voxel
rec voxel (with  $\mathcal{L}_{\mathrm{OCC}}$
(a) Reconstruction Results of 3D VQGAN
smoothed points
grid points
smoothed points
(b) Smoothing performance of Grid Smoother
Figure 2: Qualitative examples. (a) 3D VQGAN reconstruction results with or without  $\mathcal{L}_{\mathrm{OCC}}$  (Section 2.1). (b) 3D point clouds result with or without proposed Grid Smoother (Section 2.2).
second stage. Inspired by the masked point modeling approaches based on position cues [60, 98], we utilize the voxels generated in the first stage as the positional encoding for generative modeling. We reconstruct the points and predict semantic tokens to obtain sparse representations. Table 1 compares the efficiency and universality of text-conditional 3D generation methods.
In summary, our contributions are: (1) We propose a novel voxel-point progressive generation method that shares the merits of different geometric representations, enabling efficient and multi-category 3D generation. Notably, VPP is capable of generating high-quality 8K point clouds within 0.2 seconds on a single RTX 2080Ti. (2) To accommodate 3D generation, we implement unique module designs, e.g. 3D VQGAN, and propose a Central Radiative Temperature Schedule strategy in inference. (3) As a universal method, VPP can complete multiple tasks such as editing, completion, and even pre-training. To the best of our knowledge, VPP is the first work that unifies various 3D tasks.
2 VPP
Our goal is to design a 3D generation method by sharing the merits of both geometric representations, which is capable of addressing the three major issues encountered in previous generation methods. In this work, we propose VPP, a Voxel-Point Progressive Representation method for two-stage 3D generation. To tackle the problem of limited generation categories, VPP employs explicit voxels and discrete VQGAN for semantics representation in the first stage. To address the issue of low inference efficiency, VPP employs efficient point clouds as representations in the second stage and leverages the parallel generation capability of the mask Transformer. Given the wide range of applications of generative modeling, VPP resolves the problem of restricted downstream tasks. The training overview of VPP is shown in Fig. 3.
2.1 How Does 2D VQGAN Adapt to 3D Voxels?
Due to the formal similarity between voxels and pixels, we can readily draw upon various image generation methods. VQGAN [21] is capable of generating realistic images through discrete semantic codebooks and seamlessly integrating with mask generative Transformers [5, 6]. We utilize 3D volumetric SE-ResNet [35] as the backbone of VQGAN encoder  $E$ , decoder  $G$  and discriminator  $D$ .
Given one voxel observation  $\mathbf{v} \in \mathbb{R}^3$  whose occupancy is defined as function  $o: \mathbb{R}^3 \to [0,1]$ . The original VQGAN is trained by optimizing a vectorized quantized codebook with the loss  $\mathcal{L}_{\mathrm{VQ}}$  during autoencoding while training a discriminator between real and reconstructed voxels with loss  $\mathcal{L}_{\mathrm{GAN}}$ . Denoting the reconstructed voxel result as  $\hat{\mathbf{v}}$ , the overall loss of vanilla VQGAN can be written as:
$$
\underbrace {\| \mathbf {v} - \hat {\mathbf {v}} \| ^ {2} + \| \mathrm {s g} [ E (\mathbf {v}) ] - z _ {\mathbf {q}} \| _ {2} ^ {2} + \| \mathrm {s g} [ z _ {\mathbf {q}} ] - E (\mathbf {v}) \| _ {2} ^ {2}} _ {\mathcal {L} _ {\mathrm {V Q}} (E, G, \mathcal {Z})} + \underbrace {\left[ \log D (\mathbf {v}) + \log (1 - D (\hat {\mathbf {v}})) \right]} _ {\mathcal {L} _ {\mathrm {G A N}} (\{E, G, \mathcal {Z} \}, D)}.
$$
However, unlike 2D images, 3D voxels only have non-zero values in the region that encompasses the object in grid centroids. This may lead to a short-cut solution that degrades by outputting voxels with values as close to zero as possible, i.e., all empty occupancy. To address this issue, we propose to
(a) Voxel Semantic Generator
(b) Point Upsampler
Figure 3: Training process overview of VPP. (a) In the coarse stage, the VQ codebook is reconstructed via Mask Voxel Transformer conditioned on prompt embeddings. (b) In the fine stage, sparse semantic tokens are reconstructed via the Mask Point Transformer conditioned on positional embeddings.
minimize the occupancy disparity between the reconstructed and ground truth voxels. Formally, let  $\zeta \circ o(\mathbf{v})$  be the averaged occupancy in SE(3) space, the overall loss  $\mathcal{L}_{\mathrm{3D - VQGAN}}$  is:
$$
\mathcal {L} _ {\mathrm {3 D - V Q G A N}} = \mathcal {L} _ {\mathrm {V Q}} + \mathcal {L} _ {\mathrm {G A N}} + \underbrace {\left\| \zeta \circ o (\mathbf {v}) - \zeta \circ o (\hat {\mathbf {v}}) \right\| _ {2} ^ {2}} _ {\mathcal {L} _ {\mathrm {O C C}}}, \tag {1}
$$
where  $\mathcal{L}_{\mathrm{OCC}}$  is proposed to minimize the occupancy rate for better geometry preservation.
2.2 How to Bridge the Representation Gap Between Voxels and Points?
We aim to utilize the voxel representation as the intermediate coarse generation before fine-grained point cloud results. However, voxels are uniform and discrete grid data, while point clouds are unstructured with unlimited resolutions, which allows for higher density in complex structures. Besides, point clouds typically model the surfaces, while voxel representation is solid in nature. This inevitably leads to a representation gap issue, and a smooth transformation that bridges the representation gap is needed. We employ the Lewiner algorithm [40] to extract the surface of the object based on voxel boundary values and utilize a lightweight sequence-invariant Transformer [84] network  $\mathcal{T}_{\eta}$  parametrized with  $\eta$  as Grid Smoother to address the issue of grid discontinuity. Specifically, given generated point clouds  $\mathcal{P} = \{\mathbf{p}_i \in \mathbb{R}^3\}_{i=1}^N$  with  $N$  points. The model is trained by minimizing the geometry distance between generated and ground truth point clouds  $\mathcal{G} = \{\mathbf{g}_i \in \mathbb{R}^3\}_{i=1}^N$  that is downsampled to  $N$  with farthest point sampling (FPS). To make the generated surface points more uniformly distributed, we propose  $\mathcal{L}_{\mathrm{uniform}}$  that utilizes the Kullback-Leibler (KL) divergence  $D_{\mathrm{KL}}$  between the averaged distance of every centroid  $\mathbf{p}_i$  and its K-Nearest Neighborhood (KNN)  $\mathcal{N}_i$ . Then we optimize the parameter  $\eta$  of Grid Smoother by minimizing loss  $\mathcal{L}_{\mathrm{SMO}}$ :
$$
\mathcal {L} _ {\mathrm {S M O}} = \underbrace {\frac {1}{| \mathcal {T} _ {\eta} (\mathcal {P}) |} \sum_ {\mathbf {p} \in \mathcal {T} _ {\eta} (\mathcal {P})} \min  _ {\mathbf {g} \in \mathcal {G}} \| \mathbf {p} - \mathbf {g} \| + \frac {1}{| \mathcal {G} |} \sum_ {\mathbf {g} \in \mathcal {G}} \min  _ {\mathbf {p} \in \mathcal {T} _ {\eta} (\mathcal {P})} \| \mathbf {g} - \mathbf {p} \|} _ {\mathcal {L} _ {\mathrm {C D}}} + \underbrace {D _ {\mathrm {K L}} \left[ \sum_ {j \in \mathcal {N} _ {i}} \| \mathbf {p} _ {i} - \mathbf {p} _ {j} \| _ {2} , \mathcal {U} \right]} _ {\mathcal {L} _ {\text {u n i f o r m}}} \tag {2}
$$
where  $\mathcal{L}_{CD}$  is  $\ell_1$  Chamfer Distance [23] geometry disparity loss,  $\mathcal{U}$  is the uniform distribution.
2.3 Voxel-Point Progressive Generation Trough Mask Generative Transformer
We employ mask generative Transformers [5] for rich semantic learning in both the voxel and point stages. Mask generative Transformer can be seen as a special form of denoising autoencoder [34, 85]. It forces the model to reconstruct masked input to learn the statistical correlations between local patches. Furthermore, generating reconstruction targets through a powerful tokenizer can further enhance performance [2, 19, 91, 104]. From the perspective of discrete variational autoencoder
Figure 4: Inference process overview of VPP. Point clouds are generated conditioned on the input text or images. Parallel decoding is adopted for efficient Mask Transformer token prediction.
(dVAE) [2, 26, 39, 71], the overall optimization is to maximize the evidence lower bound (ELBO) [17, 32, 73] of the log-likelihood  $\mathrm{P}(x_i|\tilde{x}_i)$ . Let  $x$  denote the original data,  $\tilde{x}$  the masked data, and  $z$  the semantic tokens, the generative modeling can be described as:
$$
\sum_ {\left(x _ {i}, \tilde {x} _ {i}\right) \in \mathcal {D}} \log \mathrm {P} \left(x _ {i} \mid \tilde {x} _ {i}\right) \geq \sum_ {\left(x _ {i}, \tilde {x} _ {i}\right) \in \mathcal {D}} \left(\mathbb {E} _ {z _ {i} \sim \mathrm {Q} _ {\phi} (\mathbf {z} \mid x _ {i})} \left[ \log \mathrm {P} _ {\psi} \left(x _ {i} \mid z _ {i}\right) \right] - D _ {\mathrm {K L}} \left[ \mathrm {Q} _ {\phi} (\mathbf {z} \mid x _ {i}), \mathrm {P} _ {\theta} (\mathbf {z} \mid \tilde {z} _ {i}) \right]\right), \tag {3}
$$
where (1)  $\mathrm{Q}_{\phi}(z|x)$  denotes the discrete semantic tokenizer; (2)  $\mathrm{P}_{\psi}(x|z)$  is the tokenizer decoder to recover origin data; (3)  $\tilde{z} = \mathrm{Q}_{\phi}(z|\tilde{x})$  denotes the masked semantic tokens from masked data; (4)  $\mathrm{P}_{\theta}(z|\tilde{z})$  reconstructs masked semantic tokens in an autoencoding way. In the following sections, we extend the mask generative Transformer to voxel and point representations, respectively. The semantic tokens of voxel and point are described as  $z^v$  and  $z^p$  in Figs. 3 and 4.
Voxel Semantic Generator We generate semantic voxels with low resolution in the first stage to balance efficiency and fidelity. We utilize Mask Voxel Transformer for masked voxel inputs, where the 3D VQGAN proposed in Section 2.1 is used as the tokenizer to generate discrete semantic tokens. Following prior arts [5, 6], we adopt Cosine Scheduling to simulate the different stages of generation. The masking ratio is sampled by the truncated arccos distribution with density function  $p(r) = \frac{2}{\pi} (1 - r^2)^{-\frac{1}{2}}$ . The prompt embedding comes from the CLIP [70] features of cross-modal information. We render multi-view images from the 3D object [68] and utilize BLIP [41] to get the text descriptions of the rendered images. Furthermore, in order to mitigate the issue of excessive dependence on prompts caused by the high masking ratio (e.g., the same results will be generated by one prompt), we introduce Classifier Free Guidance [33] (CFG) to strike a balance between diversity and quality. Following Sanghi et al. [77], the prompt embeddings are added with Gaussian noise to reduce the degree of confidence. We use  $\epsilon \sim \mathcal{N}(0,1)$  and  $\gamma \in (0,1)$  as the level of noise perturbation for the trade-off control, which will later be studied.
Point Upsampler Point clouds are unordered and cannot be divided into regular grids like pixels or voxels. Typically, previous masked generative methods [19, 60] utilize FPS to determine the center of local patches, followed by KNN to obtain neighboring points as the geometric structure. Finally, a lightweight PointNet [66, 67] or DGCNN [89] is utilized for patch embedding. However, these methods rely on pre-generated Positional Encodings (PE), rendering the use of fixed positional cues infeasible in conditional 3D generation since only conditions like texts are given. To tackle this issue, we adopt the first-stage generated voxels-smoothed point clouds as the PEs. A pretrained Point-MAE [60] encoder is used as the tokenizer (i.e., teacher [19]). The Mask Point Transformer learns to reconstruct tokenizer features, which will be used for the pretrained decoder reconstruction.
2.4 Inference Stage
Parallel Decoding An essential factor contributing to the efficient generation of VPP is the parallel generation during the inference stage. The model takes fully masked voxel tokens as input and generates semantic voxels conditioned on the CLIP [70] features of text or image prompts. Following [5, 6], a cosine schedule is employed to select the fixed fraction of the highest confidence masked tokens for
Figure 5: Qualitative comparison of VPP, CLIP-Sculptor [77] and CLIP-Forge [76] on text conditioned 3D generation. VPP generates shapes of higher fidelity with smoother surfaces while being consistent with the input text. More diverse results are shown in Appendix C.
prediction at each step. Furthermore, we pass the generated voxels through Grid Smoother to obtain smoothed point clouds, which will serve as the positional encoding for the Point Upsampler in the upsampling process. Note that all the stages are performed in parallel. The inference overview of VPP is illustrated in Fig. 4.
Central Radiative Temperature Schedule Due to the fact that 3D shape voxels only have non-zero values in the region that encompasses the object in the center, we hope to encourage diversity in central voxels while suppressing diversity in edge vacant voxels. Consequently, we propose the Central Radiative Temperature Schedule to accommodate the characteristics of 3D generation. As for the voxel with a size of  $R \times R \times R$ , we calculate the distance  $r$  to the voxel center for each predicted grid. Then, we set the temperature for each grid as  $T = 1 - (r / R)^2$ , where  $T$  represents the temperature. Thus, we can achieve the grid that is closer to the center will have a higher temperature to encourage more diversity and vice versa.
3.1 Conditional 3D Generation
Text-conditioned Generation Following [59, 76, 77], we use Accuracy (ACC), Fréchet Inception Distance (FID) [31], and Inception Score (IS) [75] as metrics to assess the generation quality and diversity. We follow [76] to define 234 predetermined text queries and apply a classifier for evaluation. For a fair comparison, all methods use "a/an" as the prefix for text prompts. Similar to [59], we employ a double-width PointNet++ model [67] to extract features and predict class probabilities for point clouds in ShapetNet [4]. Table 2 shows the comparison between VPP and other methods. It can be observed that our VPP outperforms all other methods substantially in all metrics. Notably, the FID metric of ours is much lower than other methods, demonstrating that our method generates higher quality and richer diversity of 3D shapes.
By employing a Shape As Points [52, 63] model pre-trained on multi-category ShapeNet data, VPP is capable of performing smooth surface reconstruction on generated point clouds. In Blender rendering, we employ varnish and automatic smoothing techniques to improve the display quality of the generated mesh. We qualitatively show in Fig. 5 that our method can generate higher-quality shapes and remain faithful to text-shape correspondence. It is noteworthy that VPP can generate smoother 3D shapes, e.g., the truck. Besides, our generation results are visually pleasing across different object categories with complex text descriptions. Therefore, the above advantages enable our method to be much more useful in practical applications.
Figure 6: Qualitative examples of single image conditioned 3D generation with VPP.
Table 2: Text conditioned 3D generation results on ShapeNetCore13 [4] dataset. Point-E [59] is pre-trained on a large-scale private point-text paired dataset containing millions of samples, and we report its zero-shot transfer results on ShapeNet. Accuracy (ACC), Fréchet Inception Distance (FID), and Inception Score (IS) are reported.
Method	ACC↑	FID↓	IS↑
CLIP-Forge [76]	83.33%	2425.25	-
CLIP-Sculptor [77]	87.08%	1480.11	-
Point-E (40M text-only) [59]	62.56%	85.37	7.93
VPP (ours)	88.04%	29.82	10.64
Image-Conditioned Generation By leveraging the CLIP features of 2D images as conditions in the Voxel Semantic Generator, VPP also possesses the capability of single-view reconstruction. Fig. 6 illustrates image-conditional generated samples of VPP. It can be observed that the generated point clouds are of detailed quality and visually consistent with the input image. For example, the flat wings of an airplane and the curved backrest of a chair.
3.2 Text-Conditioned 3D Editing and Upsampling
Text-conditioned 3D editing and upsample completion tasks are more challenging than most conditional 3D generation tasks, which most of previous methods can not realize due to the inflexible model nature. In contrast, our VPP can be used for a variety of shape editing and upsample completion without extra training or model fine-tuning.
Text-Conditioned 3D Editing Since we use the masked generative Transformer in VPP, our model can be conditioned on any set of point tokens: we first obtain a set of input point tokens, then mask the tokens corresponding to a local area, and decode the masked tokens with unmasked tokens and text prompts as conditions. We show the examples in Fig. 7 (a). The figure provides examples of VPP being capable of changing part attributes (view) and modifying local details (view) of the shape, which correctly corresponds to the text prompts.
Point Cloud Upsampling Except for editing, we also present our conditional generative model for the point cloud upsample completion, where the sparse point cloud inputs are completed as a dense output. We use the second stage Point Upsampler to achieve this, and the results are illustrated in Fig. 7 (b). It is observed that VPP generates completed and rich shapes of high fidelity while being consistent with the input sparse geometry.
3.3 Transfer Learning on Downstream Tasks
After training, the Mask Transformer has learned powerful geometric knowledge through generative reconstruction [30], which enables VPP to serve as a self-supervised learning method for downstream representation transferring. Following previous works [19, 60, 68], we fine-tune the Mask Point Transformer encoder, i.e., the second-stage Point Upsampler for 3D shape recognition.
Figure 7: Qualitative examples of text conditioned 3D editing and upsample completion with VPP.