SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We implement our Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) based on perceiver-based transformer architecture [23], where the 3D shape encoder consists of 1 cross-attention block and eight self-attention blocks. At the same time, the neural field decoder has 16 sefl-attention blocks with a final cross-attention block for the implicit neural field modeling. All attention modules are the transformer [62] style with multi-head attention mechanism (with 12 heads and 64 dimensions of each head), Layer Normalization (Pre-Norm) [3], Feed-Forward Network (with 3072 dimensions) [62] and GELU activation [17]. The learnable query embeddings are  $\mathbf{E} \in \mathbb{R}^{513 \times 768}$  with one head-class token for multi-modal contrastive learning and left 512 shape tokens with a linear projection layer to the VAE space  $\in \mathbb{R}^{512 \times 64}$  for the 3D shape reconstruction. Moreover, we employ pre-train encoders in the CLIP (ViT-L-14) [48] as our visual encoder and text encoder and freeze them during training and sampling. Besides, our aligned shape latent diffusion model (ASLDM) builds on a UNet-like transformer [52, 62, 5] consisting of 13 self-attention blocks with skip-connection by default. It contains 12 heads with 64 dimensions for each and 3076 dimensions in the Feed-Forward Network. Both models use an AdamW-based gradient descent optimizer [34] with a 1e-4 learning rate. Our framework is implemented with PyTorch [44], and we both train the SITA-VAE and ASLDM models with 8 Tesla V100 GPUs for around 5 days. We use the DDIM sampling scheduler [60] with 50 steps, which generates a high-quality 3D mesh within 10 seconds.
4.2 Datasets and Evaluation Metrics
Dataset. We use a standard benchmark, ShapeNet [11], to evaluate our model, which provides about 50K manufactured meshes in 55 categories. Each mesh has a category tag and corresponding texts, like fine-grained categories or brief descriptions given by the creator. We follow the train/val/test protocol with 3DILG [70]. We further collected 811 Cartoon Monster 3D shapes with detailed structures, with 615 shapes for training, 71 for validation, and 125 for testing, to evaluate the models' ability to generate complex 3D shapes. To prepare the triplet data (3D shape, image, text), we first
augment the provided texts in two ways. First, we string the shape tag and corresponding description in the format "a 3D model of (shape tag), in the style of (description)" or "a 3D model of (shape tag), (description)". Then, inspired by ULIP [67], we also leverage multiple templates containing 65 predefined phrases to provide more text information during training. As for the image data, we render each mesh under four camera poses, augmenting and improving the rendering diversity via the depth-condition-based ControlNet [73].
Metrics. We use the Intersection of Union (IoU) to reflect the accuracy of reconstructions. Then, we propose two new metrics for evaluating 3D shape generation methods. The first is a shape-image score (SI-S). We use a 3D shape encoder and image encoder to extract corresponding shape embedding and image embedding and compute the Cosine Similarity of these two modalities. Another is a shape-text score (ST-S), which computes the similarity between the generated 3D shape and the conditional text input in the aligned shape embedding and text embedding space. Both metrics evaluate the similarity between results and their corresponding conditions. Moreover, we use both the pre-trained ULIP [67] and our SITA to compute SI-S and ST-S, in terms of SI-S (ULIP), ST-S (ULIP), SI-S (SITA) and ST-S (SITA), respectively. Besides, we follow the metrics of P-IS and P-FID as introduced in Point-E [42] and use a pre-trained PointNet++ [47] to compute the point cloud analogous Inception Score [55] and FID [18] to evaluate the diversity and quality of the generated 3D shapes.
4.3 Experimental Comparison
Baselines. In the representation stage, we compare our method with Occ [37], ConvOcc [45], IF-Net [14], 3DILG [70] and 3DS2V [71] on reconstruction tasks to valid the ability of the model to recover a neural field given shape embeddings on the ShapeNet dataset [11]. For the conditional generation stage, we choose the baselines of two recent powerful 3D generation methods, 3DILG and 3DS2V. We first finetune their shape representation module on a mixture dataset of the ShapeNet and the 3D Cartoon Monster. Then, we retrain the text and image conditional generative models of 3DILG and 3DS2V with all the same protocols as ours.
Overall	Selected	Table	Chair	Airplane	Car	Rifle	Lamp
OccNet [37]	0.825	0.81	0.823	0.803	0.835	0.911	0.755	0.735
ConvOccNet [45]	0.888	0.873	0.847	0.856	0.881	0.921	0.871	0.859
IF-Net [14]	0.934	0.924	0.901	0.927	0.937	0.952	0.914	0.914
3DILG [70]	0.950	0.948	0.963	0.95	0.952	0.961	0.938	0.926
3DS2V [71]	0.955	0.955	0.965	0.957	0.962	0.966	0.947	0.931
Ours	0.966	0.964	0.965	0.966	0.966	0.969	0.967	0.95
Table 1: Numerical results for reconstruction comparison on IoU(↑, a larger value is better). The results show that our model performs best in 55 categories. The results of selected categories further prove that our model could reconstruct each category faithfully.
Image-Conditioned	Text-Conditioned
SI-S (ULIP)↑	SI-S (SITA)↑	P-FID↓	P-IS↑	ST-S (ULIP)↑	ST-S (SITA)↑	P-FID↓	P-IS↑
3DILG	9.134	11.703	4.592	12.247	10.293	6.878	10.283	12.921
3DS2V	13.289	15.156	2.921	12.92	12.934	9.833	5.704	13.149
Ours	13.818	15.206	1.586	13.233	16.647	13.128	2.075	13.558
Table 2: Numerical results for conditional generation comparison. The results show that our model achieves the best generative performance. The SI-S and ST-S indicate that our model generates high-fidelity results by well-mapping the condition information to its related 3D shapes. Moreover, P-FID reflects that our model generates the most realistic 3D shapes, and P-IS indicates that the generated samples are diverse.  $\uparrow$  means a larger value is better, and  $\downarrow$  otherwise.
Numerical Comparison. We report the numerical results in Table 1 and Table 2. Table 1 shows that our model has the best reconstruction performance on 55 overall categories, surpassing the rest. Results of the selected category further prove that our model could faithfully reconstruct 3D shapes in each of the 55 categories. Table 2 reports the numerical results for conditional 3D shape generation. Our model achieves the best on all the SI-S and ST-S, indicating that it could map the information from the image or text to its corresponding 3D shape information for generating high-fidelity results. Moreover, the P-FID proves that our model could produce high-quality shape-tokens for generating realistic 3D shapes, and P-IS indicates the diversity of the samples. Specifically, the four left columns
show that our model surpasses the baselines on image-conditioned generation, proving that our model can better map visual information to 3D shapes. The four right columns validate the generative quality of text-conditioned generation. Since natural language, compared to the 2D image, usually provides limited and abstract information, learning a model to map text information to the 3D shape is challenging. However, benefiting from training on the aligned latent space, our model significantly improves text-conditioned generation, shown in the right of Table 2, which reflects that our model well-maps natural language information to 3D shapes and generates diverse and high-quality results.
Figure 3: Visual results for image-conditioned generation comparison. The figure shows that 3DILG [70] generates over-smooth surfaces and lacks details of shapes, whereas 3DS2V [71] generates few details with noisy and discontinuous surfaces of shapes. In contrast to baselines, our method produces smooth surfaces and portrays shape details. Please zoom in for more visual details.
Figure 4: Visual results for text-conditioned generation comparison. In the first two rows, we test the model with abstract texts, and the result shows that only our model could generate a 3D shape that conforms to the target text with a smooth surface and fine details. The last two rows show the result given texts containing detailed descriptions, which further shows that our model could capture the global conditional information and the local information for generating high-fidelity 3D shapes. Keywords are highlighted in red; please zoom in for more visual details.
Visual Comparison. The visual comparisons of the image- and text-conditional 3D shape generations are illustrated in Figure 3 and Figure 4. Figure 3 shows that 3DILG [70] pays more attention to the global shape in the auto-regressive generation process, where its results lack depictions of details of
3D shapes. While 3DS2V [71] generates more details of 3D shapes, the results have discontinuous or noisy surfaces. Besides, both methods are unstable to generate a complete shape when the given conditions map to a complex object, fine machine, or rare monster. Figure 4 shows the visual comparison of text-conditional generation. In the upper-half rows, we show the results given simple and abstract concepts, while in the lower-half rows, we show the results given detailed texts like descriptions for deterministic parts of the target shape. Similar to the observation above, 3DILG [70] generates an over-smooth shape surface with fewer details, and 3DS2V [71] produces fewer details on the discontinuous object surface. Therefore, only our model produces correct shapes that conform to the given concepts or detailed descriptions with delicate details on smooth surfaces.
4.4 Ablation Studies and Analysis
We ablation study our model from three perspectives: the effectiveness of training generative model in the aligned space, the effectiveness of vision-language models (VLMs) on the SITA-VAE, and the impact of learnable query embeddings.
The effectiveness of training generative model in the aligned space. We perform a visual comparison for ablation study the effectiveness of training the generative model in the aligned space, as illustrated in the Figure 5. The uppers are sampled from the generative model that trains in the aligned space, while the lowers are sampled from the generative model that trains without aligned space. It proves that the uppers conform to the given text and the lower do not, which indicates that training the generative model in the aligned space leads to high-fidelity samples.
a 3d model of table, regulation table, tennis table, bench. a 3d model of vintage car, 1938 Alfa Romeo 8C 2900B.
Figure 5: Ablation study the effectiveness of training generative model in the aligned space. This figure illustrates visual comparisons for ablation studies on the effectiveness of training the generative model in the aligned space. Compared with the lower samples based on the conditional texts, the upper samples are closer to the conditions semantically, which indicates the effectiveness of the training generative model in the aligned space.
The effectiveness of vision-language models. Except for the well-known vision-language model (VLM) CLIP [48], we introduce another vision-language model (VLM) SLIP [40] for training the SITA-VAE for a comprehensive comparison. First, we evaluate the impact of the vision-language model on SITA-VAE's reconstruction ability, and the results are shown in Figure 6. It shows that our model composed with CLIP achieves the best performance. Then, we evaluate the vision-language model's impact on the ability to align multi-modal space. We select standard and zero-shot classification tasks to reflect the impact of the vision-language model. Note that the classification is performed by a feature matching operation, where we provide multiple 3D shapes and phrases to the SITA-VAE; it returns the similarity between 3D shapes to each phrase as classification results, which indicates that the more the multi-modal space is aligned, leading the higher classification accuracy. The results show that our model composed with CLIP achieves the best performance.
The impact of the learnable query embeddings. We ablation study learnable query embeddings with the same experiments as the above, and the results show that using 512 learnable query embeddings leads to the best performance on reconstructions and classifications.
4.5 Nearest Neighbor Analysis
In order to verify whether the model learns to generate results based on the given conditions or memorizes the whole training split of the dataset, we traverse the training set to retrieve the nearest
Figure 6: Ablation study the effectiveness of vision-language models and the impact of learnable query embeddings. This figure shows the ablation study on the effectiveness of the vision-language model and the impact of learnable query embeddings. According to the table, our model with CLIP and 512 learnable query embeddings achieves the best reconstruction and classification performance, indicating its ability to recover 3D shapes and align multi-modal space.
neighbors of the generated samples, as illustrated in Figure 7. Specifically, we exhibit three nearest neighbors for each sample for comparison. According to the visualization, the generated 3D shape differs from each retrieved nearest neighbor, demonstrating that our model hallucinates a new 3D shape rather than overfitting the training sets.
Figure 7: Nearest Neighbor Analysis. We traverse the whole training set to find three nearest neighbors for the generated 3D shapes, and the results reveal that our model could produce novel 3D shapes based on given images instead of memorizing specific ones.
Though our method has achieved excellent results, it still has some limitations. First, our method needs the ground truth 3D shapes from training, while 3D data is usually an order of magnitude smaller than the 2D data. Learning the shape representation with a 3D shape-image-text aligned space from only 2D (multi-view) images via differentiable rendering is a promising direction. Furthermore, since we represent each 3D shape as an occupancy field, it needs to convert the 3D mesh into a watertight one, which will inevitably degrade the original quality of the 3D mesh.
In conclusion, we propose a novel framework for cross-modal 3D shape generation that involves aligning 3D shapes with 2D images and text. We introduce a new 3D shape representation that can reconstruct high-quality 3D shapes from latent embeddings and incorporate semantic information by aligning 3D shapes, 2D images, and text in a compatible space. This aligned space effectively closes the domain gap between the shape latent space and the image/text space, making it easier to learn a better probabilistic mapping from the image or text to the aligned shape latent space. As a result, our proposed method generates higher-quality and more diverse 3D shapes with greater semantic consistency that conform to the conditional image or text inputs.
Acknowledgement The work was supported by NSFC #61932020, #62172279, Program of Shanghai Academic Research Leader, and "Shuguang Program" supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission