Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
Authors: Zibo Zhao
Wen Liu
Xin Chen
Xianfang Zeng
Rui Wang
Pei Cheng
BIN FU
Tao Chen
Gang YU
Shenghua Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: Conditional 3D Shape Generation
Neural 3D Representation
3D Reconstruction
TL;DR: We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts.

Abstract: We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
Zibo Zhao!in 12+
Wen Liu
Xin Chen Bin Fu²
Xianfang Zeng? Tao Chen
Rui Wang
Pei Cheng
Gang Yu?
Shenghua Gao 1,4,5f
ThangghaiTec University
"Tencent PCG, China
School of Information Science and Technology. Fudan University, China
Shanghai Engineering Research Center of Intelligent Vision and Imaging
Shanghai Engineering Research Center of Energy Efficient and Custom IC
Michelangelo
1900 bais My honethopee Civientaiio)
Sorw
Figure 1: Visualization of the 3D shape produced by our framework, which splits into triplets with conditional input the left, normal map the middle, and triangle mesh on the right. The generated 3D shapes semantically conform to the visual or textural conditional inputs.
Abstract
We present a novel alignment- -eeoooo-eooogeegtionnn approach to tackle challeng- ing task of generating general 3D shapes based on 2D images texts. Directly lcarning a conditional generative model from images or texts to 3D shapes prone to produce inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and fa- cilitate
conditiood 3D shape generation, we explore representing 3D shapes in shape-image wataaae-eet--llnneeee space. Our framework comprises models: a Shape-Image-Texxx Text- Aliggnd Variational Auto-Encoder (SITA-VAB) and
#Contributed equally and work done while Zibo Zhao was Research Intern with Tencent PCG. |Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embed- dings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textu- ral conditional inputs, validating the effectiveness of the shape-image-tex--llineee space for cross- moos-mto 3D shape generation.
1 Introduction
Conditional generative model-based 3D shape generations, such as GAN [9. 35, 65], VAE [8, 36, 6]. Auto-Regressie model [68, 39, 70], and Diffusion- model [69, 41, 15, 13, 28, 42, 25], have great potential to increase productivity in the asset design of games, AR/VR, film production, the furniture industry, manufacturing, and architecture construction. However, two obstacles limit their ability to produce high-quality and diverse 3D shapes conforming to the conditional inputs: 1) diverse 3D shape topologies are complicated to process into a neural eewoor--rreendly representation: 2) since generating a high-quality 3D shape from a 2D image or textual description is an ill-pose problem, and also the distribution between the 3D shape space and image or text space is quite different, is hard to learn a probabilistic mapping function from the image or text to 3D shape.
Recently, the neural fields in terms of occupancy [37, 45], Signed Distance Function (SDF) [43], and radiance field [38] have been driving the 3D shape representation in the computer vision and graphies community because their topology-free data structure, such as global latent [43], regular grid latent [45, 14], and point latent [69, 70], are easier to process for neural networks in an implicit functional manner. Once arrive at compatible space to represent different topological 3D shapes, in light of the great success of auto-regressive and diffusion-based models in audio [26, 27J, image [50, 51, 49, 54, 4J, video [63, 59, 19, 71, and 3D human motions [72, 61, 66], a conditional auto-regressive or diffusion-basse image or text. However, generating high-quality 3D shape from a 2D image or textual description is an ill-posed problem, and it usually requires more prior information for 3D shapes. In contrast, the distribution of the 3D shape space is significantly different from the 2D image or text space, and directly learning probabilistic mapping function from the image or text to the 3D shape might reduce the quality, diversity, and semantic consistency with the conditional inputs. Prior research [69, 42] has endeavored to mitigate this concern through a coarse-to -oorse-to-foee fo-fine approach, whereby the initial step involves generating coarse point clouds as an intermediary representation, followed by the regression of neural field based on the point cloud.
generative model [15, 69, 70] is learned to sample a 3D shape in latent from an
Unlike the previous 3D shape representation. where the neural fields only characterize the geometric information of each 3D shape and capture the shape distribution by regularizing the shape latent with KL-divergence via VAE [13, 28, 71] or VQ- (Q-VAE [39, 70], we investigate a novel 3D shape representation that further brings the semantic information into the neural fields and designs Shape- Mmate-Text-Allgneee Variational
(SITA-VA VAE). Specifically, it uses a perceiver- based transformer [62, 23] to encode each 3D shape into the shape embeddings and utilizes a contrastive learning loss to align the 3D shape embeddings with pre-aligned CLIP [48] image/text feature space neural implicit decoder reconstructs
from large-scale image-text pairs. After that, a transformer-base the shape of latent embeddings to a neural occupancy or SDF field with a high-quality 3D shape. With the help of the aligned 3D shape, image, and text space, which closes the domain gap between the shape latent space and the image/text space, we propose an Aligned Shape Latent Diffusion Model (ASLDM) with UNet-like skip connection-basee transformer architecture [52, 5] to learn a better probabilistic mapping from the image or text to the aligned shape latent space and thereby generate a higher-quality and more diverse 3D shape with more semantic consistency conforming to the conditional image or text inputs.
To summarize, we explore bridging the semantic information into 3D shape representation via aligning the 3D shape, 2D image, and text into a compatible space. The encoded shape latent embeddings could also be decoded to neural field that preserves high-quality details of a 3D shape. Based on the powerful aligned 3D shape, image, and text space, we propose an aligned shape latent diffusion model to generate a higher-quality 3D shape with more diversity when given the image or text input.
2

---

We perform extensive experiments on a standard 3D shape generation benchmark, ShapeNet [11], and further collected 3D Cartoon Monster dataset with geometric details to validate the effectiveness of our proposed method. All codes will be publicly available.
2 Related Work
2.1 Neural 3D Shape Representation
Neural Fields have dominated the research of recent 3D shape representation, which predict the occupancy [37, 45], Sign Distance Function (SDF), density 57]. or feature vectors [10] of each 3D coordinate in the field via a neural network to preserve the high-fidelity of a specific 3D shape in topology-ficee way. However, the vanilla neural field can only model single 3D shape and cannot be generalized to other shapes. To this end, the researchers usually take additional latent codes, such as a global latent [43], regular latent grid [45, 14], set of latent points [69, 70], or latent embeddings [71. 25], which describe a particular shape along with each 3D coordinate to make the neural fields generalizable to other 3D shapes scenes. Though current neural fields' 3D representation can characterize the low-level shape geometry information and preserve the high- fidelity shape details, bringing the high-level semantic information into the neural fields is still relatively poorly studied problem. However, semantic neural fields are significant to downstream tasks, such conditional 3D shape generations and 3D perception [22, 58].
2.2 Conditional 3D Shape Generation
Optimization-based approaches which employ text-image matching loss function to optimize a 3D representation the neural radiance field (NeRF). Dreamfields and AvatarCLIP [24, 21J adopt a pre-trained CLIP [48] model to measure the similarity between the rendering image and input text as the matching objective. On the other hand, DreamFusion [46] and Magie3D [29] utilize a powerful pre-trained diffusion-basee complex and view-consistent results. However, per-scene optimization-bant low success rate and long optimization time in hours to generate high-quality 3D shape. However, they only require a pre-trained CLIP or text-to-image model and do not require any 3D data.
text-to-image model as the optimization guidance and produce more
methods suffer from a
Optimization-free methods [33, 31, 32] are an alternative approach to conditional 3D shape gen- eration that leverages paired texts/3D shapes or images/3D shapes directly learn a conditional generative model from the text or image to the 3D shape representations. CLIP-Forge [56] employs an invertible normalizing flow model learn distribution transformation from the CLIP image/text embedding to the shape embedding. AutoSDF [39], ShapeFormer [68], and 3DILG [70] explore an auto-regressive model to learn a marginal distribution of the 3D shapes conditioned on images or texts and then sample a regular grid latent irregular point latent shape embeddings from the conditions. In recent years, diffusion-based generative models have achieved tremendous success in text-to-image, video, and human motion generation. Several contemporaneous works, including SDFusion [13], Diffusion-SDD [28, 15], 3D-LDM [41]. 3DShape2 VecSet [71], and Shap-E [25], propose to learn a probabilistic mapping from the textual or visual inputs the shape latent embeddings via diffusion model. Since these approaches Iearn the prior information of the 3D shape data, they could improve the yield rate of high-quality shape generation. Moreover, there is no long-time optimization process, and the inference time is orders of magnitude faster than the optimization based approaches. However, directly learning conditional generative model to sample the 3D shape from the conditions might produce low qualityy less- diversstyyt results due to the significant distribution gap between the shape and image/text spaces.
2.3 Contrastive Learning in 3D
Contrastive Language- Language-mmage Pre-training (CLIP) [48] has emerged fundamental model in 2D visual recognition tasks and cross-modal image synthesis by building the representation connection between vision and language within an aligned space. Recent works [12, 1, 53, 64] have extended the multi-modal contrastive learning paradigm to 3D. CrossPoint [2] learns the 3D-2D alignment to enhance the 3D point cloud understanding. PointCI.IP [74] takes full advantage of the CLIP model pre-trained on large-scale image-text pairs and performs alignment between CInPeedcooee point cloud and 3D category texts to generalize the ability of 3D zero-shot and few-shot classification.
3

---

Query a):
-aae--e-e---e---lggeee Variational Auto-Encoder (SITA- VAE)
Recomstruced
Volune Poim (ky.2)
Mesh
14
3D Shape
3D Shape Decoder
Oceapancy
Marching Cabes
Encoder
Shape Larent
br
Embeddings
Imoage
CLIP Image Eneoder
Littppsimage: Conzasive
Liphape,tezt
CLIP Text Encoder
3d model guilor Kil-Ouija
Instrument
A):gnee Shape Latent Diffusion Model (ASLDM).
Figure 2: Miggament-befor--eeeeeeee pipeline. Our method contains two models: the Shape- Image-Text-Alignee Variational Auto- Eicoder (SITA-VAE) and the Aligned Shape Latent Diffusion Model (ASLDM). The SITA-VAL consists of four modules: an image encoder, text encoder, a shape encoder, and 3D shape decoder. Encoders encode inputs pair into aligned space, and the 3D shape decoder reconstructs 3D shapes given embeddings from the aligned space. The ASLDM maps the image text condition to the aligned shape latent space for sampling a high-quality 3D shape embedding, which is later reconstructed to high-fidelity 3D shapes by the 3D shape decoder.
ULIP [67]. CLIP-goes-3) [16] and OpenShape [30] further learn unified and aligned representation of images, texts. and 3D point clouds by pre- pre-training with object triplets from the three modalities to improve 3D understanding. While most of these works focus on 3D recognition tasks, establishing the connection between 3D recognition and generation tasks remains under-explored problem.
3 Our Approach
The direct learning of a probabilistic mapping from images or texts to 3D shapes is prone to produce inconsistent results due the significant distribution gap between the 3D shapes and the 2D images and texts. To address this issue, we propose an 3D shape generation, as illustrated in Figure Our approach involves models: the Shape-Image- Text-Aligned Variational Auto-Encoder (SITA-VABH) SITA-VAE)(Secting
wewtttteeooooorreaattiiiee solution for cross-modal
3.1) and the Aligned Shape Latent Diffusion Model (ASLDM) (Section 3.2). The former model learns an alignment among the 3D shapes, images, and texts via contrastive learning and then reconstructs the shape embeddings back the neural field. The latter model is based on the aligned space and designed to learn better conditional generative model from the images or texts to shape latent embeddings. By adopting this alignment- before- generrtignn approach, we aim overcome the challenges posed by the distribution gap and produce more consistent and high-quality results in cross- moos--mdaal 3D shape generation.
3.1 Shape-Image-Ter Text Aligned Variational Auto-Encoder
Our SITA-VAE contains four components: pre-trained and fixed CLIP image encoder & and CLIP text encoder ε,, a trainable 3D shape encoder & and neural field decoder D.. The CLIP image encoder and text encoder take 2D images I E H XXX x3 and tokenized texts T E Lixd as input, and generate image tokens E, E R (1++)))x and text tokens E, E IR xdd where (1 + L,) and L, are sequence length of image tokens E, and text tokens E,, We take advantage of the pre-trained image encoder and text encoder from CL.P. These two encoders are trained on large-scale image-text pairs and robust enough to capture a well-aligned vision-langue lanounl space, which will enrich the semantics of the 3D shape representation after multi-modal alignment via contrastive learning.

---

3D shape encoder aims to extract powerful feature representations to characterize each 3D shape effectively. To achieve this, we first sample point clouds P E RxVx(3-C) from the surface of 3D shapes, where N represents the number of points, and C' denotes additional point features such as normal or color. Next, we use a linear layer to project the concatenation of the Fourier positional encoded point clouds P to the 3D shape encoder input X E R2 Drawing inspiration from previous transformer- based architectures for point cloud understanding [23], we build our 3D shape encoder on a perceiver- based transformer. Specifically, we use a cross- aotss-tttnntoon layer to inject the 3D shape information from the input X into series of learnable query tokens Q R(++,,) where 1 1+ L, is the length of query tokens Q consisting of one global head token Q2 E IR 1xd with high- level semantics and Lg local tokens Q E RLxd containing low-level geometric structure information. Then, several self-attention blocks are used iteratively improve the feature representation and obtain the final shape embeddings, E, R (1++.))xd
Alignment among 3D shapes, images, and texts plays a crucial role in SITA- VAF and the conditional generative models. Since the 3D data is the order of magnitudes smaller than the images and texts data, to learn a better-aligned shape among 3D shapes, images, and texts, we enforce the 3D shape encoder close to pre-aligned vision vision-language space, which is pre-trained on a large-scale image-text pair with rich image and text representations by leveraging the contrastive learning strategy. Consider an imput pair of 3D shapes X, images I and tokenized texts T. The triplet encoders generate the corresponding shape embedding es, image embedding e; and text- text-enbedding e, by projecting the extracted shape tokens E,, image tokens E, and text tokens E, as three vectors with the same dimension, which is expressed as: e, = F. (E.), e; = (E,), and e, = (E,), where J, is a learnable shape embedding projector, image embedding projector F, and text embedding projector F. are pre-trained and frozen during training and inference. The contrastive loss is:
1 Σ (log exp(e(ef)
+ log exp(e,e,) Eexple
L (shape,image)
Σ exp(esee)
(1.b)
(I)
1
Σ (log exp(c!o*)
log exp(c!at)
L ((shoee,teat)
2
Y exp((ee)
Σ Eexp(etet)
(J.6)
where (j. k) indicates the positive pair in training batches, and since we utilize pre-trained encoders from CLIP, the model is free from constraint (image,teer))
3D shape decoder, D.. takes the shape embeddings E, as inputs to reconstruct the 3D neural field in high quality. We use the KL divergence loss & facilitate the generative process maintain the latent space as a continuous distribution. Besides, we leverage a projection layer to compress the latent from dimension projection layer is used to transform the sampled latent from dimension do back to high dimension d for reconstructing neural fields of 3D shapes. Like the encoder, our decoder model also builds on a transformer with the cross-attentio its corresponding shape latent embeddings E,, the decoder computes cross attention iterative for predicting the occupancy of the query point (xx). The training loss is expressed as:
lower dimensions dg for compact representation. Then, another
mechanism. Given query 3D point x E I in the field and
L RR [BCE(D(x EE,), (x))),
where BCE binary cross-entropy loss, and the total loss for training Shape-Image-1 Aligned Variational Auto-Encoder (SITA) is written as:
ASITA (shape,, & (sshap,)tor)) + Lrr + AKI
3.2 Aligned Shape Latent Diffusion Model
After training the SITA-VAE. we obtain an alignment space among 3D shapes, images, and texts, as well as 3D shape encoder and decoder that compress the 3D shape into low-dimensional shape latent embeddings and reconstruct shape latent embeddings to a neural field with high quality. Building on the success of the Latent Diffusion Model (LDM) [51] in the text-to-image generation, which strikes a balance between computational overhead and generation quality, we propose shape latent diffusion model on the aligned space to learn a better probabilistic mapping from 2D images or texts to 3D shape latent embeddings. By leveraging the alignment space and the shape latent diffusion model, we can generate high-quality 3D shapes that better conform to the visual textural conditional inputs.
5

---

Our Aligned Shape Latent Diffusion Model (ASLDM) builds on a UNet-like transformer [52, 62, 5], aim to fit distribution of the shape latent embeddings, accompanied by an auto-encoder for encoding data samples into the latent space and reconstructing the data samples given the sampled latent. By learning in the latent space, the latent diffusion model is computationally efficient, and leveraging such a compact representation enables the model to fit the target distribution faster. Specifically, the model eq focuses on generating shape latent embeddings E, conditioned on C, which is represented by the CLIP image or text encoder. Following LDM [51], the objective is
L -E,,,-NN0.1,
co (E C)
where i is uniformaly samppled from {1,..., T and E.") is noisy version of E. .During inference, sampling a Gaussian noise, the model gradually denoises the signal until reaching E Followed with classifier-free guidance (CFG) [20], we train our conditional latent diffusion model with classifier-fis free guidance. In the training phase, the condition C randomly converts to an empty set 0 with a fixed probability 10%. Then, we perform the sampling with the linear combination of conditional and unconditional samples:
co (E C,t) = co (E!", t) + M(co (E!", C,t) - co (E!", 0, f)),
(5)
where X is the guidance scale for trading off the sampling fidelity and diversity.
4 Experiments
To validate the effectiveness of our proposed framework, we conducted extensive experiments. In this section, we provide implementation details of our model in Section 4.1. We also describe the data preparation process, including comparisons with baselines and metrics used in our evaluation, in Section 4.2. Of particular importance, we present quantitative comparison results to validate our model's generation ability. Additionally, we provide visual comparison results to illustrate the quality of the generative outputs in Section 4.3, Also, we conduct ablation studies in Section 4.4 to validate the effectiveness of training the generative model in the aligned space, the effectiveness of pre- trainee dision-languaa models (VLM) on the SITA-VAY and the impact of learnable query embeddings.
4.1 Implementations
We implement our Shape-InaggeTText-Aligee Variational Auto-Encoder (SITA-VAB) based on perceiver-based transformer architecture [23], where the 3D shape encoder consists of 1 cross- attention block and eight self-attention blocks. At the same time, the neural field decoder has 16 sefl-attention blocks with final cross-attention block for the implicit neural field modeling. All attention modules are the transformer [62] style with multi-head attention mechanism (with 12 heads and 64 dimensions of each head), Layer Normalization (Pre-Norm) [3], Feed-Forwarrd Network (with 3072 dimensions) [62] and GELU activation [17]. The learnable query embeddings are E E 5113 768 with one head-class token for multi- modal contrastive learning and left 512 shape tokens with a linear projection layer to the VAF space E 1112261 for the 3D shape reconstruction. Moreover, we employ pre-train encoders in the CL.PP (ViT-L-144) [48] as our visual encoder and text encoder and freeze them during training and sampling. Besides, our aligned shape latent diffusion model (ASLDM) builds on a UNet-like transformer [52, 62, 5] consisting of 13 self-attention blocks with skip-connection by default. It contains 12 heads with 64 dimensions for each and 3076 dimensions in the Feed-Forward Network. Both models use an AdamW-basee Our framework is implemented with Py PyTcc [44], and we both train the SITA-VAE and ASLDM models with 8 Tesla V100 GPUs for around 5 days. We use the DDIM sampling scheduler [60] with 50 steps, which generates a high-quality 3D mesh within 10 seconds.
gradient decent optimizer [34] with le-4 learning rate.
4.2 Datasets and Evaluation Metrics
Dataset. We use a standard benchmark, ShapeNet [11], to evaluate our model, which provides about 50K manufactured meshes in 55 categories. Each mesh has a category tag and corresponding texts, like fine- ane-grained categories or brief descriptions given by the creator. We follow the train/val/test protocol with 3DILG [70]. We further collected 811 Cartoon Monster 3D shapes with detailed structures, with 615 shapes for training, 71 for validation, and 125 for testing, to evaluate the models' ability to generate complex 3D shapes. To prepare the triplct data (3D shape, image, text), we first

---

augment the provided texts in two ways. First, we string the shape tag and corresponding description in the format "a 3D model of (shape tag), in the style of (description)" or "a 3D model of (shape lag), (description)". Then, inspired by ULIP [67], we also leverage multiple templates containing 65 predefined phrases to provide more text information during training. As for the image data, we render each mesh under four camera poses, augmenting and improving the rendering diversity via the depth-condition-baat ControlNet [73].
Metries. We use the Intersection of Union (IoU) to reflect the accuracy of reconstructions. Then, we propose two new metries for evaluating 3D shape generation methods. The first is a shape-image score (SI-S). We use a 3D shape encoder and image encoder to extract corresponding shape embedding and image embedding and compute the Cosine Similarity of these two modalities. Another a shape-text score (ST-S), which computes the similarity between the generated 3D shape and the conditional text input in the aligned shape embedding and text embedding space. Both metrics evaluate the similarity between results and their corresponding conditions. Moreover, we use both the pre-trained ULIP [67]
and our SITA to compute SI-S and ST-S, in terms of SI-S (ULIP), ST-S (ULIP), SI-S (SITA) and ST-S (SITA), respectively. Besides, we follow the metrics of P-IS and P-FID as introduced in Point-E [42] and use a pre-trained PointNet++ [47] to compute the point cloud analogous Inception Score [55] and FID [18] to evaluate the diversity and quality of the generated 3D shapes.
4.3 Experimental Comparision
Baselines. In the representation stage, we compare our method with Oce [37], ConvOcc [45], IF- Net [14], 3DILG [70] and 3DS2V [71] on reconstruction tasks to valid the ability of the model to recover a neural field given shape embeddings on the ShapeNet dataset [11]. For the conditional generation stage, choose the baselines of two recent powerful 3D generation methods, 3DILG and 3DS2V. We first finetune their shape representation module on mixture dataset of the ShapeNet and the 3D Cartoon Monster. Then, we retrain the text and image conditional generative models of 3DILG and 3DS2V with all the same protocols as ours.
Overall
Selected
Table
Chair
Airplane
Car
Rifle
Lamp
OccNet [37]
0,825
0.81
0,823
0,803
0,835
0.911
0.755
0.735
ConvOceNet [45]
0.888
0.873
0.847
0.856
0.881
0.921
0.871
0.859
IF-Net [14]
0.934
0.924
0.901
0.927
0.937
0.952
0.914
0.914
3DILG [70]
0.950
0.948
0.963
0.95
0.952
0.961
0.938
0.926
3DS2V [71]
0.955
0.955
0.965
0.957
0.962
0.966
0.947
0.931
Ours 0.966
0.964
0.965
0.966
0.966
0.969
0.967
0.95
Table I: Numerical results for reconstruction comparison on IoU(t, a larger value is better). The results show that our model performs best in 55 categories. The results of selected categories further prove that our model could reconstruct each category faithfully.
Image-Conee Condit-ond
Text-Conditioned
SI-S (ULIP))
SI-S (SITA))
P-FID]
P-ISt
ST-S (ULIP))
ST-S (SITA))
P-FID.
P-IS
3DILG
9.134
11.703
4.592
12.247
10.293
6.878
10,283
12,921
3DS2V
13.289
15.156
2.921
12.92
12.934
9.833
5.704
13.149
Ours
13.818
15.206
1.586
13.233
16.647
13.128
2.075
13,558
Table 2: Numerical results for conditional generation comparison. The results show that our
model achieves the best generative performance. The SI-S and ST-S indicate that our model generates high-fidelity results by well-mapping the condition information to its related 3D shapes. Moreover, P-FID reflects that our model generates the most realistic 3D shapes, and P-IS indicates that the generated samples are diverse. means larger value is better, and 1 otherwise.
Numerical Comparison. We report the numerical results in Table and Table 2. Table shows that our model has the best reconstruction performance on 55 overall categories, surpassing the rest. Results of the selected category further prove that our model could faithfully reconstruct 3D shapes in each of the 55 categories. Table 2 reports the numerical results for conditional 3D shape generation. Our model achieves the best on all the SI-S and ST-S, indicating that it could map the information from the image or text to its corresponding 3D shape information for generating high-fidelity results. Moreover, the P-FID proves that our model could produce high-quality shape-token for generating realistic 3D shapes, and P-IS indicates the diversity of the samples. Specifically, the four left columns
7

---

show that our model surpasses the baselines on image-conditioned generation, proving that our model can better map visual information to 3D shapes. The four right columns validate the generative quality of text-conditioned generation. Since natural language, compared to the 2D image, usually provides limited and abstract information, learning a model map text information the 3D shape is challenging. However, benefiting from training the aligned latent space, our model significantly improves eexl-conditioed generation, shown the right of Table 2, which reflects that our model well-maps natural language information to 3D shapes and generates diverse and high-quality results.
Image
Image
Image
Condition
3DILG
3DS2V
Ours
Condition
3DILG 3DS2V
Condition
3DILG 3DS2V
Ours
Figure 3: Visual results for image-conditioned generation comparison. The figure shows that
3DILG [70] generates over-smooth surfaces and lacks details of shapes, whereas 3DS2V [71] generates few details with noisy and discontinuous surfaces of shapes. In contrast baselines, our method produces smooth surfaces and portrays shape details. Please zoom in for more visual details.
Text Condition 3DILG 3DS2V Ours
Text
Text
Condition
3DILG 3DS2V
Ours
Condition
3DILG 3DS2V
Ours
ekglase Piyang Anrrigh
orspiie
vanel
dett Bisqueo"
Leather Suncare
fasgg monnter Cate sonntr the fire ffiing tods
sasshines
Cate monter in foom sourtias anaoe mik feet and basil"
fusaroid wanser, jed 7 hooai"
sanding mueusorr Cute Star dipord wossor jevt
bug Aageroootrr. filng doonner form of suurgaiao
th of dragon milk 2 fos and Plansii
Menotte model
Figure 4: Visual results for text-conditioned generation comparison. In the first rows, test the model with abstract texts, and the result shows that only our model could generate 3D shape that conforms the target text with smooth surface and fine details. The last result given texts containing detailed descriptions, which further shows that our model could capture the global conditional information and the local information for generating high-fidelity 3D shapes. Keywords are highlighted in red; please zoom in for more visual details.
show the
Visual Comparison. The visual comparisons of the image- and text-conditional 3D shape generations are illustrated in Figure and Figure 4. Figure shows that 3DIL. [70] pays more attention to the global shape in the auto-regressive generation process, where its results lack depictions of details of

---

3D shapes. While 3DS2V [71] generates more details of 3D shapes, the results have discontinuous or noisy surfaces. Besides, both methods are unstable to generate complete shape when the given conditions map to complex object, fine machine, or rare monster. Figure 4 shows the visual comparison of text-conditional generation. In the upper-half rows, we show the results given simple and abstract concepts, while in the lower-half rows, we show the results given detailed texts like descriptions for deterministic parts of the target shape. Similar to the observation above, 3DILG [70] generates an over-smooth shape surface with fewer details, and 3DS2V [71] produces fewer details on the discontinuous object surface. Therefore, only our model produces correct shapes that conform to the given concepts or detailed descriptions with delicate details smooth surfaces.
4.4 Ablation Studies and Analysis
We ablation study our model from three perspectives: the effectiveness of training generative model in the aligned space, the effectiveness of vision-language models (VLMs) on the SITA-VAE VAE, and the impact of learnable query embeddings.
The effectiveness of training generative model in the aligned space. We perform a visual com- parison for ablation study the effectiveness of training the generative model in the aligned space, as illustrated in the Figure 5. The uppers are sampled from the generative model that trains in the aligned space, while the lowers are sampled from the generative model that trains without aligned space. It proves that the uppers conform to the given text and the lower do not, which indicates that training the generative model in the aligned space leads to high-fidelity samples.
3d model of table,
table, tennis table, bench.
3d model of vintage 1938 Alfa Romeo 8C 2900B
a) With Aligned Space
b) Without Aligned Space
Figure 5: Ablation study the effectiveness of training generative model in the aligned space. This figure illustrates visual comparisons for ablation studies on the effectiveness of training the generative model in the aligned space. Compared with the lower samples based on the conditional texts, the upper samples are closer the conditions semantically, which indicates the effectiveness of the training generative model in the aligned space.
The effectiveness of vision-language models. Except for the well-known vision- lssion-angguage model (VLM) CLIP [48], we introduce another vision-language model (VLM) SLIP [40] for training the SITA-VAE for a comprehensive comparison. First, we evaluate the impact of the vision-language model on SITA-VAE' reconstruction ability, and the results are shown in Figure 6. It shows that our model composed with CLIP achieves the best performance. Then, we evaluate the vision- language model's impact the ability to align multi-modal space. We select standard and zero-shot model. Note that the classification is
classification tasks to reflect the impact of the vision-languee performed by feature matching operation, where we provide multiple 3D shapes and phrases to the SITA-VAE: it returns the similarity between 3D shapes to each phrase as classification results, which indicates that the more the multi-modal space is aligned, leading the higher classification accuracy. The results show that our model composed with CLIP achieves the best performance.
The impact of the learnable query embeddings. We ablation study learnable query embeddings with the same experiments as the above, and the results show that using 512 learnable query embeddings leads to the best performance reconstructions and classifications.
4.5 Nearest Neighbor Analysis
In order to verify whether the model learns to generate results based on the given conditions or memorizes the whole training split of the dataset, we traverse the training set to retrieve the nearest

---

1663
Sundard Clasificatio
Zero- amo-thartt Cee.shhttooooooooo Top 1)
Zen- short Top 5
W/b costrrstt
SLIP
CLIP
Figure Ablation study the effectiveness vision-language models and the impact of learnable query embeddings. This figure shows the ablation study on the effectiveness of the vision- model and the impact of learnable query embeddings. According to the table, our model with CLIP and 512 learnable query embeddings achieves the best reconstruction and classification performance, indicating its ability to recover 3D shapes and align multi-modal space.
neighbors of the generated samples, as illustrated in Figure 7. Specifically, exhibit three nearest neighbors for each sample for comparison. According to the visualization, the generated 3D shape differs from each retrieved nearest neighbor, demonstrating that our model hallucinates new 3D shape rather than overfitting the training sets.
Image
Gevcare
Narol
Iraee
Unperal
Sceor
NeuDE
Figure 7: Nearest Neighbor Analysis. We traverse the whole training set to find three nearest neighbors for the generated 3D shapes, and the results reveal that our model could produce novel 3D shapes based on given images instead of memorizing specific ones.
5 Discussion and Conclusion
Though method has achieved excellent results, it still has some limitations. First, method needs the ground truth 3D shapes from training, while 3D data is usually an order of magnitude smaller than the 2D data. Learning the shape representation with a 3D shape-image text aligned space from only 2D (multi-view) images via differentiable rendering is a promising direction. Furthermore, since represent each 3D shape as occupancy field, it needs to convert the 3D mesh into watertight one, which will inevitably degrade the original quality of the 3D mesh.
In conclusion, we propose a novel framework for cross-modal 3D shape generation that involves aligning 3D shapes with 2D images and text. We introduce new 3D shape representation that can reconstruct high-quality 3D shapes from latent embeddings and incorporate semantic information by aligning 3D shapes, 2D images, and text in a compatible space. This aligned space effectively closes the domain gap between the shape latent space and the image/text space, making it casier to learn a better probabilistic mapping from the image or text to the aligned shape latent space. As a result, proposed method generates higher-quality and more diverse 3D shapes with greater semantic consistency that conform to the conditional image or text inputs.
Acknowledgement The work was supported by NSFC #61932020, #62172279, Program of Shanghai Academic Research Leader, and "Shuguang Program" supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission
10

---

References
[1] Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Goodman, and Leonidas Guibas. Shapeglot: Learning language for shape differentiation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8938-8947, 2019.
[2] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9902-9912, June 2022.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.0666..060 2016.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Krcis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01:3
2022.
[5] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu, All worth words: vit backbone for scorc-based diffusion models. arXiv preprint arXiv:2209. 220.122099..152,, 2022.
[6] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: clothed 3d humans. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 344-359, Springer, 2020.
[7] Andreas Blattmann. Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint arXiv:2304.00000 08818, 2023.
[8] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint 1xXv:16088002366 2016.
[9] Ruojin Cai, Guandao Yang. Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[10] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay. Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123- 16133, 2022.
[11] Angel X. Chang. Thomas Funkhouser, Leonidas Guibas. Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information- Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
[12] Kevin Chen, Christopher B Choy. Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2 2-6, 2018, Revised Selected Papers, Part III 14, pages 100-116. Springer, 2019.
[13] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv, 2022.
[14] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2020.
[15] Gene Chou, Yuval Bahat, and Felix Heide. Diffusionsdf: Conditional generative modeling of signed distance functions. arXiv preprint arXiv:2211.13755 2022.
[16] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303.1 1131, 2023.
[17] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 180.04460..88455 2016.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two sime-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

[19] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2100002303, 2022.
[20] Jonathan Ho and Tim Salimans. Classifier-frce diffusion guidance. arXiv preprint arXiv:2207.12599, 2022.
[21] Fangzhou Hong. Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars, ACM Transactions on Graphics (TOG), 119):1119, 2022.
[22] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2023.
[23] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651- 4664. PMLR, 2021.
[24] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot Lext-guided object generation with dream fields. 2022.
[25] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2330.002 024466 2023.
[26] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[27] Yichong Leng, 7ehua Chen, Junliang Guo, Haohe Liu, Jiawei Chen, Xu Tan, Danilo Mandic, Lei He, Xiangyang Li, Tao Qin, et al. Binauralgrad: two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances Neural Information Processing Systems, 35:23689-23700 2022.
[28] Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[29] Chen-Hsuan I Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[30] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han. Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. arXiv preprint arXiv:2305. 10764, 2023.
[31] Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. Iss: Image as stetting stone for text-guided 3d shape generation. arXiv preprint (=XXi::22990014455 2022.
[32] Zhengzhe Liu, Peng Dai. Ruihui Xiaojuan Qi. and Chi- ChigWing Fu. Iss++: Image as stepping stone for text-guided 3d shape generation. arXiv preprint 23333 15181, 2023.
[33] Zhengzhe Liu, Wang. Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. In Proceedings of the IEEL/CVF Conference on Computer Vision and Pattern Recognition, pages 17896- 17906, 2022.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1 2017.
[35] Andrew Luo, Tiangin Li, Wen-Hao Zhang, and Tai Sing Lee. Surfgen: Adversarial 3d shape synthesis with explicit surface discriminators. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 16238 16248, 2021.
[36] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black, Learning to dress 3d people generative clothing. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6469-6478, 2020.
[37] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460 4470, 2019.
12

---

[38] Ben Mildenhall, Pratul P Srinivasan. Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
[39] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 306-315, 2022.
[40] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27. 2022, Proceedings, Part XXVI, pages 529-544 Springer, 2022.
[41] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Lingi Zhou, and Paul Guerrero, 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arX0v:2212.00082. 2022.
[42] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e A system for generating 3d point clouds from complex prompts. arXiv preprint (2x.iv:212.0877.. 2022.
[43] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference compuder vision and puttern recognition, pages 165-174, 2019.
[44] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[45] Songyou Peng. Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[46] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
[47] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning point sets metric space. Advances in neural information processing systems, 30, 2017.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[49] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint wxXiv:220.006222 2022.
[50] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pages 8821 18311 PMLR, 2021.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffuuson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages
2022.
[52] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer, 2015.
[53] Yue Ruan, Han-Hung Lee, Ke Zhang, and Angel X Chang, Tricolo: Trimodal contrastive loss for fine-grained text to shape retrieval. arXiv preprint 22t::2201.0736666 2022.
[54] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36A4 364944 2022.
[55] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[56] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang. Chin-V Yi Cheng. Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot shappe generation. In Proceedings of the 2022.
IEEE/CVF Conference Computer Vision and Puttern Recognition, pages
13

---

[57] Tianchang Shen, Jun Gao. Kangxue Yin. Ming- Yu Liu, and Sanja Fidler. Deep marching tetrabedra: hybrid representation for high-resolution 3d shape synthesis. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[58] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[59] Uriel Singer, Adam Polyak. Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et avieeo-video: Text-to-vides generation without text-video data. arXiv preprint (tt::::22991199992 2022.
[60] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02500 2020.
[61] Guy Tevet. Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or. Human motion diffusion model. In International Conference on Learning Representations (ICLR), 2023.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[63] Ruben Villegas. Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang. Moham- mad Taghi Saffar. Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399) 2022.
[64] Ye Wang, Bowei Jiang. Changqing Zou, and Rui Ma. Mxm-clr: A unified framework for contrastive learning of multifold cross modal representations. arXiv preprint arXiv:3303.1083) 2023.
[65] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d anverrtiveaddvesariial modeling. Advances in neural information processing systems, 29, 2016.
[66] Chen Xin, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[67] Le Xue, Mingfei Gao. Chen Xing, Roberto Martin Martti, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. arXiv preprint arXiv:22 /2.05 2022.
[68] Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or and Hui Huang. Shape- former: Transformer- based based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6239 02299,6242229 2022.
[69] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[70] Biao Zhang. Matthias Nießnnrr and Peter Wonka. 3dilg: Irregular latent grids for 3d generative modeling. In NeurlPS, 2022.
[71] Biao Zhang, Jiapeng Tang. Matthia Nießner, and Peter Wonka. 3dshape2veeset: A 3d shape representation for neural fields and generative diffusion models. ACM Trans. Graph., 42(4), jul 2023.
[72] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun. Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual deseriptions with discrete representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[73] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[74] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li. Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8542-8552, 2022.
14


Reviews:
[{"review_id": "KOufU8ipj9", "reviewer": "Reviewer_fo3T", "summary": "This paper proposed a VAE-based text-to-3D shape generation method. The authors designed an alignment-before-generation approach to narrow the gap between 3D shapes and the 2D or text condition. They first train a Shape-Image-Text-Aligned Variational Auto-Encoder to align the representations between the 3D shapes and the 2D or text inputs. Then, the use latent diffusion model to denoise the shape embeddings to match the conditions. The overall presentation is good and the experiments are extensive.", "questions": "1. Since this method is trained with ground truth 3D data, which limited its generalization ability, would it be better to show some failure cases when the condition fails to lie in the aligned representation space?", "limitations": "The author mentioned the limitation of the requirement of the 3D training data, which could be a huge problem in the generalization of this method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The narratives are good and sound.\n2. The SITA-VAE helps to align the representations between the three modalities.\n3. The generation only requires some denoising steps on the latent embeddings.\n4. The experiments are extensive and the authors provide sufficient visual demos and show the effectiveness of the method.", "weaknesses": "1. It seems that shape-image-text align training requires a lot of paired shape-image-text data, which could be a huge challenge in the generalization to new categories.\n2. The experimental implementation is not clear enough and could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to reviewer fo3T for the positive feedback and thoughtful comments, and we are encouraged that the reviewer recognizes our effort on visual demonstrations and enjoys reading the manuscript. In the following, we reply to the individual concerns in three aspects.\n\n**Q1: What if the condition fails to lie in the aligned representation space?**\n\n**A1:**\n\n1. We additionally show results that conditions on the MVImgNet images contain different image styles and some unseen categories in Figure.3 in the rebuttal PDF. The visual demos validate the generalization ability of our model.\n\n2. Alignment-before-generation enhances the generalization ability of our model to some extent, as the CLIP latent space is a highly expressive and semantically rich space that captures both visual and textual information. By aligning the 3D space to the CLIP latent space, the model can utilize the semantic information captured in the CLIP space to close the semantic distance among the 3D latent. Benefiting from this, when the condition is unseen data to our training set, the model at least generates a semantically coherent 3D shape latent.\n\n3. We employ the feature of the last layer from the CLIP encoder rather than the final embeddings of it. The last hidden layer tokens capture global and local information from the conditions, leading the model learning to match the conditions to the target 3D shape latent in terms of global semantic information and aspects of the local semantic parts. Therefore, when the global condition information fails to lie in the aligned representation space, some local parts always lie in the space, which facilitates the model's generalization ability.\n\n4. We acknowledge that generalization is a significant challenge, highlighting the need for more efficient methods and larger-scale 3D datasets from the entire research community. In this work, we propose the alignment-before-generation approach, which enhances the generalization ability of our model. Additionally, the recently released Objaverse dataset is valuable for developing more effective 3D shape generation models. The continued efforts of the research community in this direction will lead to significant advancements in the field of 3D shape generation.\n\n**Q2: Shape-image-text align training requires a lot of paired shape-image-text data, which could be a huge challenge in generalizing to new categories.**\n\n**A2:**\n1. **Data preparation:** Please refer to global response.\n\n2. **Generalization to new categories:** Our model is the first that performs image-/text-conditioned generation on the whole Shapenet dataset and shows potent generative ability on some common categories after training on the aligned latent space from the entire Shapenet dataset. Moreover, benefiting from the autonomous data pre-processing procedure, we could quickly scale the training data triplet on the recent 3D shape benchmark. Moreover, we find it practicable to fine-tune the model on an enlarged dataset containing the Shapenet dataset, Objaverse, and some other released 3D datasets. We also show some visual results in Figure.1 in the rebuttal  PDF. Our model shows vast potential to scale up as a foundation model for conditional 3D shape generation tasks.\n\n**Q3: Experimental Implementations.**\n\n**A3:** Since we present **Model Implementations** in *global response* and generations comparison details in **Q3: Clarification on Table 2.** of response to *Reviewer rVoM*. We present the **ablation study** supplementary.\n\nWe ablation study our method in several aspects.\n1. The effectiveness of training the generative model in the aligned space. Figure.4 from the main text compares the results of two generative models under the same conditions. Our ASLDM produced the upper results, and the lower results produced by the generative model that trains in a single 3D shape latent space.\n2. The effectiveness of the vision-language model (VLM). We ablation study the efficacy of the VLM on four tasks and two VLM models, SLIP and CLIP. We compare three VAEs, training without contrastive loss, training with an image-encoder and text-encoder from SLIP, and training with an image-encoder and text-encoder from CLIP. \n\n    a. The first is reconstruction. We use 3D shape encoders from three VAEs to conduct the shape reconstruction tasks on the testing set of Shapenet.\n\n    b. The second is Standard Classification. We use the 3D shape encoders and text encoders from three VAEs to perform a classification task on the testing set of Shapenet. As there are 55 categories in Shapenet, we set 55 texts with a template: a 3D model of (*), where * indicates a category. After computing the Cosine Similarity between a given 3D shape and the 55 texts, we regard the category with the highest Cosine Similarity as the classification results.\n\n    c. The third and the fourth are Zero-shot Classifications, similar to the second task, but evaluating the testing set of ModelNet, which contains 40 categories. Specifically, we compute a Top 1 accuracy and Top 5 accuracy. When calculating the Top 1 accuracy, we only record the highest Cosine Similarity with the category corresponding to the 3D shapes as successful classification. Moreover, it is correct to compute the Top 5 accuracy if the category conforms to the 3D shape that comes in the Top 5 Cosine Similarity.\n"}, {"review_id": "VaRzTsoSrV", "reviewer": "Reviewer_rVoM", "summary": "This paper proposed a conditional generation model which aims to solve the alignment issue in image-to-shape or text-to-shape generation. The key idea is to learn a aligned representation among 3D shapes, images, and texts. To achieve that, the author proposes SITA-VAE with contrastive loss to force the shape's latents to be aligned with the pretrained vision-language model. After that, a LDM is trained to learn the diffusion process in the latent space. In the test time, it follows the previous works to use classifier-free guidance to perform conditional generation. The proposed method is evaluated on ShapeNet and Cartoon Monster 3D shapes.", "questions": "1. The results in Fig. 2 seems a bit too sharp and I wonder if the model is memorizing the outputs. Can the author shows the nearest neighbor sample from the datasets?\n\n2. In figure 1, the model is outputting occupancy but the final result is denoted as \"mesh\". I think the output should be occupancy and the mesh is obtained via marchingcube, right? Learning to generate mesh directly is very different from generating occupancies. The writing also use the word \"mesh\" and it is a bit confusing.\n\n3. In Table 2, the use of SITA to evaluate the performance of conditional generation is unfair compared to other baselines since the LDM is trained on the space learned with SITA.", "limitations": "Authors have address the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Propose to learn a aligned space for 3D shapes, images, and texts with 2D vision-language model. This essentially leverages the abundant 2D data into 3D modeling. The scarcity of paired data is one crucial reason why conditional 3D modeling does not perform as well as its 2D counterpart.\n\n* The conditional generation results look great and align with the inputs when compared with the baselines.\n\n* It is surprised that the aligned space does hurt the reconstruction results.", "weaknesses": "* Novelty is somewhat limited. The first stage model is mostly based on 3DILG and the multi-modality conditional generation has been explored in previous work such as SDFusion. \n\n* Scalability is an issue as the model need paired 3D-image-text data to work.\n\n* Some parts of the writing and figure are misleading. The choices of the metric is problematic in Table 2.\n\nPlease see the \"Questions\" for the details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to reviewer rVoM for the thoughtful comments. We appreciate the reviewer's approval of our align-before-generation approach. Furthermore, we are encouraged that the reviewer recognize our visual demonstrations. We dedicated replies to the reviewers' comments and questions in five aspects below.\n\n**Q1: Novelty.**\n\n**A1**:In the beginning, we acknowledge that image-/text-conditioned 3D shape generation tasks exist. Still, please reiterate that our alignment-before-generation approach is the first to train an image-/text-conditioned 3D shape generative model on the whole Shapenet Dataset. Furthermore, we are working on scaling our model on a larger scale 3D benchmark, and the model exhibits good scalability in our experiments. \n1. The difference between 3DILG and the first stage model, Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE).\n\n    a. First, 3DILG only encodes 3D shape information in their latent space. In contrast, our SITA-VAE aligns semantic information into the 3D shape latent space by leveraging contrastive learning;\n\n    b. The second difference is that 3DILG encodes the 3D shape as paired explicit center-coordinates and points features, which requires Farthest Points Sampling and k-nearest-neighbor clustering to handle the input. At the same time, our SITA-VAE utilizes the learnable query embedding to encode the 3D shapes implicitly, and our encoder could directly learn geometric and semantic information from the whole input point clouds;\n\n    c. The third is that 3DILG employs eight-layer multi-layer perceptrons (MLPs) to reweight explicit centers-coordinates and the output tokens from its decoder and predict the occupancy of query points, while the decoder in our SITA-VAE utilizes cascade attention modules to predict the occupancy of query points;\n\n    d. 3DILG trains in a VQ-VAE manner.\n\n2. The difference between our Aligned Shape Latent Diffusion Model (ASLDM) and previous multi-modality conditional generation, such as SDFusion.\n\n    a.First, our ASLDM differs from the previous methods in that ASLDM learns on an aligned latent space. Still, previous conditional generative models almost train on a single modal space.\n\n    b.The second is the representation of 3D shape latents. SDFusion encodes 3D shapes as discrete latent voxels via VQ-VAE, which is potentially challenging to scale up to fit higher resolutions shapes or larger datasets because it achieves it by using higher-resolution discrete latent voxels or larger codebooks. In contrast, our ASLDM learns on the continuous shape latent with low dimensions, which are flexible. After successfully training on the whole Shapenet dataset, we further train the model on a larger dataset, and the results in Figure.1 in the rebuttal PDF prove our preliminary achievement.\n\n**Q2: Scalability.**\n\n**A2:** We acknowledge that 3D data is crucial for the scalability of conditional 3D shape generation tasks. Fortunately, another large-scale benchmark, Objaverse, which is ten times the entire Shapnet dataset, has been released. Since we have developed an autonomous pipeline (mentioned in the global response part) for producing shape-image-text triplet, described below, we could quickly start training on the new dataset. Moreover, we have some preliminary achievements, shown in Figure.1 in the rebuttal PDF. \n\n**Q3: Clarification on Table 2.**\n\n**A3:** Before explaining the content of Table 2, we clarify the typo in Table 2, as the repeated SI-S (ULIP) and SI-S (SITA) under 'Text-Conditioned' should be ST-S (ULIP) and ST-S (SITA). Thanks for the correction.\nWe propose two new metrics to evaluate the conditional 3D shape generative models. The first is the shape-image score (SI-S), measuring the similarity between image conditions and generated 3D shapes. In particular, we use an image encoder to convert the image condition into an image embedding and a 3D shape encoder to transform the sampled 3D shape into a shape embedding. The SI-S is defined as the Cosine Similarity of the extracted shape and image embeddings. Another metric is the shape-text score (ST-S), which measures the similarity between text conditions and generated 3D shapes and implements a similar approach to SI-S. During the evaluation, although the 3D encoders in the SITA-VAE could extract 3D shape embeddings aligned to CLIP space, as the reviewer points out, only employing the 3D shape encoders from SITA-VAE to compare the generated results is unfair since ASLDM trains on the latent space of SITA-VAE. Therefore, for a comprehensive evaluation and comparison, we utilize an additional 3D shape encoder from ULIP, which pre-trains for aligning to the CLIP space. In detail, SI-S (SITA) and ST-S (SITA) indicate the score computed with the 3D encoder from SITA-VAE, and SI-S (ULIP) and ST-S (ULIP) indicate the score calculated with the 3D encoder from ULIP. The result in Table 2 proves that our model performs better under both ULIP and SITA-VAE. \n\n**Q4: Nearest Neighbor of the generated results from datasets.**\n\n**A4:** Please refer to the visual demos in Figure.3 in the rebuttal PDF, which illustrates the Top 3 nearest neighbors of the generated 3D shapes from the **training** dataset. The visual demos indicate that our model can generate novel shapes based on the learned 3D shapes in the training set rather than retrieve or memorize a specific shape.\n\n**Q5: Clarification on Marching Cubes.**\n\n**A5:** We apologize for the misleading due to our negligence while drawing the pipeline. The direct output from the 3D shape decoder is the occupancy of the query point, and the mesh is produced via conducting the marching cube algorithm on the occupancy of sampled volume points. Thanks for the rectification, and we will add a legend that indicates the marching cubes. Moreover, the abuse of 'mesh' and '3D neural shape' is somewhat confounded in our paper. In the revision, we will correct the inappropriate words."}, {"review_id": "pMiupaa2Oc", "reviewer": "Reviewer_XFVg", "summary": "This work is about 3D shape generative model focused on image-conditioned and text-conditioned generation. The authors aligned the latent space of a shape autoencoder to CLIP's image encoder and text encoder. Then generative diffusion models are trained on the aligned latent space. This enables shape generation given image or text as conditional input. The authors showed some good results in both tasks.", "questions": "See the weakness section.", "limitations": "This work proposed a method for image-conditioned and text-conditioned shape generation. The authors combined several components (3DS2V-style shape autoencoding network, clip and unet-style transformer denoising network). All these components are not proposed by this work which weakens the novelty of this paper. However, I still believe the authors delivered some good results in shape generation. I hope the authors can clarify my concerns mentioned in the weakness section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors showed some good generation results for both the task of image-conditioned generation and text-conditioned generation. The writing is also clear and easy to follow.", "weaknesses": "1. The autoencoder network (Fig 1 a) is similar to the network used in [63]. Thus the performance boost shown in Table 1 seems to be because of CLIP. This should be emphasized or ablated somewhere in the main paper.\n2. Following the above comment (Table 1), the authors only compared with \"Learned queries\" results from [63]. According to [63], another design \"Point queries\" achieved better results than both this work and \"Learned queries\".\n3. It would be better if the authors can show some visualization comparisons of the autoencoding results.\n4. L136, the length is (1+L_i) instead of L_i. This is explained in later sections (L217) but is still causing confusion.\n5. Another difference with respect to [63] is, when training the diffusion models, this work used a unet-style transformer [4] instead of a simple stacked self attention network.\n6. Some pieces of writing can still be improved. For example, when talking about a design or an equation, we should mention why we are doing this or discuss some insights behind this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to reviewer XFVg for the positive feedback and insightful comments. Moreover, we are encouraged that the reviewer appreciates our results and enjoys reading the manuscript. We reply to the concerns in six aspects below. Furthermore, we will update Table 1 and related discussion in future revisions.\n\n**Q1: Insights and motivations.**\n\n**A1**: At the beginning of our response, we reiterate our goals, insights, and discovery.\n1. We study generating general 3D shapes based on given 2D images or texts.\n\n2. Even if a conditional 3D shape generative model trains successfully on the whole Shapenet dataset, the models' samples do not meet the given conditions enough. Besides, for those categories with few training samples, the model is hard to map the condition to the corresponding 3D shapes.\n\n3. We observe that generating a 3D shape based on a 2D image or text is an ill-posed problem since conditions only provide partial information while the remaining information is missing, necessitating priors from the latent space of 3D shape distribution. It leads to difficulties for the model mapping 2D and textual information to valid priors in a pure 3D shapes latent space. \n\n4. However, most previous works tend to encode the 3D shapes into the latent space that contains only geometry information without considering the semantic information, and none of the existing works manage to train an image-/text-conditioned generative model on the whole Shapenet dataset. \n\n5. We turn to represent 3D shapes in a latent space that captures geometric and semantic information, alleviating the generative models' difficulties in mapping semantic information to 3D shape priors. Moreover, the success of DALLE-2, which learns on aligned CLIP latent space, inspires us to devise the alignment-before-generation approach. \n\n6. As a result, our model is the first to perform image- and text-conditioned 3D shape generation tasks on the Shapenet dataset.\n\n**Q2: Ablation on CLIP.**\n\n**A2:** Thanks for the thoughtful comments. In the main paper, we only show an unremarkable chart in the first column in Figure.5 in the main texts to ablate the effectiveness of utilizing different Vision-Language models (VLMs). We extend the experiments to ablate the impact of VLMs, and the results are shown in the table. \n\nThe last three rows in the table are three cases we used to train the VAE. Ours (w/o C) is a single VAE train without the contrastive loss, Ours (w SLIP) indicates to utilize the image- and text-encoder from SLIP, and Ours (w CLIP) is the SITA-VAE, where we employ CLIP's image- and text-encoder to train the VAE. The result showcases the effectiveness of the utilization of CLIP.\n\n|                    | Overall | Selected | Table  | Chair  | Airplane | Car    | Rifle  | Lamp   |\n|:-------------------|:-------:|:--------:|:------:|:------:|:--------:|:------:|:------:|:------:|\n| 3DS2V (PQ)         | 0.967   | 0.967    | 0.971  | 0.964  | 0.969    | 0.969  | 0.96   | 0.956  |\n| Ours (w/o Contrsat)| 0.961   | 0.958    | 0.958  | 0.962  | 0.961    | 0.968  | 0.952  | 0.945  |\n| Ours (w SLIP)      | 0.962   | 0.956    | 0.96   | 0.959  | 0.957    | 0.966  | 0.954  | 0.937  |\n| Ours (w CLIP)      | 0.966   | 0.964    | 0.965  | 0.966  | 0.966    | 0.969  | 0.967  | 0.95   |\n\n**Q3: Comparison with Point Queries in 3DS2V.**\n\n**A3:** Thanks for carefully figuring out the incomplete comparison in Table 1. We replenish Table 1 and report the results above.\n\n1. Performance and analysis: The performance difference is due to the different sizes of the 3D shape latent. 3DS2V employs a deterministic autoencoder, while SITA-VAE contains a KL block that compresses the 3D shape latent. Therefore, the 3D shape decoder in SITA-VAE reconstructs 3D shapes with the low dimension 3D shape latent, leading to lower performance. Moreover, such results are similar to the conclusion from section 8.1 in 3DS2V, which shows that the compressed latent decreases performance. \n\n2. Motivation for using learnable query embeddings: We design the learnable query embeddings for utilizing the local embedding setting from Perceiver and the global embedding setting from CLIP. Specifically, there embed (L_s + 1) learnable query embeddings in the 3D shape encoder. To learn a semantically and geometrically aligned 3D shape representation globally and locally via the learnable query embeddings, it contains a global head-class token and L_s shape tokens. Moreover, the learnable query embeddings and the cross-attention mechanism allow the model to handle significant inputs, potentially suitable for further scaling up.\n\n**Q4: Comparison of auto-encoding.**\n\n**A4:** We show the visual comparison of auto-encoding in Figure.4 in the rebuttal PDF.\n\n**Q5: Concern 5: Clarification for the notion in line 136.**\n\n**A5:** The image encoder in CLIP is a ViT-based architecture. Thus, the sequence in the encoder consists of one head-class token and L_i local tokens from the patch embeddings, and the sequence length of the last hidden layer in ViT is (1 + L_i). The following is the utilization of tokens.\n\n1. When computing the multi-modal contrastive losses, only the head-class token will be projected into an image embedding in the training process of SITA-VAE. \n\n2. In training generative models, we extract the sequence from the last hidden layer in the image encoder, consisting of one head-class token and L_i local tokens to better capture global and local information. \n\n**Q6: The architecture of the Diffusion Model.**\n\n**A6:** We implement the denoiser based on two architectures. A UNet-like transformer and a stacked attention block network, where each attention block contains a self-attention module and a cross-attention module. \nDuring experiments, we found that the UNet-like transformer denoiser converges with fewer iterations, as concluded in [4], and the training phase is stable. Therefore, we employ the UNet-like transformer architecture for the rest of the experiments. "}, {"review_id": "He7QKSlbee", "reviewer": "Reviewer_73Gb", "summary": "This paper proposes a conditional 3D generation method by pre-aligning features of different modalities when training a 3D AE. After that, a latent diffusion model is applied to generate latent vectors for 3D decoder conditioned on text/image. The results quantitatively and qualitatively shows that the proposed method is able to achieve impressive results.\n", "questions": "1. If you have paired data for text-image-shape alignment training, why not just directly train a latent diffusion model conditioned on image/text to generate 3D objects like SDFusion[1]? In contrast, if you already aligned text/image/shape features in latent space, is it still necessary to use a heavy conditional generator such as latent diffusion to achieve your tasks? It feels like you only need one of these two techniques. \n2. In your qualitative comparison, you mentioned that 3DS2V generates more details of 3D shapes and discontinuous surfaces and noisy results. However, such results are not seen in their paper, I wonder the reason why they fail to produce good results in Figure 2 and 3. Is it because you re-train their model on your dataset or because you directly use their pre-trained checkpoint or other reasons? More discussion and insight should be addressed.\n3. You pre-aligned the text/image/shape feature first and use text/image feature to train a latent diffusion model to achieve conditional generation. However, as far as I know, if the input condition and the target feature are too similar when training the diffusion model, the diversity of the generation result would be limited and the purpose of using the diffusion model would disappear. How do you avoid such a situation from happening?\n4. Please answer the problem mentioned in the weakness part.\n", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The results show that this method is able to achieve multiple tasks, which is good and important for a generation-related paper.\n2. The main figure is easy to understand and follow, making this paper easier to read. \n3. The core idea in this paper looks original.\n", "weaknesses": "1. No qualitative results show the diversity of the proposed method. Such a result is important to a generation-related paper.\n2. The concept of pre-alignment is not very convincing. Other conditional generation methods are also aligning text/image to the shape latent space by either pulling their representations together or learning diffusion models. Further discussion on how this concept is better than others should be added.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our paper's originality and importance to the community. Thanks for the appreciation for our visual results demonstrations and thoughtful comments. We will attempt to address your concerns in four aspects in the following. \n\n**Q1: Qualitative results of the diversity.**\n\n**A1:** We show qualitative results in Figure.1 of the rebuttal PDF, demonstrating the generated results' diversity. Moreover, we further illustrate the Nearest Neighbors from the training samples to verify that the trained model does not memorize the training samples in Figure.2 or the rebuttal PDF.\n\n\n**Q2: Motivations and effectiveness of alignment-before-generation approach.**\n\n**A2:** We recall our goals at the beginning of the response, developing a conditional 3D shape generative model to generate high-quality 3D shapes to conform to the given images/texts. And the model should work on all the categories of data, as many as we have, instead of a single category or a few. \n\nTo achieve this, we start exploring training conditional 3D shape generative models on the whole ShapeNet, a large-scale 3D shape benchmark, several months ago. During the exploration, we find that even if a conditional 3D shape generative model trains successfully, the samples from the model do not conform to the corresponding conditions well, like the generated results of the text-to-image generative models. Besides, it is difficult for the model to map the condition to the corresponding 3D shapes for those categories with few training data.\n\nWe observe that generating a 3D shape based on a 2D image or text is an ill-posed problem since conditions only provide partial information while the remaining information is missed, necessitating priors from the latent space of 3D shape distribution. However, the latent space of 3D shapes significantly differs from 2D images and texts and leads to difficulties for the model in mapping 2D information and textual information to valid priors in a pure 3D shapes latent space. \n\nHowever, most previous works tend to encode the 3D shapes into the latent space that contains only geometry information without considering the semantic information, and none of the existing works manage to train an image-/text-conditioned generative model on the whole ShapeNet. Therefore, we turn to represent 3D shapes in a latent space that captures both geometric and semantic information, which alleviates the difficulties for the generative models in mapping semantic information to 3D shape priors. Moreover, the success of DALLE-2, a generative model that first trains a prior model from an aligned CLIP text space to an aligned CLIP image space, inspires us to devise the alignment-before-generation approach. \n\nAs a result, our model is the first that performs image- and text-conditioned 3D shape generation tasks on the whole ShapeNet categories, and the model exhibits generalization ability to unseen categories to some extent. Moreover, with the release of another large-scale dataset, Objaverse, and the incoming Objaverse-XL, we believe that our method can extend to and scale up to such a large-scale dataset (Currently, we have successfully scaled up our method in Objaverse, and the qualitative results in Figure.1 of the rebuttal PDF and the pre-trained model will be released.).\n\n**Q3: Relations between alignment and conditional generation.**\n\n**A3:** Thanks for the reviewer's interest in our alignment-before-generation approach. Alignment and conditional generative models could coexist, and our experiments reveal that alignment before generation enhances the generative model. \n\n* Contrastive pre-training cannot bring the generative ability to a model such as CLIP. It aims to train the model to capture multi-modal information in an aligned latent space. For SITA-VAE, contrastive learning constrains the model to capture geometric and semantic information in an aligned latent space. However, SITA-VAE can only reconstruct the neural field of a given 3D shape latent, necessitating a generative model for producing a 3D shape latent. \n\n* Our experimental results further prove the effectiveness of the training generative model on the aligned latent space. Besides, the recent foundation text-to-image generative model DALLE-2 also indicates that learning on the aligned latent space enhances the generation process. Specifically, DALLE-2 trains a prior network that maps text embeddings to image embeddings on the latent space of CLIP and a decoder network to generate images with given image embeddings. The generative ability of DALLE-2 also proves the effectiveness of training the generative models on the aligned latent space.\n\n**Q4: Clarification on the comparison with 3DS2V.**\n\n**A4:**  According to the paper, 3DS2V only trains image-/text-conditioned generation tasks on a subset of the ShapeNet. And no pre-trained checkpoints are available on the all-category ShapeNet at that time. To make a fair comparison, we re-train 3DS2V with the same CLIP condition and parameters as ours.  It has two failure cases: the first is that although the model successfully maps given conditions to high-quality 3D shape latent, it only decodes a high-quality 3D shape, which is not visually or semantically close to the given condition. Another is even worse since mapping 2D information to 3D shapes is an ill-posed problem. When the model fails to sample validated 3D latent codes, and results in a noisy shape by the decoder. Visual and numerical results prove our statements. We observe that the image-conditioned generated results from 3DS2V's paper in Figure 13 are not as good as the category-conditioned generated results in Figure 8. It reflects our observation.\n\n"}]

Author rebuttal:
# To All Reviewers:

We express our gratitude and appreciation to all the reviewers contributing to the review process. The reviewers, fo3T, rVoM, XFVg, and 73Gb, have commended the paper for:
1. well-written presentation.
2. Good visual demos.
3. Solid technical foundation. 

We are also thankful for their insightful comments, which have provided valuable feedback to enhance our work and generate promising ideas for future research. We have endeavored to respond to the reviewers' queries and remarks with utmost clarity and detail in our individual replies. If the paper is accepted, we will restructure the final camera-ready version and incorporate more discussion into the main text.

**Motivation and novelty (Reviewer fo3T, rVoM, XFVg).**

We present a novel alignment-before-generation approach for conditional 3D shape generation tasks, showcasing the shape-image-text-aligned space's effectiveness for cross-modality 3D shape generation. 
1. Compared to the previous 3D shape representation methods characterizing the geometric information of each 3D shape only, we investigate a 3D shape representation that further brings the semantic information into the neural fields. Meanwhile, the 3D shape representation is decodable. 
2. Compared to the previous 3D shape generation methods training on the partial Shapenet dataset, the proposed Aligned Shape Latent Diffusion Model (ASLDM) successfully trains on the composition of the whole Shapenet dataset and 3D Cartoon Monster dataset based on the proposed 3D shape representation.

**What does this work achieve:**

The contribution of this work lies in training an image-/text-conditioned 3D shape generative model on the composition of the whole Shapenet and 3D Cartoon Monster dataset, which is the first of its kind. The generated results conform to the given conditions, demonstrating the effectiveness of the proposed approach. 

**What does this paper propose:**

This paper proposes an alignment-before-generation approach to tackle conditional generative models' challenges in producing consistent 3D shapes with the given conditions.

**What does this project mean:**

This project reveals a fundamental but systematic pipeline of 3D shape generation tasks. The proposed alignment-before-generation approach enhances and stabilizes the training process of the generative model, and the introduced scalable data pre-processing scheme manages to create Shape-Image-Text triplets based on existing and potential large-scale 3D shape benchmarks.

**What does the current ASLDM differ from that at the paper submission:**

Benefiting from the release of more extensive scale benchmarks, we scaled up the ASLDM, resulting in improved performance.

**Data preparation:**

During the data pre-processing, we build shape-image-text triplets to exploit large-scale models' power. We especially employ the depth-condition-based ControlNet to render images under different views. We design some render prompts for rendering 3D shapes into corresponding images via its different views' depth images. The images and render prompts are then collected as the paired images and texts. Moreover, the texts are collected in three manners:
1. The category and descriptions of the shape from the Shapenet dataset.
2. The render prompts mentioned above closely pair with rendered images. 
3. Followed by the text template in recent works, ULIP.

**Model Implementation**:

1. Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE).

    a. SITA-VAE implements perceiver-based transformer architecture. The 3D shape encoder consists of 1 cross-attention block and eight self-attention blocks. All the attention modules consist of multi-head attention (with 12 heads and 64 dimensions of each head), layer normalization (Pre-Norm), Feed-Forward Network (with 3072 dimensions), and GELU activation. 

    b. The learnable query embeddings (with 513 tokens and 768 dimensions of each token) consist of one head-class token and 512 shape tokens. The shape tokens will be projected into VAE space (with 512 tokens and 64 dimensions of each token ) after processing by the 3D shape encoder, and another projection layer transforms the sampled latent back (with 768 dimensions) for the 3D shape reconstruction. We compute the multi-modal contrastive losses on the head-class token and the KL-divergence loss on the shape tokens.

    c. The 3D shape decoder (neural field decoder) has 16 self-attention blocks with a final cross-attention block for the implicit neural field modeling. All the attention modules consist of multi-head attention (with 12 heads and 64 dimensions of each head), layer normalization (Pre-Norm), Feed-Forward Network (with 3072 dimensions), and GELU activation. Given a query 3D point and its corresponding shape tokens, the decoder computes self-attention iterative and cross-attention for predicting the occupancy of the query point. We compute the binary cross-entropy loss on the result and ground-truth.

    d. Image-/text-encoder from CLIP (ViT-L-14) in SITA-VAE are frozen during training and inference.

    e. During the training phase, we follow 3DILG, which normalizes all mesh into [-1,1] first and then separately and equally samples volume points (randomly sampled) and near-surface points with ground-truth inside/outside labels from the watertight mesh.

2. Aligned Shape Latent Diffusion Model (ASLDM)

    a. ASLDM builds on a UNet-like transformer architecture. It consists of 13 self-attention blocks with skip-connection between shallow and deep layers. All the attention modules consist of multi-head attention (with 12 heads and 64 dimensions of each head), Layer Normalization (Pre-Norm), and Feed-Forward Network (with 3072 dimensions).

    b. The ASLDM trains with the MSE loss and . The training diffusion steps are 1000, and beta \in [0.00085,0.012] with a scaled linear scheduler.

Both module uses an AdamW-based gradient decent optimizer with a 1e-4 learning rate.

Meta-review: This work proposes using an aligned shape-image-text with a variational auto-encoder for conditioned shape generation from either text or image.  The aligned latent space for shape-image-text is trained to align shape embddings from a perceiver-based transformer encoder with CLIP image and text embeddings using contrastive losses.  At the same time, a transformer-based decoder is trained to reconstract the shape occupancy.  A diffusion model is trained to probabilistically map text and image embeddings to the shape space. 

Reviewers are positive on this work, noting that the generation results look better than baseline methods.

The authors are encouraged to update their paper with feedback from the reviewers, and incorporate clarification and additional results from the rebuttal including:
1. Add discussion of limitations and generalization to new categories 
2. Incorporate additional visualizations, ablations, and other clarifications 
3. Improve writing to clarify deisng decisions, relation to prior work, and architecture details 
4. Inclusion of relevant related work for Shape-Image-Text aligned representations and text to shape generation such as:
    
   Work on shape-image-text aligned spaces:
   - OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding [Liu et al. 2023]
   - Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings [Chen et al. ACCV 2018]
   - ShapeGlot: Learning Language for Shape Differentiation [Achilioptas et al. ICCV 2019]
   - TriCoLo: Trimodal Contrastive Loss for Fine-grained Text to Shape Retrieval [Ruan et al. 2022]
   - MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations [Wang et al. 2023]. 

   Work on text to shape generation
   - Towards implicit text-guided 3d shape generation, Liu et al. CVPR 2022
   - ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation, Liu et al. ICLR 2023

5. Proofreading pass to fix typos (L115: UNIP => ULIP, L310: Disscusion => Discussion)
Decision: Accept (poster)