Title: Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Conditional 3D Shape Generation

Neural 3D Representation

3D Reconstruction
Abstract: We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Abstract

We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to produce inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and

a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.

1 Introduction

Conditional generative model-based 3D shape generations, such as GAN [9, 35, 65], VAE [8, 36, 6], Auto-Regressive model [68, 39, 70], and Diffusion-based model [69, 41, 15, 13, 28, 42, 25], have great potential to increase productivity in the asset design of games, AR/VR, film production, the furniture industry, manufacturing, and architecture construction. However, two obstacles limit their ability to produce high-quality and diverse 3D shapes conforming to the conditional inputs: 1) diverse 3D shape topologies are complicated to process into a neural network-friendly representation; 2) since generating a high-quality 3D shape from a 2D image or textual description is an ill-pose problem, and also the distribution between the 3D shape space and image or text space is quite different, it is hard to learn a probabilistic mapping function from the image or text to 3D shape.

Recently, the neural fields in terms of occupancy [37, 45], Signed Distance Function (SDF) [43], and radiance field [38] have been driving the 3D shape representation in the computer vision and graphics community because their topology-free data structure, such as global latent [43], regular grid latent [45, 14], and point latent [69, 70], are easier to process for neural networks in an implicit functional manner. Once arrive at a compatible space to represent different topological 3D shapes, in light of the great success of auto-regressive and diffusion-based models in audio [26, 27], image [50, 51, 49, 54, 4], video [63, 59, 19, 7], and 3D human motions [72, 61, 66], a conditional auto-regressive or diffusion-based generative model [15, 69, 70] is learned to sample a 3D shape in latent from an image or text. However, generating a high-quality 3D shape from a 2D image or textual description is an ill-posed problem, and it usually requires more prior information for 3D shapes. In contrast, the distribution of the 3D shape space is significantly different from the 2D image or text space, and directly learning a probabilistic mapping function from the image or text to the 3D shape might reduce the quality, diversity, and semantic consistency with the conditional inputs. Prior research [69, 42] has endeavored to mitigate this concern through a coarse-to-fine approach, whereby the initial step involves generating coarse point clouds as an intermediary representation, followed by the regression of a neural field based on the point cloud.

Unlike the previous 3D shape representation, where the neural fields only characterize the geometric information of each 3D shape and capture the shape distribution by regularizing the shape latent with KL-divergence via VAE [13, 28, 71] or VQ-VAE [39, 70], we investigate a novel 3D shape representation that further brings the semantic information into the neural fields and designs a ShapeImage-Text-Aligned Variational Auto-Encoder (SITA-VAE). Specifically, it uses a perceiver-based transformer [62, 23] to encode each 3D shape into the shape embeddings and utilizes a contrastive learning loss to align the 3D shape embeddings with pre-aligned CLIP [48] image/text feature space from large-scale image-text pairs. After that, a transformer-based neural implicit decoder reconstructs the shape of latent embeddings to a neural occupancy or SDF field with a high-quality 3D shape. With the help of the aligned 3D shape, image, and text space, which closes the domain gap between the shape latent space and the image/text space, we propose an Aligned Shape Latent Diffusion Model (ASLDM) with a UNet-like skip connection-based transformer architecture [52, 5] to learn a better probabilistic mapping from the image or text to the aligned shape latent space and thereby generate a higher-quality and more diverse 3D shape with more semantic consistency conforming to the conditional image or text inputs.

To summarize, we explore bridging the semantic information into 3D shape representation via aligning the 3D shape, 2D image, and text into a compatible space. The encoded shape latent embeddings could also be decoded to a neural field that preserves high-quality details of a 3D shape. Based on the powerful aligned 3D shape, image, and text space, we propose an aligned shape latent diffusion model to generate a higher-quality 3D shape with more diversity when given the image or text input.

We perform extensive experiments on a standard 3D shape generation benchmark, ShapeNet [11], and a further collected 3D Cartoon Monster dataset with geometric details to validate the effectiveness of our proposed method. All codes will be publicly available.

2 Related Work

2.1 Neural 3D Shape Representation

Neural Fields have dominated the research of recent 3D shape representation, which predict the occupancy [37, 45], Sign Distance Function (SDF), density [43, 57], or feature vectors [10] of each 3D coordinate in the field via a neural network to preserve the high-fidelity of a specific 3D shape in a topology-free way. However, the vanilla neural field can only model a single 3D shape and cannot be generalized to other shapes. To this end, the researchers usually take additional latent codes, such as a global latent [43], a regular latent grid [45, 14], a set of latent points [69, 70], or latent embeddings [71, 25], which describe a particular shape along with each 3D coordinate to make the neural fields generalizable to other 3D shapes or scenes. Though current neural fields' 3D representation can characterize the low-level shape geometry information and preserve the highfidelity shape details, bringing the high-level semantic information into the neural fields is still a relatively poorly studied problem. However, semantic neural fields are significant to downstream tasks, such as conditional 3D shape generations and 3D perception [22, 58].

2.2 Conditional 3D Shape Generation

Optimization-based approaches which employ a text-image matching loss function to optimize a 3D representation of the neural radiance field (NeRF). Dreamfields and AvatarCLIP [24, 21] adopt a pre-trained CLIP [48] model to measure the similarity between the rendering image and input text as the matching objective. On the other hand, DreamFusion [46] and Magic3D [29] utilize a powerful pre-trained diffusion-based text-to-image model as the optimization guidance and produce more complex and view-consistent results. However, per-scene optimization-based methods suffer from a low success rate and a long optimization time in hours to generate a high-quality 3D shape. However, they only require a pre-trained CLIP or text-to-image model and do not require any 3D data.

Optimization-free methods [33, 31, 32] are an alternative approach to conditional 3D shape generation that leverages paired texts/3D shapes or images/3D shapes to directly learn a conditional generative model from the text or image to the 3D shape representations. CLIP-Forge [56] employs an invertible normalizing flow model to learn a distribution transformation from the CLIP image/text embedding to the shape embedding. AutoSDF [39], ShapeFormer [68], and 3DILG [70] explore an auto-regressive model to learn a marginal distribution of the 3D shapes conditioned on images or texts and then sample a regular grid latent or irregular point latent shape embeddings from the conditions. In recent years, diffusion-based generative models have achieved tremendous success in text-to-image, video, and human motion generation. Several contemporaneous works, including SDFusion [13], Diffusion-SDF [28, 15], 3D-LDM [41], 3DShape2VecSet [71], and Shap-E [25], propose to learn a probabilistic mapping from the textual or visual inputs to the shape latent embeddings via a diffusion model. Since these approaches learn the prior information of the 3D shape data, they could improve the yield rate of high-quality shape generation. Moreover, there is no long-time optimization process, and the inference time is orders of magnitude faster than the optimization-based approaches. However, directly learning a conditional generative model to sample the 3D shape from the conditions might produce low-quality, less-diversity results due to the significant distribution gap between the shape and image/text spaces.

2.3 Contrastive Learning in 3D

Contrastive Language-Image Pre-training (CLIP) [48] has emerged as a fundamental model in 2D visual recognition tasks and cross-modal image synthesis by building the representation connection between vision and language within an aligned space. Recent works [12, 1, 53, 64] have extended the multi-modal contrastive learning paradigm to 3D. CrossPoint [2] learns the 3D-2D alignment to enhance the 3D point cloud understanding. PointCLIP [74] takes full advantage of the CLIP model pre-trained on large-scale image-text pairs and performs alignment between CLIP-encoded point cloud and 3D category texts to generalize the ability of 3D zero-shot and few-shot classification.

Network

Point

Cloud

3D Shape

Encoder

Shape Latent

Embeddings

!

!

"

"#

"

(!%&'(,*+&,()

"

(!%&'(,.(/.)

#

!

CLIP Image

Encoder

Image

a 3d model of guitar,

ESP KH-Ouija V2,

instrument.

CLIP Text

Encoder

!

!

!

*

!

.

$

!

!

(0)

!

!

(1)

#

*

#

.

a): Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE)

b): Aligned Shape Latent Diffusion Model (ASLDM)

Contrastive

Contrastive

3D Shape

Decoder

Volume Points

(x,y,z)

Reconstructed

Mesh

"

2

Occupancy

Marching

Cubes

→

ULIP [67], CLIP-goes-3D [16] and OpenShape [30] further learn a unified and aligned representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities to improve 3D understanding. While most of these works focus on 3D recognition tasks, establishing the connection between 3D recognition and generation tasks remains an under-explored problem.

3 Our Approach

The direct learning of a probabilistic mapping from images or texts to 3D shapes is prone to produce inconsistent results due to the significant distribution gap between the 3D shapes and the 2D images and texts. To address this issue, we propose an alignment-before-generation solution for cross-modal 3D shape generation, as illustrated in Figure 2. Our approach involves two models: the Shape-ImageText-Aligned Variational Auto-Encoder (SITA-VAE)(Section 3.1) and the Aligned Shape Latent Diffusion Model (ASLDM) (Section 3.2). The former model learns an alignment among the 3D shapes, images, and texts via contrastive learning and then reconstructs the shape embeddings back to the neural field. The latter model is based on the aligned space and is designed to learn a better conditional generative model from the images or texts to shape latent embeddings. By adopting this alignment-before-generation approach, we aim to overcome the challenges posed by the distribution gap and produce more consistent and high-quality results in cross-modal 3D shape generation.

3.1 Shape-Image-Text Aligned Variational Auto-Encoder

Our SITA-VAE contains four components: a pre-trained and fixed CLIP image encoder E i and CLIP text encoder E t , a trainable 3D shape encoder E s and neural field decoder D s . The CLIP image encoder and text encoder take 2D images I ∈ R H × W × 3 and tokenized texts T ∈ R L t × d t as input, and generate image tokens E i ∈ R (1+ L i ) × d and text tokens E t ∈ R L t × d , where (1 + L i ) and L t are the sequence length of image tokens E i and text tokens E t . We take advantage of the pre-trained image encoder and text encoder from CLIP. These two encoders are trained on large-scale image-text pairs and robust enough to capture a well-aligned vision-language space, which will enrich the semantics of the 3D shape representation after multi-modal alignment via contrastive learning.

3D shape encoder aims to extract powerful feature representations to characterize each 3D shape effectively. To achieve this, we first sample point clouds P ∈ R N × (3+ C ) from the surface of 3D shapes, where N represents the number of points, and C denotes additional point features such as normal or color. Next, we use a linear layer to project the concatenation of the Fourier positional encoded point clouds P to the 3D shape encoder input X ∈ R N × d . Drawing inspiration from previous transformer-based architectures for point cloud understanding [23], we build our 3D shape encoder on a perceiver-based transformer. Specifically, we use a cross-attention layer to inject the 3D shape information from the input X into a series of learnable query tokens Q ∈ R (1+ L s ) × d , where 1 + L s is the length of query tokens Q , consisting of one global head token Q g ∈ R 1 × d with highlevel semantics and L s local tokens Q l ∈ R L × d containing low-level geometric structure information. Then, several self-attention blocks are used to iteratively improve the feature representation and obtain the final shape embeddings, E s ∈ R (1+ L s ) × d .

Alignment among 3D shapes, images, and texts plays a crucial role in SITA-V AE and the conditional generative models. Since the 3D data is the order of magnitudes smaller than the images and texts data, to learn a better-aligned shape among 3D shapes, images, and texts, we enforce the 3D shape encoder close to a pre-aligned vision-language space, which is pre-trained on a large-scale image-text pair with rich image and text representations by leveraging the contrastive learning strategy. Consider an input pair of 3D shapes X , images I and tokenized texts T . The triplet encoders generate the corresponding shape embedding e s , image embedding e i and text-embedding e t by projecting the extracted shape tokens E s , image tokens E i and text tokens E t as three vectors with the same dimension, which is expressed as: e s = F s ( E s ) , e i = F i ( E i ) , and e t = F t ( E t ) , where F s is a learnable shape embedding projector, image embedding projector F i and text embedding projector F t are pre-trained and frozen during training and inference. The contrastive loss is:

where ( j, k ) indicates the positive pair in training batches, and since we utilize pre-trained encoders from CLIP, the model is free from constraint L ( image,text ) .

3D shape decoder , D s , takes the shape embeddings E s as inputs to reconstruct the 3D neural field in a high quality. We use the KL divergence loss L KL to facilitate the generative process to maintain the latent space as a continuous distribution. Besides, we leverage a projection layer to compress the latent from dimension d to lower dimensions d 0 for a compact representation. Then, another projection layer is used to transform the sampled latent from dimension d 0 back to high dimension d for reconstructing neural fields of 3D shapes. Like the encoder, our decoder model also builds on a transformer with the cross-attention mechanism. Given a query 3D point x ∈ R 3 in the field and its corresponding shape latent embeddings E s , the decoder computes cross attention iterative for predicting the occupancy of the query point O ( x ) . The training loss is expressed as:

where BCE is binary cross-entropy loss, and the total loss for training Shape-Image-Text Aligned Variational Auto-Encoder (SITA) is written as:

3.2 Aligned Shape Latent Diffusion Model

After training the SITA-VAE, we obtain an alignment space among 3D shapes, images, and texts, as well as a 3D shape encoder and decoder that compress the 3D shape into low-dimensional shape latent embeddings and reconstruct shape latent embeddings to a neural field with high quality. Building on the success of the Latent Diffusion Model (LDM) [51] in the text-to-image generation, which strikes a balance between computational overhead and generation quality, we propose a shape latent diffusion model on the aligned space to learn a better probabilistic mapping from 2D images or texts to 3D shape latent embeddings. By leveraging the alignment space and the shape latent diffusion model, we can generate high-quality 3D shapes that better conform to the visual or textural conditional inputs.

Our Aligned Shape Latent Diffusion Model (ASLDM) builds on a UNet-like transformer [52, 62, 5], aim to fit a distribution of the shape latent embeddings, accompanied by an auto-encoder for encoding data samples into the latent space and reconstructing the data samples given the sampled latent. By learning in the latent space, the latent diffusion model is computationally efficient, and leveraging such a compact representation enables the model to fit the target distribution faster. Specifically, the model ϵ θ focuses on generating shape latent embeddings E s conditioned on C , which is represented by the CLIP image or text encoder. Following LDM [51], the objective is

where t is uniformaly samppled from { 1 , ..., T } and E ( t ) s is a noisy version of E (0) s . During inference, sampling a Gaussian noise, the model gradually denoises the signal until reaching E (0) s . Followed with classifier-free guidance (CFG) [20], we train our conditional latent diffusion model with classifier-free guidance. In the training phase, the condition C randomly converts to an empty set ∅ with a fixed probability 10% . Then, we perform the sampling with the linear combination of conditional and unconditional samples:

where λ is the guidance scale for trading off the sampling fidelity and diversity.

4 Experiments

To validate the effectiveness of our proposed framework, we conducted extensive experiments. In this section, we provide implementation details of our model in Section 4.1. We also describe the data preparation process, including comparisons with baselines and metrics used in our evaluation, in Section 4.2. Of particular importance, we present quantitative comparison results to validate our model's generation ability. Additionally, we provide visual comparison results to illustrate the quality of the generative outputs in Section 4.3. Also, we conduct ablation studies in Section 4.4 to validate the effectiveness of training the generative model in the aligned space, the effectiveness of pre-trained vision-language models (VLM) on the SITA-VAE and the impact of learnable query embeddings.

4.1 Implementations

We implement our Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) based on perceiver-based transformer architecture [23], where the 3D shape encoder consists of 1 crossattention block and eight self-attention blocks. At the same time, the neural field decoder has 16 sefl-attention blocks with a final cross-attention block for the implicit neural field modeling. All attention modules are the transformer [62] style with multi-head attention mechanism (with 12 heads and 64 dimensions of each head), Layer Normalization (Pre-Norm) [3], Feed-Forward Network (with 3072 dimensions) [62] and GELU activation [17]. The learnable query embeddings are E ∈ R 513 × 768 with one head-class token for multi-modal contrastive learning and left 512 shape tokens with a linear projection layer to the V AE space ∈ R 512 × 64 for the 3D shape reconstruction. Moreover, we employ pre-train encoders in the CLIP ( ViT-L-14 ) [48] as our visual encoder and text encoder and freeze them during training and sampling. Besides, our aligned shape latent diffusion model (ASLDM) builds on a UNet-like transformer [52, 62, 5] consisting of 13 self-attention blocks with skip-connection by default. It contains 12 heads with 64 dimensions for each and 3076 dimensions in the Feed-Forward Network. Both models use an AdamW-based gradient decent optimizer [34] with a 1e-4 learning rate. Our framework is implemented with PyTorch [44], and we both train the SITA-VAE and ASLDM models with 8 Tesla V100 GPUs for around 5 days. We use the DDIM sampling scheduler [60] with 50 steps, which generates a high-quality 3D mesh within 10 seconds.

4.2 Datasets and Evaluation Metrics

Dataset . We use a standard benchmark, ShapeNet [11], to evaluate our model, which provides about 50K manufactured meshes in 55 categories. Each mesh has a category tag and corresponding texts, like fine-grained categories or brief descriptions given by the creator. We follow the train/val/test protocol with 3DILG [70]. We further collected 811 Cartoon Monster 3D shapes with detailed structures, with 615 shapes for training, 71 for validation, and 125 for testing, to evaluate the models' ability to generate complex 3D shapes. To prepare the triplet data (3D shape, image, text), we first

augment the provided texts in two ways. First, we string the shape tag and corresponding description in the format "a 3D model of ( shape tag ), in the style of ( description )" or "a 3D model of ( shape tag ), ( description )". Then, inspired by ULIP [67], we also leverage multiple templates containing 65 predefined phrases to provide more text information during training. As for the image data, we render each mesh under four camera poses, augmenting and improving the rendering diversity via the depth-condition-based ControlNet [73].

Metrics . We use the Intersection of Union (IoU) to reflect the accuracy of reconstructions. Then, we propose two new metrics for evaluating 3D shape generation methods. The first is a shape-image score (SI-S). We use a 3D shape encoder and image encoder to extract corresponding shape embedding and image embedding and compute the Cosine Similarity of these two modalities. Another is a shape-text score (ST-S), which computes the similarity between the generated 3D shape and the conditional text input in the aligned shape embedding and text embedding space. Both metrics evaluate the similarity between results and their corresponding conditions. Moreover, we use both the pre-trained ULIP [67] and our SITA to compute SI-S and ST-S, in terms of SI-S (ULIP), ST-S (ULIP), SI-S (SITA) and ST-S (SITA), respectively. Besides, we follow the metrics of P-IS and P-FID as introduced in Point-E [42] and use a pre-trained PointNet++ [47] to compute the point cloud analogous Inception Score [55] and FID [18] to evaluate the diversity and quality of the generated 3D shapes.

4.3 Experimental Comparision

Baselines . In the representation stage, we compare our method with Occ [37], ConvOcc [45], IFNet [14], 3DILG [70] and 3DS2V [71] on reconstruction tasks to valid the ability of the model to recover a neural field given shape embeddings on the ShapeNet dataset [11]. For the conditional generation stage, we choose the baselines of two recent powerful 3D generation methods, 3DILG and 3DS2V. We first finetune their shape representation module on a mixture dataset of the ShapeNet and the 3D Cartoon Monster. Then, we retrain the text and image conditional generative models of 3DILG and 3DS2V with all the same protocols as ours.

Numerical Comparison . We report the numerical results in Table 1 and Table 2. Table 1 shows that our model has the best reconstruction performance on 55 overall categories, surpassing the rest. Results of the selected category further prove that our model could faithfully reconstruct 3D shapes in each of the 55 categories. Table 2 reports the numerical results for conditional 3D shape generation. Our model achieves the best on all the SI-S and ST-S, indicating that it could map the information from the image or text to its corresponding 3D shape information for generating high-fidelity results. Moreover, the P-FID proves that our model could produce high-quality shape-tokens for generating realistic 3D shapes, and P-IS indicates the diversity of the samples. Specifically, the four left columns

show that our model surpasses the baselines on image-conditioned generation, proving that our model can better map visual information to 3D shapes. The four right columns validate the generative quality of text-conditioned generation. Since natural language, compared to the 2D image, usually provides limited and abstract information, learning a model to map text information to the 3D shape is challenging. However, benefiting from training on the aligned latent space, our model significantly improves text-conditioned generation, shown in the right of Table 2, which reflects that our model well-maps natural language information to 3D shapes and generates diverse and high-quality results.

Image

Condition

3DILG

3DS2V

Ours

Image

Condition

3DILG

3DS2V

Ours

Image

Condition

3DILG

3DS2V

Ours

一一楼

Text

Condition

3DILG

3DS2V

Ours

Text

Condition

3DILG

3DS2V

Ours

Text

Condition

3DILG

3DS2V

Ours

airplane, Flying

Aircraft Carrier

*

aeroplane,

space ship

*

vessel, StarLite

25' Sailboat

*

chair, Banqueta

*

suitcase, Leather

Suitcase

*

Litter CRTR-32

*

flying monster,

Cute monster in

the form of flying

birds, with 2 feet

*

sandshrew monster,

Cute monster in the

form of mountain

armor, with 2 feet

and 2 hands

*

standing humanoid

shape of the

monster, with 2

feet and 2 hands

*

standing monster,

Cute four -legged

monster, with 4 feet

*

fire monster, Cute

flying monster

in the shape of a

dragon, with 2 feet

and 2 hands

*

bug monster,

flying monster in

the form of

mosquitoes with 4

feet

*

*denotes 'a 3D model of …'

Visual Comparison . The visual comparisons of the image- and text-conditional 3D shape generations are illustrated in Figure 3 and Figure 4. Figure 3 shows that 3DILG [70] pays more attention to the global shape in the auto-regressive generation process, where its results lack depictions of details of

3D shapes. While 3DS2V [71] generates more details of 3D shapes, the results have discontinuous or noisy surfaces. Besides, both methods are unstable to generate a complete shape when the given conditions map to a complex object, fine machine, or rare monster. Figure 4 shows the visual comparison of text-conditional generation. In the upper-half rows, we show the results given simple and abstract concepts, while in the lower-half rows, we show the results given detailed texts like descriptions for deterministic parts of the target shape. Similar to the observation above, 3DILG [70] generates an over-smooth shape surface with fewer details, and 3DS2V [71] produces fewer details on the discontinuous object surface. Therefore, only our model produces correct shapes that conform to the given concepts or detailed descriptions with delicate details on smooth surfaces.

4.4 Ablation Studies and Analysis

We ablation study our model from three perspectives: the effectiveness of training generative model in the aligned space, the effectiveness of vision-language models (VLMs) on the SITA-V AE, and the impact of learnable query embeddings.

The effectiveness of training generative model in the aligned space . We perform a visual comparison for ablation study the effectiveness of training the generative model in the aligned space, as illustrated in the Figure 5. The uppers are sampled from the generative model that trains in the aligned space, while the lowers are sampled from the generative model that trains without aligned space. It proves that the uppers conform to the given text and the lower do not, which indicates that training the generative model in the aligned space leads to high-fidelity samples.

a 3d model of table, regulation table, tennis table, bench.

a 3d model of vintage car, 1938 Alfa Romeo 8C 2900B.

a)

With

Aligned Space

b)

Without

Aligned Space

The effectiveness of vision-language models . Except for the well-known vision-language model (VLM) CLIP [48], we introduce another vision-language model (VLM) SLIP [40] for training the SITA-VAE for a comprehensive comparison. First, we evaluate the impact of the vision-language model on SITA-VAE's reconstruction ability, and the results are shown in Figure 6. It shows that our model composed with CLIP achieves the best performance. Then, we evaluate the visionlanguage model's impact on the ability to align multi-modal space. We select standard and zero-shot classification tasks to reflect the impact of the vision-language model. Note that the classification is performed by a feature matching operation, where we provide multiple 3D shapes and phrases to the SITA-VAE; it returns the similarity between 3D shapes to each phrase as classification results, which indicates that the more the multi-modal space is aligned, leading the higher classification accuracy. The results show that our model composed with CLIP achieves the best performance.

The impact of the learnable query embeddings . Weablation study learnable query embeddings with the same experiments as the above, and the results show that using 512 learnable query embeddings leads to the best performance on reconstructions and classifications.

4.5 Nearest Neighbor Analysis

In order to verify whether the model learns to generate results based on the given conditions or memorizes the whole training split of the dataset, we traverse the training set to retrieve the nearest

IoU

256

Standard Classification

256

Zero-shot (Top 1)

256

Zero-shot (Top5)

256

512

512

512

512

0.97

0.9

0.5

0.96

0.89

0.4

0.7

0.95

0.88

0.3

0.6

w/ocontrast

w SLIP

wCLIP

W SLIP

WCLIP

WSLIP

WCLIP

wSLIP

W CLIP

neighbors of the generated samples, as illustrated in Figure 7. Specifically, we exhibit three nearest neighbors for each sample for comparison. According to the visualization, the generated 3D shape differs from each retrieved nearest neighbor, demonstrating that our model hallucinates a new 3D shape rather than overfitting the training sets.

Image

Condition

Generated

Mesh

Nearest

Neighbor-1

Nearest

Neighbor-2

Nearest

Neighbor-3

Image

Condition

Generated

Mesh

Nearest

Neighbor-1

Nearest

Neighbor-2

Nearest

Neighbor-3

Image

Condition

Generated

Mesh

Nearest

Neighbor-1

Nearest

Neighbor-2

Nearest

Neighbor-3

5 Discussion and Conclusion

Though our method has achieved excellent results, it still has some limitations. First, our method needs the ground truth 3D shapes from training, while 3D data is usually an order of magnitude smaller than the 2D data. Learning the shape representation with a 3D shape-image-text aligned space from only 2D (multi-view) images via differentiable rendering is a promising direction. Furthermore, since we represent each 3D shape as an occupancy field, it needs to convert the 3D mesh into a watertight one, which will inevitably degrade the original quality of the 3D mesh.

In conclusion, we propose a novel framework for cross-modal 3D shape generation that involves aligning 3D shapes with 2D images and text. We introduce a new 3D shape representation that can reconstruct high-quality 3D shapes from latent embeddings and incorporate semantic information by aligning 3D shapes, 2D images, and text in a compatible space. This aligned space effectively closes the domain gap between the shape latent space and the image/text space, making it easier to learn a better probabilistic mapping from the image or text to the aligned shape latent space. As a result, our proposed method generates higher-quality and more diverse 3D shapes with greater semantic consistency that conform to the conditional image or text inputs.

Acknowledgement The work was supported by NSFC #61932020, #62172279, Program of Shanghai Academic Research Leader, and 'Shuguang Program' supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission

References

Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Goodman, and Leonidas J Guibas. Shapeglot: Learning language for shape differentiation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8938-8947, 2019.

Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9902-9912, June 2022.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: a vit backbone for score-based diffusion models. arXiv preprint arXiv:2209.12152 , 2022.

Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: clothed 3d humans. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 344-359. Springer, 2020.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint arXiv:2304.08818 , 2023.

Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236 , 2016.

Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16123-16133, 2022.

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University -Princeton University - Toyota Technological Institute at Chicago, 2015.

Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14 , pages 100-116. Springer, 2019.

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv , 2022.

Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, jun 2020.

Gene Chou, Yuval Bahat, and Felix Heide. Diffusionsdf: Conditional generative modeling of signed distance functions. arXiv preprint arXiv:2211.13757 , 2022.

Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303.11313 , 2023.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Transactions on Graphics (TOG) , 41(4):1-19, 2022.

Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023.

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 46514664. PMLR, 2021.

Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022.

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 , 2023.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Yichong Leng, Zehua Chen, Junliang Guo, Haohe Liu, Jiawei Chen, Xu Tan, Danilo Mandic, Lei He, Xiangyang Li, Tao Qin, et al. Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances in Neural Information Processing Systems , 35:23689-23700, 2022.

Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. arXiv preprint arXiv:2305.10764 , 2023.

Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. Iss: Image as stetting stone for text-guided 3d shape generation. arXiv preprint arXiv:2209.04145 , 2022.

Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. Iss++: Image as stepping stone for text-guided 3d shape generation. arXiv preprint arXiv:2303.15181 , 2023.

Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1789617906, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Andrew Luo, Tianqin Li, Wen-Hao Zhang, and Tai Sing Lee. Surfgen: Adversarial 3d shape synthesis with explicit surface discriminators. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16238-16248, 2021.

Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black. Learning to dress 3d people in generative clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6469-6478, 2020.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4460-4470, 2019.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 306-315, 2022.

Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets languageimage pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 529-544. Springer, 2022.

Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 , 2022.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 , 2022.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 165-174, 2019.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 523-540. Springer, 2020.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv , 2022.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Yue Ruan, Han-Hung Lee, Ke Zhang, and Angel X Chang. Tricolo: Trimodal contrastive loss for fine-grained text to shape retrieval. arXiv preprint arXiv:2201.07366 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18603-18613, 2022.

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or. Human motion diffusion model. In International Conference on Learning Representations (ICLR) , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399 , 2022.

Ye Wang, Bowei Jiang, Changqing Zou, and Rui Ma. Mxm-clr: A unified framework for contrastive learning of multifold cross-modal representations. arXiv preprint arXiv:2303.10839 , 2023.

Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems , 29, 2016.

Chen Xin, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023.

Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. arXiv preprint arXiv:2212.05171 , 2022.

Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6239-6249, 2022.

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Biao Zhang, Matthias Nießner, and Peter Wonka. 3dilg: Irregular latent grids for 3d generative modeling. In NeurIPS , 2022.

Biao Zhang, Jiapeng Tang, Matthias Nießner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Trans. Graph. , 42(4), jul 2023.

Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8542-8552, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a conditional 3D generation method by pre-aligning features of different modalities when training a 3D AE. After that, a latent diffusion model is applied to generate latent vectors for 3D decoder conditioned on text/image. The results quantitatively and qualitatively shows that the proposed method is able to achieve impressive results.\n\nSTRENGTHS:\n1. The results show that this method is able to achieve multiple tasks, which is good and important for a generation-related paper.\n2. The main figure is easy to understand and follow, making this paper easier to read. \n3. The core idea in this paper looks original.\n\nWEAKNESSES:\n1. No qualitative results show the diversity of the proposed method. Such a result is important to a generation-related paper.\n2. The concept of pre-alignment is not very convincing. Other conditional generation methods are also aligning text/image to the shape latent space by either pulling their representations together or learning diffusion models. Further discussion on how this concept is better than others should be added.\n\nQUESTIONS:\n1. If you have paired data for text-image-shape alignment training, why not just directly train a latent diffusion model conditioned on image/text to generate 3D objects like SDFusion[1]? In contrast, if you already aligned text/image/shape features in latent space, is it still necessary to use a heavy conditional generator such as latent diffusion to achieve your tasks? It feels like you only need one of these two techniques. \n2. In your qualitative comparison, you mentioned that 3DS2V generates more details of 3D shapes and discontinuous surfaces and noisy results. However, such results are not seen in their paper, I wonder the reason why they fail to produce good results in Figure 2 and 3. Is it because you re-train their model on your dataset or because you directly use their pre-trained checkpoint or other reasons? More discussion and insight should be addressed.\n3. You pre-aligned the text/image/shape feature first and use text/image feature to train a latent diffusion model to achieve conditional generation. However, as far as I know, if the input condition and the target feature are too similar when training the diffusion model, the diversity of the generation result would be limited and the purpose of using the diffusion model would disappear. How do you avoid such a situation from happening?\n4. Please answer the problem mentioned in the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is about 3D shape generative model focused on image-conditioned and text-conditioned generation. The authors aligned the latent space of a shape autoencoder to CLIP's image encoder and text encoder. Then generative diffusion models are trained on the aligned latent space. This enables shape generation given image or text as conditional input. The authors showed some good results in both tasks.\n\nSTRENGTHS:\nThe authors showed some good generation results for both the task of image-conditioned generation and text-conditioned generation. The writing is also clear and easy to follow.\n\nWEAKNESSES:\n1. The autoencoder network (Fig 1 a) is similar to the network used in [63]. Thus the performance boost shown in Table 1 seems to be because of CLIP. This should be emphasized or ablated somewhere in the main paper.\n2. Following the above comment (Table 1), the authors only compared with \"Learned queries\" results from [63]. According to [63], another design \"Point queries\" achieved better results than both this work and \"Learned queries\".\n3. It would be better if the authors can show some visualization comparisons of the autoencoding results.\n4. L136, the length is (1+L_i) instead of L_i. This is explained in later sections (L217) but is still causing confusion.\n5. Another difference with respect to [63] is, when training the diffusion models, this work used a unet-style transformer [4] instead of a simple stacked self attention network.\n6. Some pieces of writing can still be improved. For example, when talking about a design or an equation, we should mention why we are doing this or discuss some insights behind this.\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nThis work proposed a method for image-conditioned and text-conditioned shape generation. The authors combined several components (3DS2V-style shape autoencoding network, clip and unet-style transformer denoising network). All these components are not proposed by this work which weakens the novelty of this paper. However, I still believe the authors delivered some good results in shape generation. I hope the authors can clarify my concerns mentioned in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a conditional generation model which aims to solve the alignment issue in image-to-shape or text-to-shape generation. The key idea is to learn a aligned representation among 3D shapes, images, and texts. To achieve that, the author proposes SITA-VAE with contrastive loss to force the shape's latents to be aligned with the pretrained vision-language model. After that, a LDM is trained to learn the diffusion process in the latent space. In the test time, it follows the previous works to use classifier-free guidance to perform conditional generation. The proposed method is evaluated on ShapeNet and Cartoon Monster 3D shapes.\n\nSTRENGTHS:\n* Propose to learn a aligned space for 3D shapes, images, and texts with 2D vision-language model. This essentially leverages the abundant 2D data into 3D modeling. The scarcity of paired data is one crucial reason why conditional 3D modeling does not perform as well as its 2D counterpart.\n\n* The conditional generation results look great and align with the inputs when compared with the baselines.\n\n* It is surprised that the aligned space does hurt the reconstruction results.\n\nWEAKNESSES:\n* Novelty is somewhat limited. The first stage model is mostly based on 3DILG and the multi-modality conditional generation has been explored in previous work such as SDFusion. \n\n* Scalability is an issue as the model need paired 3D-image-text data to work.\n\n* Some parts of the writing and figure are misleading. The choices of the metric is problematic in Table 2.\n\nPlease see the \"Questions\" for the details.\n\nQUESTIONS:\n1. The results in Fig. 2 seems a bit too sharp and I wonder if the model is memorizing the outputs. Can the author shows the nearest neighbor sample from the datasets?\n\n2. In figure 1, the model is outputting occupancy but the final result is denoted as \"mesh\". I think the output should be occupancy and the mesh is obtained via marchingcube, right? Learning to generate mesh directly is very different from generating occupancies. The writing also use the word \"mesh\" and it is a bit confusing.\n\n3. In Table 2, the use of SITA to evaluate the performance of conditional generation is unfair compared to other baselines since the LDM is trained on the space learned with SITA.\n\nLIMITATIONS:\nAuthors have address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposed a VAE-based text-to-3D shape generation method. The authors designed an alignment-before-generation approach to narrow the gap between 3D shapes and the 2D or text condition. They first train a Shape-Image-Text-Aligned Variational Auto-Encoder to align the representations between the 3D shapes and the 2D or text inputs. Then, the use latent diffusion model to denoise the shape embeddings to match the conditions. The overall presentation is good and the experiments are extensive.

STRENGTHS:
1. The narratives are good and sound.
2. The SITA-VAE helps to align the representations between the three modalities.
3. The generation only requires some denoising steps on the latent embeddings.
4. The experiments are extensive and the authors provide sufficient visual demos and show the effectiveness of the method.

WEAKNESSES:
1. It seems that shape-image-text align training requires a lot of paired shape-image-text data, which could be a huge challenge in the generalization to new categories.
2. The experimental implementation is not clear enough and could be improved.

QUESTIONS:
1. Since this method is trained with ground truth 3D data, which limited its generalization ability, would it be better to show some failure cases when the condition fails to lie in the aligned representation space?

LIMITATIONS:
The author mentioned the limitation of the requirement of the 3D training data, which could be a huge problem in the generalization of this method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07