ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation
Authors: Zhengzhe Liu
Peng Dai
Ruihui Li
XIAOJUAN QI
Chi-Wing Fu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Text
3D shape
CLIP
differentiable rendering

Abstract: Text-guided 3D shape generation remains challenging due to the absence of large paired text-shape dataset, the substantial semantic gap between these two modalities, and the structural complexity of 3D shapes. This paper presents a new framework called Image as Stepping Stone (ISS) for the task by introducing 2D image as a stepping stone to connect the two modalities and to eliminate the need for paired text-shape data. Our key contribution is a two-stage feature-space-alignment approach that maps CLIP features to shapes by harnessing a pre-trained single-view reconstruction (SVR) model with multi-view supervisions: first map the CLIP image feature to the detail-rich shape space in the SVR model, then map the CLIP text feature to the shape space and optimize the mapping by encouraging CLIP consistency between the input text and the rendered images. Further, we formulate a textguided shape stylization module to dress up the output shapes with novel structures and textures. Beyond existing works on 3D shape generation from text, our new approach is general for creating shapes in a broad range of categories, without requiring paired text-shape data. Experimental results manifest that our approach outperforms the state-of-the-arts and our baselines in terms of fidelity and consistency with text. Further, our approach can stylize the generated shapes with both realistic and fantasy structures and textures. Codes are available at https://github.com/liuzhengzhe/ISS-Image-as-Stepping-Stone-for-Text-Guided-3D-Shape-Generation.

Published as a conference paper at ICLR 2023
ISS: IMAGE AS STEPPING STONE FOR TEXT-GUIDED 3D SHAPE GENERATION
Zhengzhe Liu'
Peng Dai
Ruihui Li
Xiaojuan Qi
Chi-Wing Ful
The Chinese University of Hong Kong
Thh University of Hong Kong
HHunan University
ABSTRACT
Text-guided 3D shape generation remains challenging due to the absence of large paired text-shape dataset, the substantial semantic gap between these two modal- ities, and the structural complexity of 3D shapes. This paper presents new framework called Image as Stepping Stone (ISS) for the task by introducing 2D im- age stepping stone to connect the two modalities and to eliminate the need for paired text-shape data. Our key contribution a two-stage cnttur-spaace-llinene approach that maps CLIP features to shapes by harnessing pre-trained single-view reconstruction (SVR) model with multi-view supervisions: first map the CLIP im- age feature to the detail-rich shape space in the SVR model, then map the CLIP text feature to the shape space and optimize the mapping by encouraging CLIP consis- tency between the imput text and the rendered images. Further, we formulate text- guided shape stylization module to dress up the output shapes with novel structures and textures. Beyond existing works 3D shape generation from text, new approach is general for creating shapes in a broad range of categories, without re- quiring paired text-shape data. Experimental results manifest that our approach out- performs the state-of- state-of-the-arte arts and our baselines in terms of fidelity and consistency with text. Further, our approach can stylize the generated shapes with both realistic and fantasy structures and textures. Codes are available at Intpps: //gs::///tttt www....................oo..... com/
Liuahençahe/ T35 1-5--3ag-eee Imagemmeee.. as Stepping Stone- for Text Guided 3D- Shape Generat lon
Text feelure space
CLIP Centure space
stape feeture
Ours
Clip-Forge
N'(f)
MUS
foature
CLIP Text
Mapper
Mapper
SVR fisturr
decoder
shape
Dream Fields
SVR
Stage
Stage2
Stage
Target shape foature
gram
mushcoome
(a) Our "Image us Stepping Stone" Framework
(b) Text:
fgreen, ninbow, mushroom) hodside lamp.
Figure 1: Our novel "Image Stepping Stone" framework (a) is able connect the text space (the CLIP Text feature) and the 3D shape space (the SVR feature) through our two-stage feature-space aligmment, such that we can generate plausible 3D shapes from text (b) beyond the capabilities the existing works (CLIP-Forge and Dream Fields), without requiring paired text-shape data.
INTRODUCTION
3D shape generation has broad range of applications, e.g., in Metaverse, CAD, games, animations, etc. Among various ways to generate 3D shapes, a user-friendly approach is to generate shapes from natural language or text descriptions. By this means, users can readily create shapes, e.g., to add/modify objects VR/AR worlds, to design shapes for 3D printing, etc. Yet, generating shapes from texts is very challenging, due to the lack of large-scale paired text-shape data, the large semantic gap between the text and shape modalities, and the structural complexity of 3D shapes.
Existing works (Chen et al.. 2018: Jahan et al.) 202 Liu et al. 2022 typically rely paired text- shape data for model training. Yet, collecting 3D shapes is already very challenging on its own, let alone the tedious manual annotations needed to construct the text-shape pairs. To our best knowledge, the largest existing paired text-shape dataset (Chen et al.) 2018) contains only two categories, i.e., table and chair, thus severely limiting the applicability of the existing works.

---

Published as a conference paper at ICLR 2023
Very recently, two annotation-free approaches. CLIP-Forge (Sanghi et al., 2022) and Dream Fields (Jain et al.; (022). were proposed to address the dataset limitation. These two state-of- are approaches attempt to utilize the joint text-image embedding from the large-scale pre-trained language vision model, i.e. CLIP (Radford et al.] 2021), to eliminate the need of requiring paired text-shape data in model training. However, it is still extremely challenging to generate 31) shapes from text without paired texts and shapes for the following reasons. First, the range of object categories that can be generated are still limited due to the scarcity of 3D datasets. For example, Clip-Forge (Sanghi et al. 2022) is built upon a shape auto-encoder; it can hardly generate plausible shapes beyond the ShapeNet categories. Also, it is challenging to learn 3D prior of the desired shape from texts. For instance, Dream Field (Jain et al. 2022) cannot generate 3D shapes like our approach due to the lack of 3D prior, as it is trained to produce only multi-view images with a neural radiance field. Further, with over an hour of optimization for each shape instance from scratch, there is still no guarantee that the multi-view consistency constraint of Dream Field (Jain et al. 2022) can enforce the model for producing shapes that match the given text; we will provide further investigation in our experiments. Last, the visual quality of the generated shapes is far from satisfactory due to the substantial semantic gap between the unpaired texts and shapes. As shown in Figure [] (b), the results generated by Dream Field typically look surrealistic (rather than real), due to insufficient information extracted from text for the shape structures and details. On the other hand, CLIP-Forge (Sanghil et al. 2022) is highly restricted by the limited 64 resolution and it lacks colors and textures, further manifesting the difficulty of generating 3D shapes from unpaired text-shape data.
Going beyond the existing works, we propose a new approach to 3D shape generation from text without needing paired text-shape data. Our key idea is implicitly leverage 2D image as a stepping stone (ISS) to connect the text and shape modalities. Specifically, we employ the joint text-image embedding in CLIP and train a CLIP2Shape mapper to map the CLIP image features to a pre-trained detail-rich 3D shape space with mulli-view supervisions; see Figure (a): stage 1. Thanks to the joint text-image embedding from CLIP, our trained mapper is able to connect the CLIP text features with the shape space for text-guided 3D shape generation. Yet, due to the gap between the CLIP text and CLIP image features, the mapped text feature may not align well with the destination shape feature; see the empirical analysis in Section 3.2 Hence, we further fine-tune the mapper specific to each text input by encouraging CLIP consistency between the rendered images and the input text to enhance the consistency between the input text and the generated shape; see Figure (a): stage 2.
Our new approach advances the frontier of 3D shape generation from text in the following aspects. First, by taking image as stepping stone, we make the challenging text-guided 3D shape generation task more approachable and cast it as a single-view reconstruction (SVR) task. Having said that, we learn 3D shape priors from the adopted SVR model directly in the feature space. Second. benefiting from the learned 3D priors from the SVR model and the joint text-image embeddings, our approach can produce 3D shapes in only 85 seconds vs. 72 minutes of Dream Fields (Jain al. 2022). More importantly, our approach is able to produce plausible 3D shapes, not vueti-view images, beyond the generation capabilities of the state-of-the- approaches; see Figure (bb.
With our two-stage feature-space alignment, we already can generate shapes with good fidelity from texts. To further enrich the generated shapes with vivid textures and structures beyond the generative space of the pre-trained SVR model, we additionally design text-guided stylization module to generate novel textures and shapes by encouraging consistency between the rendered images and the text description of the target style. We then effectively fuse with the two-stage feature-space alignment to enable the generation of both realistic and fantasy textures and also shapes beyond the generation capability of the SVR model; see Figure (b) for examples. Furthermore, our approach is compatible with different SVR models (Niemeyer et al. 2020 Alwala et al.) (022).. For example, we can adopt SS3D Alwala et al. 2022) to generate shapes from single-view in-the-wild images to broaden the range of categorical 3D shapes that our approach can generate, going beyond Sanghi let al.) (2022). which can only generate 13 categories of ShapeNet. Besides, our approach can also work with the very recent approach GET3D (Gao et al., 2022) to generate high-quality 3D shapes from text; see our results in Section
2 RELATED WORKS
Text-guided image generation. Existing -ext-g image generation approaches can be roughly cast into two branches: (i) direct image synthesis (Reed et al. 2016a by /Zhang et al. 2017 2018
2

---

Published as a conference paper at ICLR 2023
Xu et al. 2018 Li et al.
2019 2020; Qiao et al. 2019 Wang et al. 2021 and (ii) image generation
with pre- pre-trained GAN Stap et al. [000 Yuan & Peng 2019 Souza et al. 2020; Wang et al. 2020
Rombach al. 2020 Patashmik et al. 202 Xia et al.
2021]. Yet, the above works
only generate
images for limited categories. To address this issue, some recent works explore zero-shot text-guided
image generation (Ramesh et al. (02T Ding al. 202T] Nichol et 2021 Liu et al. [022 Ramesh
ct al. 2022) to learn to produce images of any category. Recently, Zhou et al. (2022) and Wang
al. 2022b) leverage CLIP for text-free texi-to-image generation. Text-guided shape generation
more challenging compared with text-to-image
generation. First, it is far more labor-intensive and
difficult to prepare large amount of paired text-shape data than paired text-image data, which can be collected from the Internet on large scale. Second, the text-to-shape task requires one to predict full 3D structures that should be plausible geometrically and consistently in all views, beyond the needs in single-view image generation. Third, 3D shapes may exhibit more complex spatial structures and topology, beyond regular grid-based 2D images.
Text-guided 3D generation. To generate shapes from text, several works (Chen al. 2018 Jahan et al. 2021: Liu et al. 2022) rely on paired text-shape data for training. To avoid paired text- shape text-shape
data, two very recent works, CLIP-Forge (Sanghi et al.. [022) and Dream Fields (Jain et al. 2022). attempt to leverage the large-scale pre-trained vision- laniual model CLIP. Yet, they still sufter from various limitations, as discussed in the third paragraph of Section [ Besides 3D shape generation, some recent works utilize CLIP to manipulate a shape or NeRF with text (Michel et al.] [202]] Jetchev 2021; Wang et al. 2022a) and generate 3D avatars (Hong et al.] (022).. In this work, we present a new framework for generating 3D shape from text without paired text-shape data by our novel two-stage feature-space alignment. Our experimental results demonstrate the superiority of this work beyond the existing ones in terms of fidelity and text-shape consistency.
Single-view reconstruction. Another topic related to this work is singeeview reconstruction (SVR). Recently, researchers have explored SVR with meshes (Agarwal & Gopi 2020), voxels (Zuicc Lio 202T), and 3D shapes (Niemeyer et al.] 2020p. Further, to extend SVR to in-the-wild categories, Alwala et al. (2022) propose SS3D to learn 3D shape reconstruction using single-view images in hundreds of categories. In our work, we propose to harness an SVR model to map images to shapes, such that we can take 2D image as a stepping stone for producing shapes from texts. Yet, we perform the mapping and feature alignment implicitly in the latent space rather than explicitly.
METHODOLOGG
3.1 OVERVIEW
This work aims to generate 3D shape S from text T. Overall, our idea to map the CLIP features to the shape space of a pre-trained SVR model, such that we can leverage the joint text-image embeddings from CLIP and also the 3D generation capability of the SVR model enhance the generation of 3D shape from text. Hence, our method only needs to be trained with multi view RGB or RGBD images and the associated camera poses without paired text-shape data. As Figure FigureBssows, shows, our framework includes (i) image encoder Ess which maps input image I to SVR shape space Ωs (ii) pre-trained CLIP text and image encoders Er and E,, which map text T' and image I to CLIP layers, each followed by Leaky-Re and (iv) decoder D to generate the final shape S. Specifically, we use DVR (Niemeyer al. 2020 as the SVR model when presenting our method, unless otherwise specified.
spaces Qy and 24 respectively, (iii) mapper M with 12 fully-connen
Overall, we introduce novel two-stage feature-ppace-aligmmend approach to connect the text, image, and shape modalities. In detail, we first train CL.IPShhppe mapper M to connect the CLIP image space Q4 and the shape space Ωg from the pre-trained SVR model (see Figure 2 (a)). Then, we fine-tune M test time using a CLIP consistency loss Le to further connect the CLIP text space Or with S2g (see Figure 2 (b)). Last, we may further optimize the texture and structure style of S by fine-tuning the decoders (see Figure
In the following, we first introduce two empirical studies on the CLIP feature space in Section 3.2 then present our two-stage presents our text-guided shape stylization method and Section 3.5 discusses the compatibility of our approach with different SVR models and our extension to generate a broad range of categories.
approach in Section 3.3] Further, Section 3.4
3

---

Published as a conference paper at ICLR 2023
Stage 1
Stop gradient
R
M
S
121
R
(b) S Stage 2
R
wooden table.
Dr
Er
Text-G Stylization
A mahoqany
table.
Dr
Frozon
Update
Texture stylizatty
[] Shape-and-testuree stylization
Figure Overview of our text-guided 3D shape generation framework, which has three major stages. (a) Leveraging pre-trained SVR model, in stage-1 feature-space alignment, we train the CLIP2Shape mapper M to map the CLIP image feature space S4 to shape space Q2 of the SVR model with Es. Ej frozen, and fine-tune decoder D with an additional background loss Log. M and D are trained with their own losses separately at the same time by stopping the gradients from SVR loss Ln and background loss Lng to propagate M. (b) In stage-2 feature- space alignment, D and fine-tune M into M' by encouraging CLIP consistency between input text T and the rendered images test time. (c) Last, optimize the style of the generated shape and texture of S for T. At the inference, we use stage 2 to generate 3D shape from T and (c) is optional.
fix
mlaU(%)
M(D
47,86
f: barget shape
40.09
0.99
M'
MCG): mapped image
40
Mi())): mapped text fenture
M (f(f:) maeped text festure
Input image
lig
Ey +
after stago
F 62
M(F+)
(a) 3D reconstruction visualization
(b) 3D reconstruction results
(c) Average feature distance on the test set
Figure Empirical studies on the CLIP feature space for text-guided 3D shape generation.
3.2 EMPIRICAL STUDIES AND MOTIVATIONS
Existing works (Sanghi et al., 2022 Zhou et al. 2022 Wang et al. 2022b) mostly utilize CLIP directly without analyzing how it works and discussing its limitations. To start, investigate the merits and drawbacks of leveraging CLIP for text-guided 3D shape generation by conducting the following two empirical studies to gain more insight into the CLIP feature space.
3.2.1 WHETHEE CLIP FEATURE SPACE SUFFICIENTLY GOOD FOR 3D SHAPE GENERATION?
First, we study the representative capability of the CLIP image feature space 24 by trying to generate 3D shapes from this space. Specifically, we replace the SVR image encoder Es with the CLIP image encoder Eg. and optimize implicit decoder D using multi-view losses like DVR (Niemeyer et al.) 2020) with Ej frozen. This approach can be extended to text-to-shape generation by replacing Ej with CLIP text encoder ET during the inference. To compare the performance of Es and E, we evaluate 3D mloU between their generated shapes and GT'. The results are as follows: the standard SVR pipeline Es+D achieves 47.86% mloU while replacing the SVR encoder Es with CL.PP encoder Ei (E++D) degrades the performance to 40.09%. From the results and qualitative comparison shown in Figures 3 (a. b), we can see that the CLIP image space Q has inferior representative capability to capture details of the input image for 3D shape generation. This is not surprising, since the pre- pre-trained Ei from CLIP is targeted to extract semantic-aligna allgged features from texts rather than extracting details from images. Hence, image details relevant to 3D reconstruction are lost, e.g., textures. On the contrary, Es from the SVR model is optimized for 3D generation from images, it maintains more
4

---

Published as a conference paper at ICLR 2023
necessary details. The above result motivates to design a mapper M from Q to Us and then generate shapes from Sgy instead of S for better generative fidelity.
3.2.2 How THE CLIP IMAGE AND TEXT FEATURE GAP INFLUENCES 3D SHAPE GENERATION?
Second, we investigate the gap between the normalized CLIP image feature fi E On and normalized CLIP text feature fT E ΩT: (see also the CLIP image and text feature spaces in Figure | (a)) and how such gap influences text-guided 3D shape generation. Specifically, we randomly sample 300 text- shape pairs from the text-shape dataset (Chen et al., 2018), , then evaluate the cosine distance between fi and fr, i.e. d = 1 cosine_ similarity(fnr fr), where fi is the CLIP feature of the rendered images from the corresponding shape. We repeat the experiment and obtain d(fr. fi) 0.783 + 0.004. The result reveals a certain gap between the CLIP text and image features in this dataset, even though they are paired. Also, the angle in the feature space between the two features is around arccos(1 - 0.783) = 1.35 rad in this dataset (Chen et al. 20188) Having said that, directly replacing fi with fr like (Sanghi et al. (0022); Zhou et al. (2022) in inference may harm the consistency between the output shape and the input text. As demonstrated in Figure 3 (c), directly replacing fi with fr causes cosine distance of 0.45 to fs E Ig (see Figure which is significantly larger than the distance between (Ji) and /s (0.21). Our finding is consistent with the findings in Liang et al.
(2022). It motivates us further fine-tune M into M' at test time, such that we can produce feature M (T). which is closer to fs than M (fr).
3.3 TWO-STAGE FEATURE- SPACE ALIGNMENT
Following the above findings, propose two-stage feature-space- spacc-alignmeet approach to first connect image space 22 and shape space Ωg and further connect text space Qr to shape space Ωg with the image space S4 the stepping stone.
Stage-1 alignment: CLIP image-to-shape mapping. Given multi-view RGB or RGBD images for training. the stage-1 alignment is illustrated in Figure 2 (a). Considering that shape space Qg contains richer object details than the image space 24, while 14 provides joint text-image embedding with the input text space ST, we introduce fully-connected CLIP2Shape mapper M to map image feature fi to shape space D2. Taking rendered image I as input, M is optimized with an Le regression between M (fi) and standard SVR feature fs = EI( according to Equation below:
N
LM = Σ [IM(ss)) - (T(
where is the number images in the training set and fi,; is the normalized CLIP feature of I,-
Also, we fine-tune decoder D to encourage it to predict a white background, which helps the model
to ignore the background and extract object-centric feature (see Figure 4. while maintaining its 3D
shape generation capability. To this end, propose new background loss Log in Equation 2 below to enhance the model's foreground object awareness to prepare for the second-stage alignment.
Log Σ1ID.b) - 1113 (ray(o,)) n F = 90)
(2)
where 1 is the indicator function; F (p: D.(p) > O) indicates the foreground region, in which the occupancy prediction DD(p) is larger than threshold (; is a query point; ray p) n F = means the ray from camera center through p does not intersect the foreground object marked by F: and Do(p) is the color prediction at query point p. In a word, Log encourages D to predict the background region white color (value 1), such that Eg can focus on and better capture the foreground object. In addition, to preserve the 3D shape generation capability of D, we follow the loss Lp. with which D has been optimized in the SVR training. In this work, we adopt Niemeyer et al. (2020).
Hence, the overall loss in stage-1 training is AM for mapper M and Aug Log + Lp for decoder D, where AM and Ang are weights. The stage-1 alignment provides a good initialization for the test-time optimization of stage 2.
Stage-2 alignment: text-to-shape optimization. Given a piece of text, the stage-2 alignment aims to further connect the text and shape modalities. Specifically, it searches for shape S that best matches
5

---

Published as a conference paper at ICLR 2023
iter 0
iter
iter
iter
result
0
iter 5
iter
iter
result:
A red car:
(a) without Log
(bb with
Figure 4: Effect of generating shapes from the same text with/without background loss Log-
the input text T. To this end, we formulate fast test-time optimization to reduce the gap between the text and image CLIP features fr and fi, as discussed earlier in the second empirical study.
As shown in Figure (b), given input text T, we replace image encoder Ej with text encoder Er to extract CLIP text feature fr. then fine-tune M with CLIP consistency loss between input and m images R; 1 rendered with random camera poses from output shape S; see Equation 3
(R) [R)
Lc = (fr
where (-) indicates the inner-product.
In stage-2 alignment, we still adopt Lhg to enhance the model's foreground awareness. Comparing Figures (a) and (b), can that the stage-2 alignment is able find a rough shape with Log in around hive iterations, yet failing to produce reasonable output without Log. since having the same color prediction on both foreground and background hinders the object awareness of the model.
Thanks to the joint text-image embedding of CLIP, the gap between text feature fI and shape feature fs has already been largely narrowed by M. Therefore, the stage-2 alignment only needs fine-tur M with 20 iterations using the input text, taking only around 85 seconds on single GeForce RTX 3090 compared with 72 minutes taken by Dream Fields (Jain et al., (022) test time. After this fine-tuning, we can readily obtain plausible result; see, e.g., the "result" shown in Figure 4 (b). Our ISS is novel and efficient approach for 3D shape generation from text.
Diversified generation. In general, shape generation from text one-to-many. Hence, we further extend our approach with diversified 3D shape generation from the same piece of input text. Unlike the existing works, which require additional and complex modules, e.g., GANs (Chen et al. 20188) IMLE (Liu al. 2022), and normalizing flow network (Sanghi et al.) 2022), the image and text features for diversified generation. Specifically. after stage-1 alignment, we initialization and fi the ground truth by adding normalized Gaussian noises 2 h ||, 22 = h2/l[/22] where h1, h2 ~ N(0, 1) to derive diversified features
simply perturb
randomly permute
i = (1 T))Z and f =
T2)
where 71, 72 are hyperparameters to control the degrees of permutation. With permuted fi and fr in stage-2 alignment, our model can converge to different 3D shapes for different noise.
3.4 TEXT-GUIDED STYLIZATION
The two-stage feature-space alignment is already able to generate plausible 3D shapes; see, e.g., Figures (b) and (b). (b). However, the generative space limited by the representation capability of
the employed SVR model, e.g., DVR (Niemeyer et al. (020) can only generate shapes with limited synthetic patterns as those ShapeNet. However, richer and wider range of structures and textures are highly desired. To this end, we equip our model with text-guided stylization module to enhance the generated shapes with novel structure and texture appearances, as shown in Figures (c) and
Specifically, for texture stylization, we first duplicate D (except for the output layer) to be D. and D., then put the output occupancy prediction layer and color prediction layer on top of D, and D... respectively. Further, fine-tune D. with the same CLIP consistency loss as in Equation encouraging the consistency between input text and the m rendered images {
Besides textures, novel structures also desirable for shape stylization. Hence, further incorpo- rate a caape-and-texturr stylization strategy to create novel structures. To enable shape sculpting, we fine-tune D with the same CL.P consistency loss in Equation 3 At the same time, to maintain the

---

Published as a conference paper at ICLR 2023
overall structure of the initial shape S, we propose 3D prior loss Lp shown in Equation 5 at preserving the 3D shape prior learned by the swo-stage feature-space alignment.
aiming
Lp Σ P
- -
(5)
where p is the query point, and D.. D' are the occupancy predictions of the initial D and the fine- tuned D in the stylization process, respectively. To improve the consistency between the generated texture and the generated shape, we augment the background color of R& with a random RGB value in each iteration. Please find more details in the supplementary material.
3.5 COMPATIBILITY WITH DIFFERENT SVR MODELS
Besides DVR (Niemeyer al. 2020), ISS framework is compatible with different SVR models. For example, we can adapt it with the most recent SVR approach SS3D (Alwala et al.) 2022) that leverages in-the-wild single images for 3D generation. With this model, our framework can generate a wider range of shape categories by using SS3D's ' encoder and decoder as shape encoder Es and decoder D in our framework, respectively. Here, we simply follow the same pipeline as in Figure 2 to derive a text- exxt-guide shape generation model for the in-the-wild categories; see our results in Section 4.4) Notably, we follow the losses Anlaaaa al. (2022) in place of LD (see Section in stage-1 training, requiring only single-view images without camera poses. More importantly, our approach's is high compatibility suggests that is orthogonal SVR, so its performance can potentially be further upgraded with more advanced SVR approaches in the future.
4 EXPERIMENTS
4.1 DATASETS, IMPLEMENTATION DETAILS,, AND METRICS
With mueti-view RGB or RGBD images and camera poses. we can train ISS on the synthetic dataset ShapeNet (Chang et al. 2015) (13 categories) and the weal-world dataset CO3D (Reizenstein et al.) 202T (50 categories). To evaluate our generative performance, we create a text description set with four texts per category on ShapeNet and two texts per category on CO3D. SS3D (Alwala et al.) 2022) takes single-view in-the-wild images in training; as their data has not been released, we only evaluate method some of their categories. To evaluate the performance, we employ Fréchet Inception Distance (FID) (Heusel et al. 2017). Fréchet Point Distance (FPD) (Liu et al.) (022) to measure shape generation quality, and conduct human perceptual evaluation further assess text-shape consistency. Please refer to the supplementary material for more details on the metrics and implementation details.
4.2 COMPARISONS WITH OOETE- METHODS
We compare our approach with existing works (anghi et al. 2022 Jain et al. 2022) both qualitatively and quantitatively. For a fair comparison, we use their official codes on GitHub to generate shapes on our text set. Table T shows quantitative comparisons, whereas Figure 5 shows the qualitative comparisons. Comparing existing works and ours in Table [ we can see that our approach out- performs two state-of-the-ar works by large margin for both generative quality and text-shape consistency scores. On the other hand, the qualitative comparisons in Figure 5 (a.b) show that CLip-Forge (Sanghi et al., (2022) produces low-resolution shapes without texture, and some generated shapes are inconsistent with the input text, e.g., "a wooden boat." Dream Fields (Jain et al.) 2022) cannot generate reasonable shapes from the input text on the top row and its generated shape on the bottom row is also inconsistent with the associated input text. On the contrary, our approach (Figure (i) can generate high-fidelity shapes that better match the input text. Note that we only utilize two-stage feature-space alignment without stylization in producing our results. Please refer to the supplcmentary file for more results and visual comparisons.
4.3 ABLATION STUDIES
To manifest the effectiveness of our approach, we conduct ablation studies on the following baselines (see Table I and Figure generate shapes from 22 (Fi + D), optimize stage-2 alignment without stage-1 (w/o stage 1), conduct stage-1 alignment without stage-2 (w/o stage 2), disable the background loss in stage 1, stage 2 and both (w/o Log_ w/o Ln w/o Lgg), and two additional baselines that first create images and then 3D shapes (GLIDE+DVR, LAFITE+DVR). More details on the setup and analysis are provided in the supplementary material Section 3. Our results outperform all the existing works and baselines in terms of fidelity and text-shape consistency by large margin.
7

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative comparisons with existing works and baselines.
Method Type
Method
FID (
Consistency Score (%t (1)
FPD (L)
A/BIC Test
CLIP-Forge
162.87
41.83 17.62
37.43
8.90 + 4.12
Existing works
Dream Fields
181.25
25.38 12.33
N.A.
N.A.
E
18188
20.97 +: 13.59
38,61
N.A.
w/o Stage
222.96
1.92 2.22
79.41
N.A.
Stage
202.33
29.52 14.86
41.71
N.A.
Ablation Studies
w/o L
149.45
29.45 14.67
40.85
N.A.
w/o Liy
156.52
31.55 8.87
38.31
N.A.
w/o Log
178.34
30.96 + 15.49
40.98
N.A.
GLIDE+DVR
212.41
8.85 7.94
41.33
N.A.
Text2Image+SVV
LARITEDDVR
135.01
52.12 11.05
37.55
11.70 + 4.11
Ours
124.42 5.11
60.0
35.67 1.09
21.70
SUV.
beet
Sofa
(0) Foage
Seory
sage-1 vi stage
wib Auct
(c) wt Auce
GLIDEHOVE -DVR
Figure 5: Qualitative comparisons with existing works and baselines.
mattl besch
recliner
(a)
(d)
(a)
(b)
(c)
(d)
Figure 6: Our approach is able generate novel shapes, not in the training set. (a) shows our results and (b,c,d) are the top-three shapes retrieved from the training set.
cheir:
Figure 7: Our approach can generate diversified results from the same input text.
sassage
holtig
benebell
Figure 8: Results on CO3D dataset. We show two different views of each result.
2000
Wereke
Avecato
LOE
Telp
Figure 9: Text-guided stylization. Left: texture stylization. Right: shape and and-texture stylization.
4.4 MORE ANALYSIS GENERATIVE RESULTS OF ISS
Next, we present evaluations the generative novelty and diversity, as well as the scalability of two-stage feature-space alignment. Then, we show more text-guided stylization results and how our ISS approach generalizes to wide range of categories and generates shapes with better fidelity.
Generation novelty. Our approach is able to generate novel shapes beyond simple retrieval from the training data. As shown in Figure 6 from the input text, we first generate our result in (a) and then take generated shape retrieve the top-three closest shapes in the associated training set
8

---

Published as a conference paper at ICLR 2023
Yellow hardyyer with black handle.
Red
groen
pormot
(8) SS3D with image input
(b) Ours with text imput
Figure 10: With single images for training (without camera poses), our approach can produce results
for broad range of categories, by adopting (Alwala et al. 2022). Two different views are rendered.
Long lible las four thin and connections lags sithe.
Square trant
shape
mobasyyle
blue
rectanguiar
table
sotvel
thair
urique
Figure 11: Results of ISS built upon the SVR model of IM-Net (left two) and GET3D (right four).
based on the cosine similarity between the CLIP features of the rendered images fi and input text fr as the retrieval metric. Our results with the two-stage feature-space alignment are not in the training sets, showing that our ISS approach
generate novel shapes beyond the training set, stylization. is not surprising, as our approach shares the generative space with the pre-trained SVR model and can potentially generate all shapes that the pre-trained SVR model can produce.
without
Generation diversity. By perturbing the features and injecting randomness on initialization, ISS able to generate diversified results from the same text input. As shown in Figure 7 ISS produces various cars and chairs from the same piece of text. Quantitative results presented in the supplementary material Section 3.4 where our method achieves high PS (Point Score).
Generation fidelity. To assess the capability of ISS generating realistic real-world 3D shapes, we train the SVR model the CO3D dataset (Reizenstein al., 2021) which is real-world dataset, and ISS leverages the learned feature space for text to shape generation without paired data. As shown in Figure 8, our model is able generate real-world shapes. As far know, this is first work that investigates text-guided shape generation and on real-world datasets can generate realistic 3D shapes.
Generation beyond the capability of SVR model. Our text-guided stylization module equips our model with the capability generate 3D shapes beyond the SVR model. As shown in Figure 2 and Figure 2 (c), our model is able to create realistic and fantasy novel structures and textures that match text descriptions. Please refer the supplementary material Section for more details.
Generality and Scalability of ISS other SVR models. Our model is generic and can work logether with other SVR models. To evaluate the generality and scalablity of our model, we employ SS3D (lwala et 2022), , IM-Net (Chen Zhang 2019). and GET3D (Gao al., 2022) as SVR models to provide the feature space. It is worth noting that SS3D is capable of generating shapes of more categories and IM-Net, GET3D can produce high fidelity results. First, as shown in Figure [T] built upon SS3D, our approach can generate shapes of more real-world categories, e.g., bird. Note that our model can generate shapes with comparable or even better qualities compared with initial SS3D model that takes image input. Second, when combined with IM-Net and GET3D, our model can fully exploit their generative capabilities and produces high-quality 3D shapes as shown in Figure 11 The above manifests that ISS is generic and compatible to advanced models for generating shapes of more categories and higher qualities.
5 CONCLUSION
In this paper, we present novel approach for text-guided 3D shape generation by leveraging the image modality as stepping stone. Leveraging the joint text-image embeddings from CLIP and 3D shape priors from pre-trained SVR model, our approach climinates the need for the paired text and shape data, Technically, we have the following contributions. First, we step-by-step reduce the semantic gap among the text, image and shape modalities through our two-stage feature-space alignment approach. Second, our text-guided stylization technique effectively enriches our generated shapes with novel structures and textures in various styles. Third, our approach is compatible with various single-view reconstruction approaches and can be further extended to generate a wide range ShapeNet, of categories with only single images without camera poses in training. Experiments CO3D, and multiple single-image categories manifest the superiority of our framework over the two fate-of-the-ar methods and various baselines. Limitations are discussed in the supplementary files.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The work has been supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14206320), General Research Fund of Hong Kong (No. 17202422), Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), and National Natural Science Foundation of China (No. 62202151).
REFERENCES
Nitin Agarwal and M Gopi. Gamesh: Guided and augmented meshing for deep point networks. In 3DV, 2020.
Kalyan Vasudev Alwala, Abhinav Gupta, and Shubham Tulsiani. Pre-train, self-train, distill: A simple recipe for supersizing 3D reconstruction. CVPR, 2022.
Angel X. Chang, Thomas Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rie 3D Model Repository. Technical Report arXiv:15 [cs.GR], 2015.
Kevin Chen, Christopher Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings, In ACCV, 2018.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurlPS, 2021.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. GET3D: A generative model of high quality 3D textured shapes learned from images. NeurlPS, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge a local Nash equilibrium. NIPS, 2017.
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM TOG (SIGGRAPH), 2022.
Tansin Jahan, Yanran Guan, and Oliver van Kaick. Semantics-guider latent space exploration for shape generation. In COMPUT GRAPH FORUM, 2021.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with Drefam Fields. In CVPR, 2022.
Nikolay Jetchev. ClipMatrix: Text-controlled creation of 3D textured meshes. arXiv preprint arXiv:2109. /2922, 2021.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr Controllable text-to-image generation. NeurIPS, 2019.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr. ManiGAN: Text-guided image manipulation. In CVPR, 2020.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. arXiv preprint arXiv:2203.02055: 2022.
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. FuseDream: Training-free frre text-to- text-to-image generation with improved CLIP+ GAN space optimization. arXiv preprint arXiv:21/2.01573 2021.
10

---

Published as a conference paper at ICLR 2023
Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3D shape generation. In CVPR, 2022.
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In CVPR, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lyy Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP:
Text-driven manipulation of StyleGAN imagery. ICCV, 2021.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. MirrorGAN: Learning text-to-image generation by redescription. In CVPR, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint 0615,,, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, 2016a.
Scott E. Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where to draw. NIPS, 2016b.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects 3D: Large-scale learning and evaluation of real-life 3D category reconstruction. In ICCV, 2021.
Robin Rombach, Patrick Esser, and Bjöm Ommer. Network-to- translation with conditional invertible neural networks. NewrIPS, 2020.
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Y Yi Cheng, and Marco Fumero. CLIP-Forge: Towards zero-shot text-to-shape generation. In CVPR, 2022.
Douglas M. Souza, Jônatas Wehrmann, and Duncan D. Ruiz. Efficient neural architecture for text-to-image synthesis. In IJCNN, 2020.
David Stap, Maurits Bleeker, Sarah Ibrahimi, and Maartje ter Hoeve. Conditional image generation and manipulation for user-specified content. CVPRW, 2020.
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. CLIP-NeRF: Text-and- image driven manipulation of neural radiance fields. In CVPR, 2022a.
Hao Wang, Guosheng Lin, Steven Hoi, and Chunyan Miao. Cycle-consistent inverse GAN for text-to-image synthesis. ACM MM, 2021.
Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. CLIP-GEN: Language-free training of text-to-image generator with CLIP. In arXiv preprint arXiv:2203.00386, 2022b.
Zixu Wang. Zhe Quan, Zhi-Jie Wang, Xinjian Hu, and Yangyang Chen. Text to image synthesis with bidirectional generative adversarial network. ICME, 2020.
Weihao Xia, Yujuu Yang, Jing-Har Hao Xue, and Baoyuan Wu. TediGAN: Textee diverse face image generation and manipulation. In CVPR, 2021.

---

Published as a conference paper at ICLR 2023
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR, 2018.
Mingkuan Yuan and Yuxin Peng. Bridge-GAN: Interpretable representation learning for text-to-image synthesis. IEEE TCSVT, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. StackGAN++: Realistic image synthesis with stacked generative adversarial networks. IEEE TPAMI, 2018.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. LAFITE: Towards language-free training for text-to-image generation. In CVPR, 2022.
Nikola Zubić and Pietro Liò. effective loss function for generating 3D models from single image without rendering. arXiv preprint
2021.
12


Reviews:
[{"review_id": "bo_Wz6k2Szk", "reviewer": "Reviewer_YJ7a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a text-to-3D shape generation method which does not required paired text-shape training data. This work exploits the pretrained CLIP feature spaces for text and images and a pre-trained single-view reconstruction (SVR) model. The text feature space is aligned to an intermediate image feature space and then the target shape feature space. Various losses are designed to ensure the alignments. A further shape stylization module is introduced to generate new structures and textures for the generated shape. Experiments are conducted with different SVR models on ShapeNet and CO3D datasets. Experimental results show the proposed method can achieve higher performance than existing methods in terms of FID and FPD.", "summary_of_the_review": "The idea and implementation put this paper above the borderline. The unclear setting of the model and unsatisfied experimental comparison degrade the quality of this paper. Overall, I vote for a positive score.", "strength_and_weaknesses": "Pros\n\n-This paper is well organized and easy to follow.\n\n-The proposed method is simple and effective. The alignment of the text feature space to the shape feature space through the intermediate image feature space avoids the requirement of paired text-shape training data. The introduced losses, especially background loss in Eq. (3) and training tricks, like diversified generation and augmentation make this model work.\n\n-The experiments cover the comparison with existing methods and ablation studies. The experimental results show better performance than existing methods with numerical results and visual examples.\n\nCons\n-Figure 2 is unclear. The authors divided the figure into three subfigures, which corresponds to different stages of training. However, from the figure, it is unclear which part of the model is trained, what loss is applied and how the training is performed.\n\n-In Section 3.2.1, the authors use the mIoU metric to measure the 3D reconstruction results and observe a degradation by replacing the SVR encoder with the CLIP encoder. However, it is unclear how the added mapper can help to solve the problem.\n\n-In Section 3.3, the authors claimed ' we fine-tune decoder D ...'. It is unclear the mapper M and decoder D are trained together or separated at the first stage.\n\n-It is unclear how the camera poses are set and selected during training, both the first stage and second stage.\n\n-It is unclear how the decoder D is duplicated to D_o and D_c in Section 3.4 and Figure 2.\n\n-The selected metric FID may be unfair to the compared method, especially CLIP-Forge as it only supports the color generation for shapes. This can be found in Figure 5. A feasible solution is to add the metric of Maximum measure distance (MMD) as in CLIP-Forge along with FID to compare different methods. Furthermore, the pretrained model used in FID is not clarified.\n\n-There is no Figure 8(b), which is stated in 'Generation beyond the capability of SVR model'.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fecp5U2ZpmQ", "reviewer": "Reviewer_HSBz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper works on text2shape generation. It proposes to use images as a 'stepping stone' of the generation process. The authors use two mapping processes to map image-shape latent space and image-text latent space sequentially. Furthermore, the authors show the model can stylize both texture and shape structure. The experiment shows the method outperforms two zero-shot text2shape generation baselines. ", "summary_of_the_review": "The paper proposes an interesting solution for the text2shape generation problem. I like the method itself. However, the author could better explain the background, notation, and equations better. I'm willing to improve the score if the author could clarify my concerns. ", "strength_and_weaknesses": "Strength:\n\n+ The paper proposes a solution to using images as a stepping method for text2shape generation. \n\n+ As a zero-shot generation method, it can generate shapes of diverse categories relatively quickly (85 seconds).\n\n+ Using latent space makes the generation and stylization step very flexible. \n\nWeakness:\n\n- The paper focuses on 'latent space mapping'. However, it didn't provide a systematic evaluation/visualization of how the latent space is mapped. Figure 3 (c) provides only one instance which is not convincing enough for me. \n\n\n- The paper is hard to follow for me because of the writing. For example: (1) In equation 3, the symbol p jumped out without explanation. To make the paper self-contained, it's better to provide an explanation for all the symbols in the equations. (2) The paper lacks enough background knowledge about the SVR model, like the rendering method and reconstruction losses. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FCGQdXDV0to", "reviewer": "Reviewer_Z3WY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of text-guided 3D shape generation. This is a challenging task due to the absence of large paired text-shape datasets. This paper proposes a method called Image as Stepping Stone (ISS) that introduces 2D images as stepping stones to connect the two modalities, which in turn avoids the need for paired data. Experimental results show some improvement over existing methods.", "summary_of_the_review": "Please see comments in the two boxes above.", "strength_and_weaknesses": "Strengths:\n1. The problem is very interesting.\n2. The idea of using images to connect two modalities is good.\n3. Results are good compared to existing methods\n\nWeaknesses:\n1. The proposed framework shown in Figure 2 is too complicated. This means that there are many learnable parameters in the framework. When reporting results with comparisons to other methods, model complexity should also be taken into consideration. For example, when looking at Table 1, it is unclear whether the performance gain over the competing methods is from designing the \"right\" algorithm or is it simply from using more learnable parameters.\n2. In Table 1, what does Ours mean in the category column?\n3. The paper claims that the approach can stylize shapes with realistic structures and textures. However, the green SUV example in Figure 5 and the results in Figure 9 don't look realistic.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4VGLRfvtqG", "reviewer": "Reviewer_CfJ1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method for text-shape transfer, namely ISS(Image as Stepping stone), based on powerful CLIP, without the need of paired text-shape dataset. The ISS consists of 2 stages:\n1. stage 1, the method trains a CLIP2Shape mapper to map the CLIP image features to a pre-trained detail-rich 3D shape space. \n2. stage2, due to further finetune the mapper for better consistency between the text and generated shape.\n3. To further enrich the generated shape according the input text, the method proposes a style transfer module to transfer the texture to the shape.\n", "summary_of_the_review": "This article proposes a method ISS based on CLIP that can effectively implement text-shape conversion, and provides a large number of credible experimental results as an illustration. ", "strength_and_weaknesses": "Strengths:\n1.\tThe method does not require a large amount of data in pairs as a reference\n2.\tDue to powerful CLIP as the base, the method is able to obtain a convincing result with less time cost comparing to other existing work.\n3.\tFurther, the method provides a style transfer module to enrich the result.\n\nWeaknesses:\n1.\tThe final step is based on SVR method, so how to ensure that the shapes obtained have good details? As shown in the experiment part, the image does not have a lot of fine details.\n2.\tAlthough using more and richer texts to obtain the final model, it seems that the method cannot generate results that not in the training domain.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%