Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper works on text2shape generation. It proposes to use images as a 'stepping stone' of the generation process. The authors use two mapping processes to map image-shape latent space and image-text latent space sequentially. Furthermore, the authors show the model can stylize both texture and shape structure. The experiment shows the method outperforms two zero-shot text2shape generation baselines. 
strength_and_weaknesses: Strength:

+ The paper proposes a solution to using images as a stepping method for text2shape generation. 

+ As a zero-shot generation method, it can generate shapes of diverse categories relatively quickly (85 seconds).

+ Using latent space makes the generation and stylization step very flexible. 

Weakness:

- The paper focuses on 'latent space mapping'. However, it didn't provide a systematic evaluation/visualization of how the latent space is mapped. Figure 3 (c) provides only one instance which is not convincing enough for me. 


- The paper is hard to follow for me because of the writing. For example: (1) In equation 3, the symbol p jumped out without explanation. To make the paper self-contained, it's better to provide an explanation for all the symbols in the equations. (2) The paper lacks enough background knowledge about the SVR model, like the rendering method and reconstruction losses. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 
+ Is the image encoder Es fixed or optimized during step 1? In Figure 2, it is better to provide the "lock/unlock" icon to Es, Ei, and ET as well, to make the notations consistent. 

+ Is the vector direction in Figure 3 (c) reasonable? In step 2, the 𝑀(𝑓𝑇) is optimized to 𝑀(𝑓𝐼), but the vector direction is moving to some other point in the latent space.

+ Analyse and Evaluation: How many shapes are generated from the text prompt at the evaluation step? I'm also concerned with the number of instances used for evaluation. If only four texts per ShapeNet category are used, there will be a total of 52 shapes, which is a very limited number of instances to calculate FPD and FID. I suggest the author run the metric multiple times and provide a mean and a derivation.  

+ In the paper, the author mentioned referring to Figure 2 (c)  & Figure 8 (b) for some results, but there is no  Figure 2 (c) or Figure 8 (b) in the paper. Is that false referred? 

+ Ablations of background loss. Is the background loss removed at stage 1, stage 2, or both?

Quality
+ Fair. But the paper is hard to follow for me because of lacking background, unexplained symbols and notations, and false references. 

Novelty. 
+ Good. 

Reproducibility.
+ Good. 
summary_of_the_review: The paper proposes an interesting solution for the text2shape generation problem. I like the method itself. However, the author could better explain the background, notation, and equations better. I'm willing to improve the score if the author could clarify my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of text-guided 3D shape generation. This is a challenging task due to the absence of large paired text-shape datasets. This paper proposes a method called Image as Stepping Stone (ISS) that introduces 2D images as stepping stones to connect the two modalities, which in turn avoids the need for paired data. Experimental results show some improvement over existing methods.
strength_and_weaknesses: Strengths:
1. The problem is very interesting.
2. The idea of using images to connect two modalities is good.
3. Results are good compared to existing methods

Weaknesses:
1. The proposed framework shown in Figure 2 is too complicated. This means that there are many learnable parameters in the framework. When reporting results with comparisons to other methods, model complexity should also be taken into consideration. For example, when looking at Table 1, it is unclear whether the performance gain over the competing methods is from designing the "right" algorithm or is it simply from using more learnable parameters.
2. In Table 1, what does Ours mean in the category column?
3. The paper claims that the approach can stylize shapes with realistic structures and textures. However, the green SUV example in Figure 5 and the results in Figure 9 don't look realistic.
clarity,_quality,_novelty_and_reproducibility: The paper reads well. The quality of the presentation is good. The problem is interesting. However, given that the framework is very complicated, it is unclear whether the paper is reproducible without making the code publicly available to the community.
summary_of_the_review: Please see comments in the two boxes above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method for text-shape transfer, namely ISS(Image as Stepping stone), based on powerful CLIP, without the need of paired text-shape dataset. The ISS consists of 2 stages:
1. stage 1, the method trains a CLIP2Shape mapper to map the CLIP image features to a pre-trained detail-rich 3D shape space. 
2. stage2, due to further finetune the mapper for better consistency between the text and generated shape.
3. To further enrich the generated shape according the input text, the method proposes a style transfer module to transfer the texture to the shape.

strength_and_weaknesses: Strengths:
1.	The method does not require a large amount of data in pairs as a reference
2.	Due to powerful CLIP as the base, the method is able to obtain a convincing result with less time cost comparing to other existing work.
3.	Further, the method provides a style transfer module to enrich the result.

Weaknesses:
1.	The final step is based on SVR method, so how to ensure that the shapes obtained have good details? As shown in the experiment part, the image does not have a lot of fine details.
2.	Although using more and richer texts to obtain the final model, it seems that the method cannot generate results that not in the training domain.

clarity,_quality,_novelty_and_reproducibility: The article describes the structure of its method clearly and provides many experiments results, including ablation and SOTA results, to prove the effectiveness of the method. Although the method depends on the CLIP model to use image space as a step stone to transfer the text feature into shape space, the method provides a novel work to complete Text-shape task.
summary_of_the_review: This article proposes a method ISS based on CLIP that can effectively implement text-shape conversion, and provides a large number of credible experimental results as an illustration.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a text-to-3D shape generation method which does not required paired text-shape training data. This work exploits the pretrained CLIP feature spaces for text and images and a pre-trained single-view reconstruction (SVR) model. The text feature space is aligned to an intermediate image feature space and then the target shape feature space. Various losses are designed to ensure the alignments. A further shape stylization module is introduced to generate new structures and textures for the generated shape. Experiments are conducted with different SVR models on ShapeNet and CO3D datasets. Experimental results show the proposed method can achieve higher performance than existing methods in terms of FID and FPD.
strength_and_weaknesses: Pros

-This paper is well organized and easy to follow.

-The proposed method is simple and effective. The alignment of the text feature space to the shape feature space through the intermediate image feature space avoids the requirement of paired text-shape training data. The introduced losses, especially background loss in Eq. (3) and training tricks, like diversified generation and augmentation make this model work.

-The experiments cover the comparison with existing methods and ablation studies. The experimental results show better performance than existing methods with numerical results and visual examples.

Cons
-Figure 2 is unclear. The authors divided the figure into three subfigures, which corresponds to different stages of training. However, from the figure, it is unclear which part of the model is trained, what loss is applied and how the training is performed.

-In Section 3.2.1, the authors use the mIoU metric to measure the 3D reconstruction results and observe a degradation by replacing the SVR encoder with the CLIP encoder. However, it is unclear how the added mapper can help to solve the problem.

-In Section 3.3, the authors claimed ' we fine-tune decoder D ...'. It is unclear the mapper M and decoder D are trained together or separated at the first stage.

-It is unclear how the camera poses are set and selected during training, both the first stage and second stage.

-It is unclear how the decoder D is duplicated to D_o and D_c in Section 3.4 and Figure 2.

-The selected metric FID may be unfair to the compared method, especially CLIP-Forge as it only supports the color generation for shapes. This can be found in Figure 5. A feasible solution is to add the metric of Maximum measure distance (MMD) as in CLIP-Forge along with FID to compare different methods. Furthermore, the pretrained model used in FID is not clarified.

-There is no Figure 8(b), which is stated in 'Generation beyond the capability of SVR model'.
clarity,_quality,_novelty_and_reproducibility: The quality of the writing is overall good. The quality/originality of the proposed method is also good. Most of the paper is clearly clarified. Some detailed settings are unclear, as suggested in the weaknesses.
summary_of_the_review: The idea and implementation put this paper above the borderline. The unclear setting of the model and unsatisfied experimental comparison degrade the quality of this paper. Overall, I vote for a positive score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
Summary:
The paper presents a framework for generating 3D shapes from text by utilizing: 1) a pretrained single-view image to 3D shape reconstruction (SVR) model 2) pretrained model that gives a joint embedding of image-text (e.g. CLIP), and training a feature alignment network to map the CLIP features to the feature space of the SVR model.  The authors also propose an optional text-guided stylization step that furthur enriches the texture of the shape.  Experiments show that the proposed method can generate reasonable shapes for an given input text.

Strengths:
- The submission provides a flexible framework for utilizing pretrained vision-language language + SVR models to generate 3D shapes without paired text-to-3D data
- The proposed method is fast compared to prior work such as DreamFields that requires per-prompt optimization
- The paper can stimulate additional work in the field of text-to-3D shape generation which is in its nascent stages,

Weaknesses:
- The quality of the generated shapes are very dependent on the SVR used, with the generated shapes in the initial version very smoothed out and lacking fine detail
- The set of prompts used to generate the shapes are fairly simplistic
- The writing can be improved as some reviewers found the parts of the paper difficult to follow initially, and some parts to be unclear


justification_for_why_not_higher_score: Reviewers initially had concerns about some aspects of the work, but increased their ratings to weak accept after the rebuttal.  The method is interesting but is fairly intuitive combination of advances in joint text-image embeddings and image-to-3D.   
justification_for_why_not_lower_score: The proposed method is flexible and will encourage additional work in the area of text-to-shape generation.
Decision: Accept: notable-top-25%