Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduced a novel task, namely recursive text-conditioned shape generation. In this setting, the model needs to generate a 3D shape given the initial text, and subsequently evolves it in multiple steps given more text descriptions.

To achieve this goal, the author built a new dataset (Text2Shape++) which contained many-to-many mappings of phrase sequences and shapes, derived from Text2Shape dataset [5] which only contained one-to-one mappings.

Together with the dataset, the paper also presented a shape generation model that supports generating and refining a shape in multiple iterations. The model is based upon AutoSDF [30]. In the first stage, a VQ-VAE-based shape generation model is trained, and a transformer-based generator is trained on top of the VQ-VAE latent, which can be used to refine a given latent. In the second stage, a recursive feature refinement model based on BERT and 3DCNN is trained directly with text and VQ-VAE latent.

strengths_and_weaknesses: ### Strengths
* The proposed recursive generation task is both novel and useful, which can inspire future research.
* The curated dataset is a valuable and unique addition to the current text-to-shape datasets.
* The new model achieved better generation quality on the proposed dataset compared to previous work, justifying the need of a model tailored for the new task.

### Weaknesses
* The proposed model is trained with strong supervision enabled by the new dataset, as the exact matching between arbitrary phrase sequences and shape sets is available. Building such a dataset is not always feasible for real-world applications with more shape classes and more complex shapes. 

questions: * The math notations in Section 4.2 is confusing. I wonder what is the $j$ in L178 representing?
* L197 mentioned that the $Z_t$ generated by the recursive language model is send through a transformer before feeding to the 3D decoder. I wonder how important this step is? And how is this implemented? Do you sample from $Z_t$ before feeding the sample into the transformer?
limitations: Limitations and societal impacts are adequately addressed in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a neural network for recursive text-conditioned 3D shape generation --- progressively updating the generated shape by a series of input phrases. To achieve this, the proposed method is built on top of vector-quantized deep implicit functions (as the shape representation) such that a set of possible shapes can be described as a distribution over 3D grids, which can then be learned using an autoregressive model. Another contribution of this paper is a enhanced dataset, Text2Shape++, that provides many-to-many shape-text pairs. The proposed method largely relies on such dataset.

The results demonstrated that the method can generate shapes from text input, and the shapes evolve gradually as more short phrases are added. The authors also include a realtime demo where the user describes a shape by speech and the generation results are shown in the screen.

strengths_and_weaknesses: Strengths:
- It is an interesting idea to represent a set of possible shapes as a probability distribution over latent feature codes. With an autoregressive generator, this design naturally supports probabilistic generation (i.e., capable of producing different shapes from the same input). This is a different and novel approach compared to previous text2shape methods.
- The Text2Shape++ dataset, even though constructed for a particular scenario, is a good contribution to the community. I like the way that the authors construct the dataset, i.e., use principled natural language techniques such as constituency parser and phrase similarity.
- For an application paper, a real-world demo is a bonus. I appreciate the good effort by the authors in making such demo.

Weaknesses:
- My main concern is about the result quality and evaluation. 
    - Poor visual quality. Many of the generated shapes (i.e., those in Fig.1, Fig.3, Fig.4) have noticable artifacts, e.g., holes, disconnected or floating parts. Given the visual quality of the demonstrated examples, I would doubt the overall result quality.
    - Missing comparison to previous SOTA text2shape methods, especially [1]. The compared method, AutoSDF is not especially designed for text-driven shape generation. Another fact that I noticed is that this work ignores the shape color (so does autoSDF), which is an important information for many text descriptions and is considered in previous text2shape literatures [1][2]. I'd like to hear the reason that the surface color is not considered. Clearly the dataset itself contains colored shapes.
    - Missing support for the made claim. Specifically, line 187-188, "As more phrases are added, the probabilities become narrower resulting in more deterministic shapes.". I think this should be a key feature of the method, but somehow it is not validated in the experiments.
- Another minor concern is the application scope or the motivation for recursive text-conditioned generation. From my perspective, the advantage of text-driven 3D shape generation is to quickly produce a base/rough shape. People (modeling experts or ordinary users) then start from that shape for further detailed editing, from where precise control is more important. However, from the presented results, I'm not convinced that recursive text input can provide such necessary precise control (as text itself is vague), compared to traditional 3D shape editing techniques.

Given that my concerns outweigh the positive contribution of the paper, I'm inclined to rejection at this point.

[1] Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. CVPR 2022.
[2] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. ACCV 2018.

__Update:__

After reading the rebuttal and other reviews, I feel more positive about this paper as some of my major concerns are well addressed (missing comparison, lacking support for the claim). Thus I raise my rating from 4 to 5.
questions: Questions and suggestions:
- Line 202-205, the sorting is a bit counter-intuitive to me. Wouldn't this sort operation make the order inconsistent among the dataset? My first thought is that such inconsistent order would hurt the training. I suggest elaborating more on the random transformer, which seems to be a key choice here.
- Line 237-240, why is it necessary to normalize clip-similarity score? Cosine similarity by itself already has limited value range.
- For all loss functions described in Sec. 4.4, I suggest including their mathematical formulations, at least in the supplementary document, to make the paper self-contained.

Typos:
- Line 139, "since since" -> "since".
- Line 148, "... is the largest such dataset" -> "... is the largest dataset of its kind".
- Line 177, $X_j$ -> $X^j$.

limitations: Limitations are well discussed and several failure cases are presented in the supplementary document. Negative social impact is also addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper discusses a method to sequentially generate a chair from a sequence of utterances provided by the user. To do so the paper first cleans the Text2Shape++ dataset by applying linguistic parsing. Their model consists extracting text features from text, that are then projected to a volumetric grid and concatenated to the previously generated volumetric features. Then this tensor is used to generate an SDF to represent the shape. The paper shows examples of a couple of sequential edits.

I wonder why authors used Recursive in the title, since it has a connotation that it uses a recurrent NN, or any other recurrent process. Here instead it's sequential, with not too many steps presented in the paper.
strengths_and_weaknesses: Strength:

S1. The paper definitely tackles an interesting project, as far as I'm aware, this is the first of one of the first papers in this domain.
S2. Their architecture is somewhat new, somewhat extended from ShapeFormer.
S3. Written very well, thank you, very easy to read and follow.

Weaknesses:

W1. **Results**. The paper report standard metrics that evaluate how well the shape corresponds to the text. The paper, however, does something different, they propose a *sequential* generation paper. While ShapeGlot-Confidence or Clip-Score are necessary, they're by no means sufficient. How can we be sure that sequential generation happened. If you say "a chair with straight legs" and then add "curved thin legs" and then add "with a hole in the seat", we'd like to be sure that only the these attributes change, and only when asked. The paper doesn't provide a metric or evaluation protocol for that. A further interesting metric to consider is to evaluate that other parts of the chair stay the same or marginally change while attributes are added. The paper doesn't provide that too. Such metrics probably don't exist, but I believe the responsibility of coming up with a reasonable evaluation protocol bears with the authors and if done well can be listed as a contribution.

These metric are very important for such a method, because, if we look at Fig 3, we see that changes to not mentioned parts of the chair are significant. For example in the bottom row "a black object" gives you a chair with four legs, "...with four legs" gives another chair with four legs, "...and cushioning" gives another chair with four legs and slightly changed geometry. I think any of these three chairs perfectly satisfies a description "a black object with four legs and cushioning" I cannot tell whether the final variant is the best. Also it's not clear why "...with cushioning" makes the legs curvy. Inconsistency of sequential updates is also the case in examples in Fig.4.

W2. **Adding vs Removing attributes**. Currently the utterances are additive, ie we add attributes to the chair, they might be conflicting such as "stand legs" and "wheels". But I wonder if it's possible to say mutually exclusive things, such as "a char with four legs", then "a chair with one metal leg" and then "a chair with four legs". OR "a chair with armrests" and then "a chair without armrests". I wonder if there is a metric to compute, when you do subtract attributes you should return to the exactly previous chair, perhaps it's even a decent loss function...

W3. **Generality**. In its current form the paper operates only on chairs, for whatever reason they add attributes of materials to the text, while not being able to render "black, metal, wooded" and so on. Have you thought about other categories? The paper title is quite generic, while implementation is category specific. It would be good to have one model to generate multiple object categories. 
questions: I'd like the authors to comment on the questions listed in the weaknesses section. I believe the paper is a good piece of work, that might be not quite ready for a publication
limitations: The limitation section is satisfactory
review_title: null

Meta-review: metareview: This paper presents a recursive text to shape generation system, introduce a new dataset for text-to-shape, and presents good performance of the proposed method. This paper has the potential of inspiring future work. 
I encourage the authors to add a discussion (e.g., at limitations or future work) of the necessity of proper metric for evaluation of shape generation. Consolidating this problem can propel the field forward. 

confidence: Less certain
Decision: Accept