ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model
Authors: Rao Fu
Xiao Zhan
Yiwen Chen
Daniel Ritchie
Srinath Sridhar
Venue: NeurIPS 2022 Accept (2022)
Keywords: language
shape generation
3D representation
text-shape dataset

Abstract: We present ShapeCrafter, a neural network for recursive text-conditioned 3D shape generation. Existing methods to generate text-conditioned 3D shapes consume an entire text prompt to generate a 3D shape in a single step. However, humans tend to describe shapes recursively---we may start with an initial description and progressively add details based on intermediate results. To capture this recursive process, we introduce a method to generate a 3D shape distribution, conditioned on an initial phrase, that gradually evolves as more phrases are added. Since existing datasets are insufficient for training this approach, we present Text2Shape++, a large dataset of 369K shape--text pairs that supports recursive shape generation. To capture local details that are often used to refine shape descriptions, we build on top of vector-quantized deep implicit functions that generate a distribution of high-quality shapes. Results show that our method can generate shapes consistent with text descriptions, and shapes evolve gradually as more phrases are added. Our method supports shape editing, extrapolation, and can enable new applications in human--machine collaboration for creative design.

ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model
Rao Fu
Xiao Zhan
Yiwen Chen
Daniel Ritchie
Srinath Sridhar
Brown University
rao w..ffubbbwmeeduu
ivl. brown www.bbowww....ooooepbb htt..////droooesss//aprratteeee
Abstract
We present ShapeCrafter, neural network for recursive text-conditioned 3D shape generation. Existing methods that generate text-conditioned 3D shapes consume an entire text prompt generate a 3D shape in a single step. How- ever, humans tend to describe shapes recursively-ww may start with an initial description and progressively add details based on intermediate results. To capture this recursive process, we introduce method to generate a 3D shape distribu- tion, conditioned on an initial phrase, that gradually evolves as more phrases are added. Since existing datasets are insufficient for training under this approach, we present Text2Shape++, large dataset of 369K shape-ter text pairs that supports recursive shape generation. To capture local details that are often used to refine shape descriptions, we build upon vector-quantized deep implicit functions that generate distribution of high-quality shapes. Results show that our method can generate shapes consistent with text descriptions, and shapes evolve gradually as more phrases are added. Our method supports shape editing, extrapolation, and can enable new applications in human-machine collaboration for creative design.
Introduction
Humans are unique in the animal kingdom in the use of language to describe objects, scenes and events [10]. We use language not only describe an object's physical and functional attributes,
but also to qualify or modify those attribute descriptions. More specifically, we use (linguistic) recursion g to better describe it or modify it. We use this ability extensively, for instance when describing an object the phone. The capability understand recursive natural language descriptions of objects is crucial for the wide accessibility of intelligent robots, computational creativity tools, and online shopping systems.
may start with initial description of object, and progressively add phrases
Driven by recent advances language modeling 12 and generative image modeling 22
there has been rapid progress in the problem of conditionn image generation [55]
44
57
58
59
For instance, methods such as DALL-E [39, (38 can generate photorealistic images from only text descriptions. This success in the 2D domain together with progress in 3D shape representations 32
29 31 methods have several shortcomings. First, these methods limited one-step shape generation and B3] Second, unlike in 2D images, there are no readily available large datasets of 3D shape-text pairs. Existing datasets are limited in size due to challenges in manually annotating 3D data and do not support recursive generation 66, 2 Finally, current methods cannot capture local details that commonly used in language descriptions.
has led to interest text-conditioned 3D shape generation [50 40 20) However, existing
lack the ability to recursively modify or improve generated shapes
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

"with no Dimeest A
Thas a long backzest" A
"legs are curved, A
haas sepazate legs A
"with amres's ts that can hold" A
"it has wheels on legs." AX
"A chair" Art wetensonk
"the back is slanted' XxI
'it slats" A
squaze
"with back" A
"the A
axwzests.
A
Figure 1: ShapeCrafter is a method for recursive text-conditioned 3D shape generation. Given an initial phrase (A chair), it generates shape distribution (4 samples shown in the leftmost column). As more phrases are added, the initial shape is refined (2 samples shown). We can handle long phrase sequences while continuously evolving an initial shape (shape editing for fixed random seed). Our method also shows extrapolation capabilities (legs are armrests, bottom right).
In this paper, we address the above limitations and present a method for recursive text-conditioned generation of 3D shapes. Our method, ShapeCrafter, consists of rranfformrrbbaee [49 neural network architecture that takes an initial text phrase (via BERT (12) and generates a distribution of latent features over 3D grid that can be sampled to obtain shapes consistent with the input. Our goal is to capture recursive phrase sequences common in language descriptions (see Figure S as more informative words/phrases are added, our method produces more relevant shape distributions. To achieve this, we design a recursive network architecture that takes the latest phrase input in a sequence and incorporates latent grid features from the previous step generate refined shapes. The shape distributions produced by our method evolve gradually as more phrases are added, thus enabling new applications in human numan-machine collaboration for creative design.
To address the limited size of existing datasets, we introduce Text2Shape++, a dataset that contains 369K many-to-many mappings of varied-length phrase sequences that map the same shape. This dataset is derived from Text2Shape (6] that contains 75K one-to-one text-shape pairs without recursive phrase sequences. We introduce method to transform (see Section 4.1 the Tex12Shape dataset into varied-length phrases resulting in a significantly larger dataset. Finally, to learn high-quality shapes, we adopt vector-quantized deep implicit functions (P-VQ-VAE) [56] 31 better represent local details and generate shape distributions rather than single shapes.
Experiments and results produced by ShapeCrafter demonstrate that can generate and evolve high-quality 3D shape distributions that are consistent with text inputs. As additional phrases are added to the sequence, the shapes gradually and continuously evolve while retaining shape details generated in the previous steps thus enabling shape editing functionality (see Figure T). Our model can handle long phrase sequences that other methods cannot. Surprisingly, the capability to extrapolate to novel inputs (e.g., chair legs that look like armrests, see Figure [ bottom right) unseen during training. Quantitative results show that shape reconstruction quality is comparable to the state of the art, while our recursive shape generation and editing capabilities are state of the art. To up, our contributions are:
model demonstrates
ShapeCrafter, a neural network architecture that enables recursive text-conditioned genera- tion of 3D shapes that continuously evolve as phrases are added. Text2Shape++, new large dataset of 369K shape-text pairs that can be used for recursive shape generation tasks.
model that captures local shape details corresponding to text descriptions and enables shape editing and extrapolation to novel text descriptions.

---

2 Related Work
In this brief review, we limit our discussion to work on 3D shape representations and 3D shape generation and manipulation. Text-conditioned 2D image generation 224 39 38 is outside of the scope of this review, please see 14 for details.
3D Shape Representation and Generation: Existing work in 3D shape generation can be categorized
based their underlying representation of choice. Major choices include meshes [6 54 15
I1I voxels [9 26 6 point clouds [ 25
34
52
35 41. neural fields/scene representation
29 Meshes and point clouds are useful for simple shapes but struggle with complex details and topology resulting in lower quality results. Voxel grids take up large amount of memory to preserve details. Implicit fields including SDFs more popular because they can capture fine-grained details, handle arbitrary topologies, and produce watertight meshes. Since many neural implicits encode the whole shape into single latent code, they often suffer from "posterior collapse", where the latent space is not fully used, and latents get
networks 30 42 or deep implicit fields [32
ignored by the decoder 39
48
To combat this, the feature volume can be divided into smaller
cells 43 33). More recently, vector quantized deep implicit functions (VQ-DIF) [36] 31 have gained popularity. VQ-DIFs encode shapes as series of discrete 2-tuples representing shape features and their position. This allows separation of specific features and effective re- re-combination in transformee-based architecture. We adopt this approach our work due to its high generation quality.
Condiiiidiio 3D Shape Generation: Text-conditioned 3D shape synthesis is significantly more challenging than texl-conditioned image synthesis due a lack of large text-shape datasets. Therefore, current methods use large pre-trained image-language embeddings like CLIP [36]. CLIP-conditional shape generation methods [50 40 20) associate encoded CLIP text features and CLIP image features
obtained by rendering 3D shapes. Alternatively, sequence-r--sequenc models 66 2 31 28) are used
to match texts and shapes. Unlike our method, none of the above methods can support long phrase
sequences or recursive shape generation that controls local shape details.
Training text-conditioned. 3D shape generation models requires supervision in the form of text-shape pairs. Unlike in 2D, obtaining this labeled data is harder in 3D due to the complexity of creating text descriptions for large datasets like ShapeNet [5]. Nonetheless, Text2Shape (6 contains such data for two object classes (chairs, tables) in ShapeNet. In total, they provide 75K pairs (30K for chairs, 40K for tables) which still insufficient for training models that capture local detail. ShapeGlot (2) provides a (2D) shape discrimination dataset with 94K rendered triplets for shape discrimination. None of these datasets support recursive shape learning. In our work, we provide method to transform the Text2Shape dataset into larger dataset of phrase sequence sopueece-shape pairs that supports recursive shape generation.
Datasets:
3 Background
We first introduce fundamental technical details and justification for our choice of shape representation, text feature extraction, and auto-regressive shape generation.
Neural implicit representations 32 have been shown to capture shapes of arbitrary topology using global latent codes. However, single global latent codes result in poor local details, so we use grid-based representations [3] which offer better local shape details. In this paper, we use vector-quantized feature embedding space which represents each shape as probability distribution of possible tokens in a fixed-size code book.
Shape Representation:
For each shape X, shape encoder Es encodes its T-SDF (Truncated-Signe) -Signd Distance Field) into a low dimensional 3D grid feature E E ga x D, where is the resolution of the 3D grid. and D is the dimension of the 31) grid features. We define a discrete latent space which has embedding vectors, where each of the vector has dimension D). For shape feature 6 at grid position i, a vector quantization operation VQ(.) is used to look up its the nearest neighbor e from the K embedding vectors in the discrete latent space. The feature put into the shape decoder Dz is the 3D discrete latent feature grid E' E g x D, with discrete features e: at grid locations i. Therefore, we have:
E Ex(X), E' (Q(E), X' (E')"),
where X' is the reconstructed shape, and di is the set of learnable parameters of the decoder. As result, shape X can be represented a 3D latent feature index grid Q gå, where the number qi at
3

---

each grid location corresponds to a discrete shape feature e/. To learn this, we use P- P-VQ- VAE [I], which extracts discrete latent features from volumetric T-SDFs. our method, we empirically set K = 512, D = 256, and the 3D shape grid resolution is g 8.
Text Feature Extraction: To extract semantic meaning from text inputs, we BERT 12
BERT is
a bidirectional transformer pretrained on a large web-scale dataset that can be fine-tuned for different tasks. For each input text /, we use the first token in last layer of the BERT BASE model as the text feature embedding B E 17765 In order to have all cells in the 3D feature grid to gather some semantic information from text input, we use a multilayer perceptron network (D-) to project the text embedding feature such that it has the same resolution as the 3D feature grid. We have:
C = (B)) 4
7 - R
(2)
where C' denotes the projected text features.
Autoregressive Generation: Both the 3D latent feature index grid Q extracted using P-VQ-VAE and the projected text feature C extracted by BERT are in the shape of 3D grid. Features within the grid are correlated with each other. Our goal is to build generation model that learns the coittdistributtion of the grid while making the code in each grid dependent on previous phrase inputs.
To achieve this, we use conditional autoregressive model for shape generation. Given grid-wise conditional feature Z E g H, where H is the dimension of the conditional feature (we set H = 512), the joint distribution of the 3D latent feature index grid Q and the conditional feature is formulated as z() II p(g: (eei, 2). We use the simplified assumption [3D that this joint distribution is product of the shape prior, coupled with independent conditional terms Z:
g3
pladless Z) = II
po polas1<e) po(a Z),
(3)
where and da are trainable parameters.
Recursive Text-conditioned 3D Shape Generation
We now describe the details of how we achieve recursive text-conditioned shape generation. We first introduce our Text2Shape++ dataset in Section 4.1 since the dataset design influences our technical approach. Next, we introduce the "shape set" feature representation we use for recursive generation in Section 4.2 We then introduce the inference procedure of our method in Section 4. Finally, we
introduce our training strategy that enables recursive generation Section 4.4
4.1 Text2Shape++: Dataset to Support Recursive Shape Generation
Our goal is to generate 3D shapes recursively from phrase input enabling the gradual evolution of generated 3D shapes. Unfortunately, no existing dataset can support this problem since text-shape pairs are limited to single phrases. Therefore,
construct new dataset, Text2Shape++. building
upon the Text2Shape [6] dataset, which has text-shape pairs of chair and table categories from ShapeNet [5). Tex22Shape++ contains 369K text-shape pairs, which to our knowledge is the largest dataset of its kind. Each text prompt in the dataset is represented as a phrase sequence, and each phrase sequence corresponds to one more shapes. On average, each phrase sequence corresponds to 16 models for the chair category, and 25 models for the table category.
Given each text prompt in the Text2Shape [6] dataset, we first split the text into sentences using a sentence splitter [99. For each sentence, we use constituency parser 2 to parse the sentence in to constituency tree. There could be many phrase types in constituency tree-we only consider 18 of them as valid phrase types that contain descriptions such as verbs, nouns, and adjectives. Meanwhile, we also mark the phrases that only contain stop-words 37 (e.g., it, there....) as invalid. During the parsing procedure, if any child of the current node is not a valid phrase, stop parsing. Thus, sentence parsed into constituency tree, where all the leaf nodes are valid phrases that contain useful information. For a parsed tree Y, we use depth-first fiistffistttt search to get all leaf nodes in the tree I [iz, 22, f2. ..., kr]. where T denotes the number of leaves. We use I, to denote the concatenation of the 1s to the tth phrases, and refer to L as a phrase sequence.
Constructing Text2Shape++:

---

Cocettueryy Paring Thee
Phoee and Shepos
wooden dimning
AP
VP
wooden dinning chair with Four logs.
Pe
Aljp
*A wooden dimning chair with four
legs. The chair has slat back"
dimning
with
turned
"A wooden dinning chair with legs.
wooden
chair
legs.
back
legs.
The chair has slat back and turned legs."
Figure 2: An example from Text2Shape++ Constituency parser [23 annotates sentence with syntactic structure by decomposing into phrases. Tex12Shape++ contains phrase sequences, and each phrase sequence corresponds one more shapes.
To get shapes corresponding to phrase sequences, we calculate the similarity score of all the phrase sequences our dataset with RoBERTa [27]. We consider two phrase sequences as similar if their similarity score is higher than certain threshold (we empirically chose 0.94). For each phrase sequence, its corresponding shapes include the original Text2Shape dataset pair as well as shapes paired with its similar phrase sequences. Therefore, Text2Shape++ is many-to-man text-shape correspondence dataset.
4.2 Shape Set Feature Representation.
Since any phrase sequences in Texi2Shape++ can have many shapes (i.e., shape set) as their ground truth, we represent the shape set as probability distribution of the discrete latent space rather set of deterministic discrete latent feature codes. Recall that we use P-VQ-VAE (3] (see Section 3) to represent a 3D shape X as a 3D discrete latent feature index grid Q 9. The discrete latent fea- ture space has K latent features. Therefore, for a shape set that corresponds to a phrase sequence, it can repre- sented as probability distribution of latent feature codes E gd x K, as illustrated in Fig.
Probability Distribution
index of code:
k-1
Figure 3: ShapeCrafter learns the prob- ability distribution latent features for each cell in Z.
Formally, each phrase sequence is provided with a shape set {X') and their corresponding similarity scores (w> M where M denotes the number of shapes in the shape Given shape X represented as 3D discrete latent feature index grid Q", its probability distribution latent feature codes Z is deterministic, which can be represented as Zu:- I[k Qi). where denotes grid at position i, and k denotes the btth dimension. Given a shape set { Jン=1 where each shape X is represented by 3D discrete latent feature index grid Q' we represent the shape set as probability distribution by simply weighting the probability of the all the shapes with their similarity score:
1==1
ΣE und 7
Z Zaer
(4)
Σ
1== wi
where zet is the shape representation of the shape set.
4.3 Recursive Shape Generation
We now describe the recursive inference procedure of ShapeCrafter which is also illustrated in Figure 4 ShapeCrafter uses the probability distribution latent feature codes (Section as its shape/Shape set representation. To achieve text-conditioning, the phrase input used change the probability distribution of the latent feature code Z. As more phrases are added, the probabilities become narrower resulting in more deterministic shapes.
At time step I, ShapeCrafter takes two inputs: text input i, from the current phrase, and the probability distribution of the latent feature code from the previous time step Z.-1- At the first time step, we set Zo to 1. where K. features B, and the multi-layer perceptron network (--) projects the feature to C, which has the same
512 for each cell. The BERT BASE model (Section extracts text
5

---

Project
Ceoratrauee
Bort
Trining
Inference
rectungsiar
f2
beckwudds
Zg
41
21
Figure 4: (Top) We take input text phrases and extract semantic features using BERT. These features are projected to feature grid C, which is concatenated with the latent feature code distribution Z,-1 from the previous time step. Residual blocks V((-) output the feature grid distribution Z, for the current time step. (2) During step of inf1 2erence, we combine C, and 7 obtain Zr which sampled to produce 3D shapes.
resolution as the 3D grid features. The text feature C' is then concatenated with shape set feature Z,-1 along the channel dimension. Several residual blocks vi(-) used to infer the probability distribution of the latent feature code at time I, which could be formulated as:
Z, = V ((Z-,,,
(5)
where [:] stands for concatenation. Please see supplementary document for more details.
The inferred probability distribution Zg includes information from both text and shape at the previous step. However, the features each grid are only weakly correlated. Therefore, Z, is fed to the autoregressive generation model (Section 3 which generates the 3D discrete latent feature index grid Q. The VAE decoder De introduced in Section 3 then decodes the index grid Q shape represented by a T-SDF.
At each recursive generation step, want the generation model to prioritize distinct shape features from the current phrase at Therefore, we sort the sequence of inputs at the autoregressive step by the difference of the probability distribution to generate the index grid sequentially, which could be formulated as:
time steps. We then random transformer 445
N
Pe 1g696,19644 c.: 2) = I (gb (9b.) Pew (4b, (Z), (b.) argsort(]] 1177--2-1111 Ze-1112)
(6)
where g is the number of grids, and argsort(-) stands for descending sort.
4.4
Training
Five components are involved in training ShapeCrafter: (1) P-VQ-VAE model for shape representa-
tion; (2) Auto-regressive model for shape generation; (3) fine-tuning BERT BASE model for text feature extraction; (4) Text feature projection model 4(-); and (5) Residuals blocks V (.) to extract the final distribution. We first follow the training strategy of AutoSDF [I] to train the P-VQ-VAE model and the autoregressive model, which are trained with both the feature of single shape and the feature of shape set. Then, we jointly train the BERT BASE model, the projection model and the residual blocks. Text2Shape++ provides us the dataset to train these three components since it provides phrase sequences = [i1, i2, .2,, ir] and the corresponding shape sets [X? ) ( At time step I,, we feed in phrase i, and the shape set probability distribution feature 7,-1 corresponding to I, Note that Z,-1 is taken from Text2Shape++. We receive as output the predicted shape set probability

---

distribution feature 2, Our loss function consists of reconstruction loss, vector quantization objective, and commitment loss as proposed by van den Oord et 47 For the reconstruction loss, we use cross-entropy loss with Ze as the label. For the vector quantization objective, we randomly sample from the the distribution Ze and acquire latent feature index Q: as the target.
5 Experiments
In this section, we provide both quantitative and qualitative evaluation of ShapeCrafter on recursive text-conditioned shape generation task. Specifically, we focus on: (1) quantitative evaluation and comparison with AutoSDF 3 to assess the ability of ShapeCrafter to generate high-quality shapes that correspond text input, (2) quantitative evaluation and comparison with AutoSDF [31 of the ability to handle phrase sequences of varied lengths, (3) qualitative and quantitative evaluation of recursive cext-conditioned shape generation, and (4) ablations to evaluate recursive generation in preserving shape details from previous steps. For consistency, we limit our experiments to the chair category, but we show results on the table category in the supplementary document.
Dataset: All our experiments use our Text2Shape++ dataset for training (20% held out for validation). For comparisons, we use full sentences from the Text2Shape dataset and phrases from Texr2Shape++ dataset for valida- tion. Metries: We use the following three met- rics to evaluate text-shape correspondence and shape quality.
black object
with four loga
cushioning.
Mittal et al
CLIP-Similarity (CLIP-S). This metric com-
ShapeCrafter
pares the teap-shape correspondence among all the methods as measured by CLIP [36). We first
rendered the generated shape into images from 20 different views. We then use the pretrained CLIP (36) to extract the text feature and images features, and calculate the similarity between text feature and all of the shape features. We use the maximum of the similarity from the 20 view images as CLIP-Similarre be different because of the different rendering settings. We normalize the ELIP-Similarity using two standard models, the Stanford bunny [46] and the Utah teapot because we want to reduce the effect of rendering differences. We calculate the cosine similarity between rendered pictures of the Stanford bunny and the word "teapot" and use this score as a lower limit. Then, we take the max cosine similarity between pictures of the Stanford bunny and the word "bunny", and use it as the upper limit.
Figure 5:
Qualitative comparison with Au- ShapeCrafter produces sequentially more consistent shapes compared to AutoSDF.
toSDF 31.
Even for the same text-shape pair, the CLIP- Simmlaaiitt value can
SiapeGGGoo-Cofideneee (SGLOT-C). The Chaneellt-Confidennee metric compares the text-shape correspondence between methods. A neural evaluator trained as proposed [2] to distinguish the target shape from distractors given the specified text. The trained neural evaluator achieves 83% accuracy on this binary classification task with the Shapeglot [2] datasct. SaapeGlot-Confiddee informs which shape most closely corresponds the text input among all shapes generated by the methods that we are comparing. We use this as proxy for perceptual study.
FID. We use the Frechet Inception Distance (FID) metric [18 to evaluate the quality of the generated shapes. For shape feature extraction, we use the voxel encoder in (53) which trained on the ShapeNet classification task 5
5.1 Comparisons
We evaluate and compare the performance of ShapeCrafter on text-conditioned shape generation [1 For this experiment, we use single step generation for AutoSDF using complete prompts from Text2Shape ( test set (since AutoSDF is not designed for recursive generation). Our model generates recursively by taking the sentence as input, parsing into phrases, and generating according to the parsed phrases recursively. Table iillsstrate the performance of ShapeCrafter and AutoSDF(3]
T
7

---

this task. We outperform AutoSDF on the CLIP- Similaiil score and the Confidence score, which means ShapeCrafter is able to generate shapes that are visually more similar to the text input. For shape quality evaluation, our model has lower FID score, indicating that we can capture details better. This experiment also shows that the recursive generation strategy is useful, compared to the single step strategy.
we also compare the performance between Auto 31 and ShapeCrafter qualitatively, We visualize the shapes generated by phrases from Texi2Shape++. Clearly, comparing with AutoSDF, the generated results of ShapeCrafter is more sequentially consistent.
In Figure 5
Table 1: AutoSDF and ShapeCrafter on text-conditioned generation. Compared to AutoSDE ShapeCrafter performs better on CLIP-S, SGLOT-C, and FID, which indicates that it provides better text-shape correspondence and shape quality.
Metric
CLIP-S
SGLOT-C1
FID +
Mittal et al. BI
48.92
0.46
18.45
ShapeCrafter (Ours)
52.43
0.53
16.36
5.2 Phrase Sequence Length
We are also interested in the ability of ShapeCrafter on generating text inputs of various lengths. We divide the texts inputs into three levels by the number of phrases they contain. All text inputs are categorized into three tiers: texts that contain 1 - 2, 2 2-4 4, and more than phrases. Table compares the performance of AutoSDF and ShapeCrafter on text inputs of various lengths. Overall, the results from these three metrics indicate that ShapeCrafter generates shapes that are more consistent with longer text inputs than AutoSDF without compromising generation quality. Specifically, for FID, the shape quality of AutoSDF decreases as the text input grows longer. while the shape quality of ShapeCrafter remains stable. For CLIP-Simillrity, the performance of ShapeCrafter is comparable with AutoSDF for shorter texts, and ShapeCrafter is better than AutoSDF at generating shapes that closely agree with medium-length and long texts. For Chaapeglot-Confidenee better with shorter/medium--lnntth text inputs, but ShapeCrafter performs better with longer inputs. The result shows that recursive generation strategy outperforms single step strategy in longer text inputs.
AutoSDF performs
Table 2: AutoSDF and ShapeCrafter on recursive text-conditioned shape generation. ShapeCrafter uses recursive strategy and it performs better with longer inputs.
Phrases
[1. 2]
(2, 4]
(4, +00)
CLIP-S SGLLOT-C FFID JCLIP-S 1SGLOT-C 1FID JCLIP-S 1SGLOT-CC 1FID +
Mittal et al.
45.72
0.53
18.13
45.27
0.42
20.33
55.77
0.43
22.47
ShapeCrafter 45.72
0.47
17.40
53.38
0.58
16.44
58.18
0.57
16.89
5.3 Recursive Text-conditioned Shape Generation
We evaluate the performance of ShapeCrafter on recursive text-conditioned shape generation task qualitatively. Fig. [ shows the shape generated at each time step. Shapes rendered in the same color are generated from the same phrase sequences. This figure provides examples that demonstrate ShapeCrafter has the ability to (1) generate high-quality shapes (row 1), (2) modify the global structure of shapes (row 2), (3) change part-level attributes (row 3-5) including part addition, part removal, and part replacement, (4) modify the local details of the shape (row 6), (5) generate from long phrase sequences (row 7 example 1), (6) generate novel shapes (row 7 example 2). The above properties shows ShapeCrafter performs well in text-conditioned shape generation and text-conditioned shape editing.
Table shows how the probability distribution Z and generated shape changes with the number of text phrases increases. For text phrase sequence. we calculate (a) the mean en)ropy(H) of the
per per-grid-cell discrete probability distributions H = Σ. Σk 2 log(zu))/4 and (b) the mean Chamfer Distance(CD) among 8 randomly sampled shapes.
The table shows that the entropy and the mean Chamfer Distance decrease with more phrases. This indicates that as more phrases are added, the probabilities become narrower resulting more deterministic shapes.
8

---

type
nolded swat
octal footswat circular base
xetallic
stand and wheels
Heat
Rooden chair
w..htth long back
and Long laga cornerx hax handlea
The ararwats and
and the chair slopes elightly
A sofa with rectangolar and padding
metal chair
with thin laga
a.d two curved
The hax space wtart the next the start of the back steelf
roders and luglaxx sedgeed back
...aingg pedeatal seens the atr
rectangle
hand reate are cyltsorical shape
cushioned chair with ash colour
co.tthh chaped stretsbed deck cbair aluntnon froking for support
orange
shaped
vodden polished yellow colour eith reclining back
square arnrest both the cides
Ssall metal office type chair
wi.h
cushioas and back it includos 4 wheels
logo
Black office chair
.i.hith
laga that square side
support
Rooden chaiz with logo
shape
this chair has balck single seats chinr
SES azdo IEOM
shape has cope
VIEA cizcle 1t иррентя competative
Moders, shaped, wooden chair
Figure 6: Qualitative results produced by ShapeCrafter. Each unique color shows results generated recursively from our model using the same seed to sample the shape distribution. Our method demonstrates consistent and gradual evolution of shape enabling us to simulate shape editing.

---

Table 3: The (H)) of the probability distribtuion and the Chamfer Distance(CD) between the generated shapes at sentence with different number of phrases. The probabilities narrow more phrases are added, resulting in more deterministic shapes.
# Phrases
[2, 4)
[4, 8)
[8, +00)
H
0,423
0.341
0.294
0.267
CD
0.0861
0.0698
0.0359
0.0261
5.4 Ablations
We design ablations to evaluate whether: (1)
Table 4: Ablation studies shows the effectiveness of the conditional training, the random transformer and the reordered random transformer sequence.
the conditional training strategy improves text- shape correspondence, (2) the random trans- former improves shape quality; and (3) reorder- ing transformer inputs preserves shape details. To prove the effectiveness of conditional train- ing, we design baseline removing the residual blocks (-) and only generating shape features recursively with AutoSDF [3I]. Specifically, at time step t. multiply the probability distribu- tion C. generated from the text input with the probability distribution generated from the random transformer t 1, and the product as the conditional distribution of the random transformer at t. We refer to this baseline as w/o condition. To prove that the random transformer helps improve the quality of the generated shapes, design baseline w/o transformer, where we sample shapes from the probability distribution generated by the residual blocks (-).. To prove the effectiveness of reordering the input sequence in the random transformer (refer to section 4.3), we design another baseline where the transformer generates features in random sequence. We refer this baseline w/o reorder. The performance of these baselines are illustrated in Table. 4 The performance drops without these components, which proves the effectiveness of our design choices.
Metric
CLIP-S f
FID 4
w/o condition
49.53
20.22
w/o transformer
49.29
19.83
w/o reorder
42.47
16.67
Ours
52.43
16.36
Conclusion
In this paper, we present ShapeCrafter, method for recursive text-conditioned 3D shape generation. Different from previous single-step methods, our focus on generating 3D shape distributions that can be gradually evolved to capture semanties described in phrase sequences. We also presented method to transform an existing dataset to support recursive shape generation tasks to produce a larger dataset called Text2Shape++ with 369K shape-text pairs. Results show that our method produces high-quality shapes while being able to handle long phrase sequences, support shape editing, and extrapolate to shapes unseen during training.
Limitations: Our method has several notable limitations. First, ShapeCrafter limited shape gen- eration only and cannot handle appearance attributes common in language descriptions. Limitations of the Text2Shape dataset extend to our method we only support two shape categories: tables and chairs. Empirically, we observe that our method demonstrates elements of "part assembly". but cannot handle large-scale deformations. The proposed recursive shape generation method is not reversible. For example, given an original shape with inputs "with armrest" and "without armrest" applied sequentially, the generated shape may not be identical to the original shape. Another noticeable problem in the text-conditioned shape editing community is that there lacks common metric to evaluate the accuracy of shape editing. We believe that we have taken a step in the direction towards addressing the above challenges. From societal impact perspective, more care is needed to ensure that our dataset represents diverse shape instances to avoid bias.
Acknowledgements This research was supported by AFOSR grant FA9550-21-1-02114 NSF CNS- 2038897, and the Google Research Scholar Program. Daniel Ritchie is an advisor to Geopipe and owns equity in the company. Geopipe is a start-up that is developing 3D technology build immersive virtual copies of the real world with applications in various fields, including games and architecture.
10

---

References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J Guibas. Learning representations and generative models for 3d point clouds. arXiv preprint N.Xkv:/707.022.. 2017.
[2] Panos Achlioptas, Judy Fan, X.D. Robert Hawkins, D. Noah Goodman, and J. Leonidas Guibas. ShapeGlot: Learning language for shape differentiation. CoRR, has/1905.02925. 2019.
[3] James Blinn and Martin E Newell. Texture and reflection computer generated images. Communications of the ACM, 19(10):542-547 1976.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:18 33:1877-1901, 2020.
[5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rel 3D Model Repository. Technical Report arXiv: 1rXXiv1112..08 1512.03012 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
[6] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian conference on computer vision, pages 100-116. Springer, 2018.
[7] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceed- ings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[8] Noam Chomsky. Minimal recursion: exploring the prospects. In Recursion: Complexity in cognition, pages 1-15. Springer, 2014.
[9] Christopher Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), 2016
[10] Michael C Corballis. From hand to mouth. In From Hand to Mouth. Princeton University Press, 2020.
[11] Angela Dai and Matthias Nicßner. Scan2mesh: From unstructured range scans to 3d meshes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5574-5583, 2019.
[12] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[13] Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira
Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. Keep drawing it: Iterative language- based image generation and editing. arXiv preprint
2018.
[14] Stanislav Frolov, Tobias Hinz, Federico Raue, Jörn Hees, and Andreas Dengel. Adversarial tontto--mmaee synthesis: A review. Neural Networks, 144:187-209, 2021.
[15] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9785-9795, 2019.
[16] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen- Or. Meshenn: a network with an edge. ACM Transactions on Graphies (TOG), 38(4):1-12, 2019.
[17] Xiaodong He and Li Deng. Deep learning for image-to-text generation: A technical overview.
IEEE Signal Processing Magazine, 1496):009-116, 2017.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] Matheew Honnibal and Mark Johnson. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1373-1378, Lisbon, Portugal, September 2015. Association for Computational Linguistics.

---

[20] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. arXiv preprint arXiv:2112.01455. 2021.
[21] Danilo Jimenez Rezende, SM Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. Advances in neural information processing systems, 29, 2016.
[22] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[23]
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2676-2686, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[24] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr. Controllable text-to-image generation. arXiv preprint 190i: 00009999990000,.... 2019.
[25] Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. Point cloud gan. arXiv preprint arXiv: 2018.
[26] Fayao Liu, Chunhua Shen, and Guosheng Lin. Deep convolutional neural fields for depth estimation from a single image. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 5162-5170, 2015.
[27] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint /907 2019.
[28] Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi- Chi-Wing Fu. Towards implicit text-guided 3d shape generation. arXiv preprins 220.::2222..422222 2022.
[29] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[30] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer, 2020.
[31] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. arXiv preprint urXiv:2203.09516 2022.
[32] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[33] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523-540. Springer, 2020.
[34] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[35] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 9748-8763. PMLR, 2021.
[37] Anand Rajaraman and Jeffrey David Ullman. Mining of massive datusets. Cambridge University Press, 2011.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2004006125, 2022.
12

---

[39] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero- shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[40] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Y Yi Cheng, and Marco Fumero. Clip-forge: Towards zero-shot sex-tt--happe generation. arXiv preprint arXiv:2110.02624. 2021.
[41] Matan Shoef, Sharon Fogel, and Daniel Cohen-Or. Pointwise: An unsupervised point-wise feature learning network. arXiv preprint urXiv:/90/.044: 2019.
[42] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks:
Continuous 3d-structure-aware neural scene representations. Advances in Newral Information Processing Systems, 32, 2019.
[43] Towaki Takikawa, Joey Litalien. Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358-111 11367, 2021.
[44] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao Yuan Jing, Fei Wu, and Bingkun Bao. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008. 05865, 2020.
[45] Shubham Tulsiani and Abhinav Gupta. Pixeltransformer: Sample conditioned signal generation. arXiv preprint arXiv:2103.
2021.
[46] Greg Turk and Marc Levoy. Zippered polygon meshes from range images. In Proceedings of the 21 annual conference on Computer graphics and interactive techniques, pages 31 311-318, 1994.
[47] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[48] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[50] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. arXiv preprint arXiv:2 12.0519,, 2021.
[51] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu- Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52 67, 2018.
[52] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[53] Udaranga Wickramasinghe, Edoardo Remelli, Graham Knott, and Pascal Fua. Voxel2mesh: 3d mesh model generation from volumetric data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 299-308. Springer, 2020.
[54] Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. Cnns surfaces using rotation- equivariant features. ACM Transactions on Graphics (TOG), 39(4):22-1, 2020.
[55] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1324-1324, 2018.
[56] Xingguang Yan, Ligiang Lin, Niloy J Mitra, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Shapeformer: Transformer-basee shape completion sparse representation, arXiv preprint arXiv:2201.10326 2022.
13

---

[57] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to-image synthesis using contrastive learning. arXiv preprint arXiv:2107.02423: 2021.
[58] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 833-842, 2021.
[59] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for -oxteto-imaee synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802-580 5810, 2019.
14


Reviews:
[{"review_id": "wJkGE9hf-FU", "reviewer": "Reviewer_Bwe4", "summary": "The paper presents a neural network for recursive text-conditioned 3D shape generation --- progressively updating the generated shape by a series of input phrases. To achieve this, the proposed method is built on top of vector-quantized deep implicit functions (as the shape representation) such that a set of possible shapes can be described as a distribution over 3D grids, which can then be learned using an autoregressive model. Another contribution of this paper is a enhanced dataset, Text2Shape++, that provides many-to-many shape-text pairs. The proposed method largely relies on such dataset.\n\nThe results demonstrated that the method can generate shapes from text input, and the shapes evolve gradually as more short phrases are added. The authors also include a realtime demo where the user describes a shape by speech and the generation results are shown in the screen.\n", "questions": "Questions and suggestions:\n- Line 202-205, the sorting is a bit counter-intuitive to me. Wouldn't this sort operation make the order inconsistent among the dataset? My first thought is that such inconsistent order would hurt the training. I suggest elaborating more on the random transformer, which seems to be a key choice here.\n- Line 237-240, why is it necessary to normalize clip-similarity score? Cosine similarity by itself already has limited value range.\n- For all loss functions described in Sec. 4.4, I suggest including their mathematical formulations, at least in the supplementary document, to make the paper self-contained.\n\nTypos:\n- Line 139, \"since since\" -> \"since\".\n- Line 148, \"... is the largest such dataset\" -> \"... is the largest dataset of its kind\".\n- Line 177, $X_j$ -> $X^j$.\n", "limitations": "Limitations are well discussed and several failure cases are presented in the supplementary document. Negative social impact is also addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It is an interesting idea to represent a set of possible shapes as a probability distribution over latent feature codes. With an autoregressive generator, this design naturally supports probabilistic generation (i.e., capable of producing different shapes from the same input). This is a different and novel approach compared to previous text2shape methods.\n- The Text2Shape++ dataset, even though constructed for a particular scenario, is a good contribution to the community. I like the way that the authors construct the dataset, i.e., use principled natural language techniques such as constituency parser and phrase similarity.\n- For an application paper, a real-world demo is a bonus. I appreciate the good effort by the authors in making such demo.\n\nWeaknesses:\n- My main concern is about the result quality and evaluation. \n    - Poor visual quality. Many of the generated shapes (i.e., those in Fig.1, Fig.3, Fig.4) have noticable artifacts, e.g., holes, disconnected or floating parts. Given the visual quality of the demonstrated examples, I would doubt the overall result quality.\n    - Missing comparison to previous SOTA text2shape methods, especially [1]. The compared method, AutoSDF is not especially designed for text-driven shape generation. Another fact that I noticed is that this work ignores the shape color (so does autoSDF), which is an important information for many text descriptions and is considered in previous text2shape literatures [1][2]. I'd like to hear the reason that the surface color is not considered. Clearly the dataset itself contains colored shapes.\n    - Missing support for the made claim. Specifically, line 187-188, \"As more phrases are added, the probabilities become narrower resulting in more deterministic shapes.\". I think this should be a key feature of the method, but somehow it is not validated in the experiments.\n- Another minor concern is the application scope or the motivation for recursive text-conditioned generation. From my perspective, the advantage of text-driven 3D shape generation is to quickly produce a base/rough shape. People (modeling experts or ordinary users) then start from that shape for further detailed editing, from where precise control is more important. However, from the presented results, I'm not convinced that recursive text input can provide such necessary precise control (as text itself is vague), compared to traditional 3D shape editing techniques.\n\nGiven that my concerns outweigh the positive contribution of the paper, I'm inclined to rejection at this point.\n\n[1] Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. CVPR 2022.\n[2] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. ACCV 2018.\n\n__Update:__\n\nAfter reading the rebuttal and other reviews, I feel more positive about this paper as some of my major concerns are well addressed (missing comparison, lacking support for the claim). Thus I raise my rating from 4 to 5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZWk0ePZsb8f", "reviewer": "Reviewer_e81L", "summary": "The paper introduced a novel task, namely recursive text-conditioned shape generation. In this setting, the model needs to generate a 3D shape given the initial text, and subsequently evolves it in multiple steps given more text descriptions.\n\nTo achieve this goal, the author built a new dataset (Text2Shape++) which contained many-to-many mappings of phrase sequences and shapes, derived from Text2Shape dataset [5] which only contained one-to-one mappings.\n\nTogether with the dataset, the paper also presented a shape generation model that supports generating and refining a shape in multiple iterations. The model is based upon AutoSDF [30]. In the first stage, a VQ-VAE-based shape generation model is trained, and a transformer-based generator is trained on top of the VQ-VAE latent, which can be used to refine a given latent. In the second stage, a recursive feature refinement model based on BERT and 3DCNN is trained directly with text and VQ-VAE latent.\n", "questions": "* The math notations in Section 4.2 is confusing. I wonder what is the $j$ in L178 representing?\n* L197 mentioned that the $Z_t$ generated by the recursive language model is send through a transformer before feeding to the 3D decoder. I wonder how important this step is? And how is this implemented? Do you sample from $Z_t$ before feeding the sample into the transformer?", "limitations": "Limitations and societal impacts are adequately addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The proposed recursive generation task is both novel and useful, which can inspire future research.\n* The curated dataset is a valuable and unique addition to the current text-to-shape datasets.\n* The new model achieved better generation quality on the proposed dataset compared to previous work, justifying the need of a model tailored for the new task.\n\n### Weaknesses\n* The proposed model is trained with strong supervision enabled by the new dataset, as the exact matching between arbitrary phrase sequences and shape sets is available. Building such a dataset is not always feasible for real-world applications with more shape classes and more complex shapes. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DbKaDEfvxSg", "reviewer": "Reviewer_BoQp", "summary": "The paper discusses a method to sequentially generate a chair from a sequence of utterances provided by the user. To do so the paper first cleans the Text2Shape++ dataset by applying linguistic parsing. Their model consists extracting text features from text, that are then projected to a volumetric grid and concatenated to the previously generated volumetric features. Then this tensor is used to generate an SDF to represent the shape. The paper shows examples of a couple of sequential edits.\n\nI wonder why authors used Recursive in the title, since it has a connotation that it uses a recurrent NN, or any other recurrent process. Here instead it's sequential, with not too many steps presented in the paper.", "questions": "I'd like the authors to comment on the questions listed in the weaknesses section. I believe the paper is a good piece of work, that might be not quite ready for a publication", "limitations": "The limitation section is satisfactory", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nS1. The paper definitely tackles an interesting project, as far as I'm aware, this is the first of one of the first papers in this domain.\nS2. Their architecture is somewhat new, somewhat extended from ShapeFormer.\nS3. Written very well, thank you, very easy to read and follow.\n\nWeaknesses:\n\nW1. **Results**. The paper report standard metrics that evaluate how well the shape corresponds to the text. The paper, however, does something different, they propose a *sequential* generation paper. While ShapeGlot-Confidence or Clip-Score are necessary, they're by no means sufficient. How can we be sure that sequential generation happened. If you say \"a chair with straight legs\" and then add \"curved thin legs\" and then add \"with a hole in the seat\", we'd like to be sure that only the these attributes change, and only when asked. The paper doesn't provide a metric or evaluation protocol for that. A further interesting metric to consider is to evaluate that other parts of the chair stay the same or marginally change while attributes are added. The paper doesn't provide that too. Such metrics probably don't exist, but I believe the responsibility of coming up with a reasonable evaluation protocol bears with the authors and if done well can be listed as a contribution.\n\nThese metric are very important for such a method, because, if we look at Fig 3, we see that changes to not mentioned parts of the chair are significant. For example in the bottom row \"a black object\" gives you a chair with four legs, \"...with four legs\" gives another chair with four legs, \"...and cushioning\" gives another chair with four legs and slightly changed geometry. I think any of these three chairs perfectly satisfies a description \"a black object with four legs and cushioning\" I cannot tell whether the final variant is the best. Also it's not clear why \"...with cushioning\" makes the legs curvy. Inconsistency of sequential updates is also the case in examples in Fig.4.\n\nW2. **Adding vs Removing attributes**. Currently the utterances are additive, ie we add attributes to the chair, they might be conflicting such as \"stand legs\" and \"wheels\". But I wonder if it's possible to say mutually exclusive things, such as \"a char with four legs\", then \"a chair with one metal leg\" and then \"a chair with four legs\". OR \"a chair with armrests\" and then \"a chair without armrests\". I wonder if there is a metric to compute, when you do subtract attributes you should return to the exactly previous chair, perhaps it's even a decent loss function...\n\nW3. **Generality**. In its current form the paper operates only on chairs, for whatever reason they add attributes of materials to the text, while not being able to render \"black, metal, wooded\" and so on. Have you thought about other categories? The paper title is quite generic, while implementation is category specific. It would be good to have one model to generate multiple object categories. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept