Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program
Authors: Tiange Luo
Honglak Lee
Justin Johnson
Venue: Accepted by TMLR ()

Abstract: 3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs of shape abstractions: $\textit{Text}$ $\Longleftrightarrow$ $\textit{Point Cloud}$ $\Longleftrightarrow$ $\textit{Program}$. We propose $\textbf{\textit{Neural Shape Compiler}}$ to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into unified discrete shape code, transforms each shape code into code of other abstract types through the proposed $\textit{ShapeCode Transformer}$, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed $\textit{Point}$VQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in $\textit{Text}$ $\Longrightarrow$ $\textit{Point Cloud}$, $\textit{Point Cloud}$ $\Longrightarrow$ $\textit{Text}$, $\textit{Point Cloud}$ $\Longrightarrow$ $\textit{Program}$, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.

Published in Transactions on Machine Learning Research (03/2023)
Neural Shape Compiler: A Unified Framework for Transform- ing between Text, Point Cloud, and Program
Tiange Luo
Honglak Lee
Justin Johnson
9f
University of Michigan
LG AT Research
Reviewed on OpenReview: https: // owwnreeew.. net/ forum? id- gR9UVgH6P2
Abstract
3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs shape abstractions: Text + Point Cloud + Program. We propose Neural Shape Compiler model the abstraction transformation as conditional generation process. It. converts 3D shapes of three abstract lypes into discrete shape code, transforms each shape code into code of other abstract types through the proposed ShapeCode Transformer, and decodes them to output the target shape abstraction. Point Cloud code is obtained in class-agnostic way by the proposed PointVQVAE, On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in Tezt = Point Cloud, Point Cloud = Text, Point Cloud = Progrum, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.
1 Introduction
Humans understand 3D shapes from different perspectives: we perceive goometries, under- stand their composing parts and regularities, and describe them with natural language. Sim- ilarly, vision researchers designed different ab- stractions for 3D shapes, including (1) low-level geometric structures that plot detailed geome- try such as point clouds (Gruen & Akca, 2005; Qi et al., 2017a); (2) structure-aware represen- tations that can tell shape parts and their rela- tions, like shape programs (Tian et al., 2019); (3) natural language to describe compositionality and functionality (Cağdaş, 1996; Chang et al.,
the target adustment holes. the armrests stick out the one that has a pole connecting to the seat wheels, computer chair
office
top
Powe/Book Call PHOLA Co
Eveny/lie V. "Oat, PHOU) 0e/ (wibod))
enew/Chhs Dear: Vaty, P-B-A Seh CH5UZZ
Hatuall
Cisn/Hor Bar Oub: Patt: 6---- Go/t.
Figure 1: Example: Neural Shape Compiler helps obtain hierarchical information for the reconstruction result of a single image (Zhang et al., 2018), including its structural descriptions, regularitics, and how to assemble it.
2014). Different shape abstractions convey com- plementary information and encode different characteristics allowing researchers design specialized models for various tasks (Mitra & Nguyen, 2003; Chaudhuri et al., 2020). This paper studies how translate between Tezt < Point Cloud = Program. Such transformation mechanism can provide multi-level information about shape (Figure 1) to support various rownstteea tasks. Additionally, we demonstrate that combining the learning of different. transformation branches within a unified framework leads to stronger representation learning and improved performance on each individual task.
Equal Advising.

---

Published in Transactions on Machine Learning Research (03/2023)
Neural Shape Compiler
Text
Text
chair has slats between legs with armrests
Neural Shape Compiler
ShapeCode
Encoder
Decoder
PointCloud Encoder
ShapeCode Transformer
PointCloud Decoder
Neural Shape Compiler
reDid Thond Sa5 4o 6-6,1121
CreeTT:: Starr, P-boicl G-RZE
sluutaaa The PrbLi SuITLI Beaaet
Program
ShapeCode
Program Decoder Backend
Neural Shape Compiler
chair with no armrest two wide legs reclining chair back
Encoder
Frontend
Figure 2: Overview, and three transformations conducted by Neural Shape Compiler, including Text = Point Cloud, Point Cloud = Program, and Program = Texf?. Red shapes are the rendered results by executing the corresponding program. ShapeCode Transformer is shared across different transformations. Neural Shape Compiler is not limited to performing the transformations shown above, and benefits from joint training with all heterogeneous data and tasks.
Modern program compilers (Lattner & Adve, 2004; Zakai, 2011) turn source codes into intermediate represen- tations (TRs), transform IRs, and decode them into other types of high-level programming languages. We take inspiration here and propose a unified architecture, Newral Shape Compiter', to perform transformation between the three shape abstractions with neural networks. It converts all shape abstractions into discrete shape codes (i.e., IR) via their respective encoders. Then, ShapeCode Trunsformer, implemented as the standard Transformer (Vaswani et al., 2017) for generality, transforms shape codes of one type to another in an autoregressive and probabilistic manner (Ramesh et al., 2021). The probabilistic way helps us overcome the issue of ambiguity that different shape geometries can correspond to very similar program or text. Finally, the compiled shape code decoded to the target shape abstraction through the corresponding decoder?. Figure? 2 shows the entire process and three example transformations.
To turn point clouds with continuous coordinates into discrete shape code, we propose Point inspired by (Van Den Oord et al., 2017). Unlike traditional point cloud process models (Qi et al., 201 7a), our encoder has restricted receptive fields (Coates & Ng, 2011; Luo et al., 2019) for Icarning shape part embeddings. Those part embeddings let the codebook (Van Den Oord et al., 2017) encode the parts of the input 3D shape rather than the entire shape. The decoder then combines all the 3D part codes to reconstruct whole point clouds in a permutation equivariant manner.
This paper focuses on modeling 3D shape structures. Our text. data contains detailed structural description ions and is devoid of color and texture, where we adjusted and annotated data from Text2Shape (Chen et al., 2018), ShapeGlot (Achlioptas et al., 2019), and ABO (Collins et al., 2021) datasets, resulting in 107,377 (Point Cloud, Structure- Beructure-Relteed Text) pairs. We synthesized 120,000 (Point Cloud, Program) pairs (Tian et al., 2019) for understanding shape parts and regularities with programs. Our experiments show that Neural Shape Compiler has the ability to generate point clouds from input text with geometric details, generate text descriptions of the structure of input point clouds, and generate programs for composing point clouds. Additionally, our experiments suggest the image-text pretraining model, CLIP (Radford et al., 2021), used by recent Tert = 3D methods (Jain et al., 2021; Sanghi et al., 2022; Nichol et al., 2022) is NOT suitable for generaling 3D shapes with structural details.
Furthermore, we train a model using all the data from tasks Text = Point Cloud, Point Cloud = Text, and Point Cloud = Program, and train a separate limited version of the model for cach task. Our experiments show that the jointly trained model consistently oulperforms each limited version model on the correspouding tasks and beats existing baselines on all aforementioned tasks.
II is important to note that Neural Shape Compiler has fundamental differences from program compilers, described in Soction 2. 2 TTaxt + Program achieved in twa-step generation Text + Point Cloud + Program. Program Point Cloud is deterministic via executing the programs
2

---

Published in Transactions on Machine Learning Research (03/2023)
Neural Shape Compiler is extensible. We show case where our framework is extended to perform Point Cloud completion by transforming partial Point Cloud code into complete Point Cloud code. Limitations and future work are discussed in Section 5, Our code, dataset, and pre-trained models will be released to facilitate future research in 3D multimodal learning.
2 Related Work
Compiler: Our proposed framework is inspired by the concept of computer program compiler (Calingaert, 1979; Appel, 2004; Aho et al., 2007; Aho & Ullma, 2022). Classical compilers translate high-level program language (e.g., Pascal, C) into executable machine code with the help of assemblers and linkers (Wirth et al., 1996; Appel, 2004). Comparod to classical compilers, our framework is more similar to modern program compilers (e.g., LLVM (Lattner & Adve, 2004) and Emscripten (Zakai, 2011)) where the target code is not limited to machine code and can be high-level programming languages. Our framework (Figure 2) shares similar architectures with LLVM: LLVM uses front-ends (encoders) to turn the corresponding source codes into intermediate representations (ShapeCode) and decode them via back-ends (decoders). A major difference between our framework and modern program compilers is that our TR transformation process probabilistic, whereas the process in a program compiler is a deterministic process with potential performance optimizations.
Multimodal Learning: With web-scale image-text data, our community achieved remarkable progress in multi-modal learning of 2D-text (Li et al., 2019; Ramesh et al., 2021; Radford et al., 2021; Patashnik et al., 2021; Alayrac et al., 2022; Ramesh et al., 2022). However, due to the lack of large-scale 3D-text pairs and baseline systems (He et al., 2017), there is slow progress in 3D-1ext modeling, Some recent works tried to leverage the progress in 2D-text multimodal learning (e.g., CLIP, Imagen (Saharia et al., 2022) (Radford et al., 2021)) for 3D-text modeling (Jain et al., 2021; Zhang et al., 2021; Sanghi et al., 2022; Liu et al., 2022a; Poole et al., 2022; Lin et al., 2022; Nichol et al., 2022). However, our experiments suggest CLIP is NOT suitable for text-guided 3D generations once text contains structural details. Compared to them, this work studies the connections between 3D and text directly (Chen et al., 2018; Fu et al., 2022), as there are significant compositional connections between shape parts and words. Our work is closer to Text-to- toxel works (Chen et al., 2018; Liu et al., 2022b), while none of them can generate desirable shapes corresponding to text prompt with levels of geometric details. Mittal et al. (2022) concurrently studied language-guided shape generation with learned autoregressive shape prior. Besides, the proposed framework is related to many-to- many-to-many deep learning frameworks (Ngiam et al., 2011; Jaegle et al., 2021; Reed et al., 2022).
Generative Model: PointVQVAE is inspired from VQVAE (Van Den Oord et al., 2017) studying variational auto-encoder (Kingma& & Welling, 2013) with discrete latent variables (Salakhutdinov & Larochelle, 2010). We leverage its most basic concept, vector quantization (Theis et al., 2017; Agustsson et al., 2017; Oord et al., 2016), and use the discrete codes as our IRs in Neural Shape Compiler. We did not exhaust the complex variants or techniques for gencrality, such as Gumbel-softmax (Jang ct al., 2016; Ramesh ct al., 2021), the exponential moving average in codebook updating, and multi-scale structures (Razavi et al., 2019; Vahdat & Kautz, 2020). Prior works studied learning 3D shape probabilistic spaces with GANs for shape synthesis (Wu et al., 2016; Nguyen-Phuoe et al., 2019); (Achlioptas et al., 2018; Yang et al., 2019; Cai et al., 2020) Iearn the continuous distribution of point clouds and sample thereon to generate shapes; (Mittal et al., 2022; Cheng et al., 2022; Yan et al., 2022) concurrently developed ways to quantize point clouds and voxels; diffusion models are also exploited in 3D shapes (Luo & Hu, 2021; Zhou et al., 2021).
Our approach is related to research on 3D shape understanding about geometry processing and structural relationship discovery. Recent learning systems for shape processing usually perform over some low-level geometry representations, such as point clouds (Bronstein & Kokkinos, 2010; Qi et al., 2017b; Liu et al., 2019; Luo et al., 2020), volumetric grids (Wu et al., 2015; Maturana & Scherer, 2015; Wu et al., 2016; Wang et al., 2019), multi-view images (Bai et al., 2016; Feng et al., 2018; Han et al., 2019), meshes (Groucix et al., 1802; Lahav & Tal, 2020; Hu et al., 2022), and implicit functions (Mescheder et al., 2019; Park et al., 2019; Genova et al., 2020). Some represent shapes in a more structured way, including hierarchies, trees, and graphs (Wang et al., 2011; Li et al., 2017; Sharma et al., 2018; Mo et al., 2019). Neural Shape Compiler adopts point cloud as one of its shape abstractions because point cloud can describe shape structures and can be easily acquired by real sensors. Regarding the types of shape regularities (Mitra et al., 2006; Pauly et al., 2008; Mitra et al., 2007; Wang et al., 2011; Xu et al., 2012), this paper mainly considers
3D Shape Understanding:
3

---

Published in Transactions on Machine Learning Research (03/2023)
extrinsic part-level symmetries, including transformational, reflectional, and rotational symmetries, which relate to part pairs and multiple parts. Our framework adopts shape program (Tian et al., 2019; Jones et al., 2020; 2021; Cascaval et al., 2022), which is one of the recent advances in relationship discovery. It designed a shape domain-specific language to encode shape relationships implicitly in programs.
3 Method
This paper presents unified framework that transforms between different shape abstractions and benefits from joint multimodal learning across multiple heterogeneous tasks. The key is tuning different shape abstractions into unified discrete space, which enables our model to handle multiple tasks simultancously. Specifically, we turn each type of input shape (Poinl Cloud, Text, or Progrum) into discrete shape code (lc)..., Gn), { E Z+) through their corresponding encoders. ShapeCode Transformer then transforms each shape code into the code that can be decoded into target shape abstraction via the corresponding decoder. Each encoder and decoder is designated to consume the specific type of shape abstraction, while ShapeCode Transformer communicates with all encoders and decoders via discrete code for all heterogeneous lasks.
3.1 Encoders& & Decoders
To turn point clouds P (pI,... Pmm (p; = (rr Vs. 2:)) into codes Cvoint = -,., pootmed {c] E Z+) while accommodating complex variations of shape structures. We draw inspiration from VQVAE (Van Den Oord et al., 2017) and propose a new model called Voint to address this task in a agnssti manner. Our approach involves designing a novel encoder and decoder architecture that enables Point VQVAE to reconstruct point clouds from shape codes, the same discrete space into which we convert Text and Program
Encoder
CodeBook
Decoder
fxt, 512
Txd,, 2048
fxt,
512,
Figure 3: Poin/VQVVA consists of three parts: (1) a hierarchical encoder that has restricted receptive fields and outputs multiple part embeddings; (2) shared codebook for vector quantization (Van Den Oord et al., 2017) (e.g., looking up the closest yellow embedding in the codebook for the yellow output of the encoder); (3) decoder with multiple residual building blocks consisting of Ix1 convolutional layers, a max max-pooling layer, and MLP layers at the end to output 2,048 points.
Point Cloud Encoder: The key point of Poin/VQVAE encoder to constrain its receptive fields (Luo el al., 2016) of the last layer neurons to very local regions (Luo al., 2019), so that the codebook behind can learn how to encode part of the 3D shape instcad of entire 3D shapes (Luo et al., 2020). Specifically, we propose hierarchical encoder with multiple down-sampling processes T, (currently implemented a f structure (Qi et al., 201 7b)). In each round T;, we first do farthest point sampling over input point clouds to sample a set of points noted as center points. Around each center point, we draw a ball with radius Rt; to gather information from all points inside the query ball. After this, the input point cloud is downsampled into smaller point clouds with updated embeddings at each point. Since the radius Ry, of our query ball is usually smaller than the scale of the whole point clouds, the receptive field of points in downsampled point clouds can be gradually increased as we repeatedly the downsampling process (i.e., increase #T).. In our experiments, we set the number of rounds #T = 2 and Rr, to be small values (Rr, 0.1, Rr, 0.4) for letting the final neurons (c.g., colorful points of the encoder's last layer in Figure 3) only receive local part information regarding the input shapes. Our experiments in Appendix C demonstrate that ensuring that the final neuron receives local information is critical for reconstruction.
Point Cloud Decoder: For each embodding in the last layer of the encoder, we look up its closest embodding in our codebook (Van Den Oord et al., 2017). We concatenate all the obtained embedding together as the input to our decoder. The concatenated embedding encodes the full information of the input shape
4

---

Published in Transactions on Machine Learning Research (03/2023)
by collecting the information encoded by different local parts of the input shape. Compared to common point clouds auto-encoders (Qi et al., 2017b), we discard the coordinate information of the final points of the encoder. Not using coordinate information may degrade auto-encoding performance but allow us to translate between codes successfully because we do not have any coordinate information if we translate from other shape ractions (i..,, lext and program). To compensate for this point, we designed a residual-networr-llin block (He et al., 2016) and made our decoder with great depth to exploit the information from the concatenate embedding fully, as shown in the right of Figure 3. A max-pool layer is added to the decoder's output to ensure the final embedding is invariant to the concatenation order of the embedding from the encoder. In our experiments, we adopt Chamfer distance (Fan et al., 2017) and Earth Mover's distance (Rubner et al., 2000) as our reconstruction loss and the straight-through gradient estimator (Bengio et al., 2013) for backpropagating gradients to the encoder by copying the gradients from the inpuls of the decoder to the encoder output. More details are included in Appendix B.1.
Program Encoder & Decoder: For program, due to the domain-spccific language of shape program (Tian et al., 2019) has limited discrete lypes of statements {A, B, C.C..... and ranges of parameters for each statement {a1, {2,,02.... ,b ...), },, We convert the shape program into code in the order of first the statement type, then its parameters, i.c., [A, 01, 02 43. B, bj. b2, by]. For example, we encode the statement draw("Top' 'Square', P=(-1,0,0), G=(2,5)) in Figure 2 as [3, 0, 0, 2, 5, 0, 0J where 3 represents the command of drawing square top, and the remains are its specific parameters. This way provides perfect precision to encode and decode shape programs while cannot handle continuous parameters. Therefore, we proposed an extra ProgramVQVAR in Appendix B.3 handle the case of the continuous parameters.
Text Encoder & Decoder: We adopt the simplest way to encode and decode text without external pre-trained parameters: leverage BPE (Sennrich et al., 2015) to encode and decode lowercase text.
3.2 ShapeCode Transformer
After developing all the encoders and decoders, turn Point Clouds, Text, and Programs into the unified discrete space Choint Ctezt, and Chrogram, respectively. ShapeCode Transformer performs over a pair of discrete codes and transforms from one type of code to another type. Similar to (Ramesh et al., 2021), we model the pair of discrete codes as a single data stream. For example, if transforming to Point Cloud code from Text code, we model data like left of data (e.g., co, ". N (xpoine)) 1) and use full attention masks (Child et al., 2019) to force ShapeCode Transformer autoregressively predicts the next loken based on all the seen ones, i.e., predict ds with the context of [0. [0...... (0..). Assume {c(²) is the corresponding ground-truth token, our loss for this transformation is min, EN --NNerrNpoto (( cI, , cx-1), o)), where L is cross- cross-entropy loss and T is ShapeCode Transformer parameterized by 0 to predict Cg by feeding (0......--1)
In training, we pad 0 at the
As modeled above, the second type of code gets full attention to the first type of code in predictions. This modeling can be viewed as maximizing the evidence lower bound on the joint likelihood of the distribution over the input two types of code pa (Ctrppe,.Cuuee ), type E {point, tezt, program}, where Pe is our ShapeCode Transformer. However, one the challenges in our modeling is that we can generate either texts or programs conditional on the same input. point clouds. How do we lel the ShapeCode Transformer know which type of object code we need? Our solution is to encode our pairs of tokens with different positional embedding, iee. have two different positional encoding P Polsst ([Cvoint (teatt)) and P Papreurum ([Croint (orogram]). Using different positional encoding helps guide the ShapeCode Transformer to generate the specified target code.
arg min
Σ C(To( Es), Pd()) (Po((C-))) (cce
(1)
ShapeCode Transformer communicates only the shape code with all the encoders and decoders. It maintains E (c) [tkeess, #dim] embedding for each type of code {Croint, Creat, Carogram) that will be optimized during training. In each optimization, once the code is received from the encoder, ShapeCode Transformer
Spadding with the leftmost enables our framework to sample data unconditionally.
5

---

Published in Transactions on Machine Learning Research (03/2023)
indexes the self- celf-maintained embedding through the received code. It transforms shape code based on the indexed self-maintained embedding, enabling optimization without interference from encoder and decoder parameters. Our total training losses are the summation of equation over multiple shape code pairs c',c² from (Creat, Cuoint), (Croint C and (Croint, Curogram) ). For each input shape code, we concatenate its positional encoding Po(e) and the embedding maintained by ShapeCode Transformer E)(c) and feed into the ShapeCode Transfromer to predict the next token. In inference, we leverage Gumble noise (Jang et al., 2016) to generate set of plausible results for given input. Please refer to B.2 for all the detailed parameters.
4 Experiments
We conduct Text = Point Cloud, Point Cloud = Text and Point Cloud = Program tasks to verify our performance, and Partial Point Cloud = Complete Point Cloud (Shape completion) task to show the extensibility of our framework (Appendix 4.4). For each task, we have two versions of our method: Shape Compiler Limited and Shape Compiler. Shape Compiler Limited means that we restrict both PointVQVAE and ShapeCode Transformer to train on the same data and task as the baselines for fair comparisons. Shape Compiler is our full model trained on all tasks and data to check if joint training on heterogeneous data and tasks leads to improvement. Our used data are briefly described below. More data collection details are listed in Appendix A.
To investigate the structural connections between text and shape, we collect a total of 107,371 (Point Cloud, Structure- Related Tezt) pairs from a total of 20,355 shapes with 9.47 words per description on average and 26,776 unique words. We use 85% shapes for training and the remaining 15% for testing. Data collection details are listed in Appendix A.
Shape- Texp Pairs:
3D Shape Assets: To achieve a general framework, we collect various 3D shapes and obtain a total of 140,419 shapes of 144 categorics from ShapeNet (Chang et al., 2015), ABO (Collins ct al., 2021), and Program Synthetic (Tian et al., 2019) datasets. For each shape, we sample 10,000 points as inputs and remove all artificial colors and textures to focus on shape geometry and compositionality.
Shape-Program Pairs: We followed (Tian et al., 2019) and synthesized 120,000 (Point Cloud, Program) pairs. For testing, Tian et al. (2019) sampled shapes from ShapeNet, and we use the same sets to evaluate in our experiments for fair comparisons.
4.1 Text
Point Cloud
Neural Shape Compiler can generate point clouds corresponding to text prompt (Figure 4). We compare methods (Jain et al., 2021; Sanghi et al., 2022) that rely primarily on CLIP Radford et al. (2021) to generate 3D shapes and methods that require 3D-text pairs (Chen al., 2018; Liu et al., 2022b). We use our shape-text training pairs to train CWGAN (Chen et al., 2018) and Shape IMLE diversified model (Liu et al., 2022b), and our Shape Compiler Lim- ited for fair comparisons. Since this paper fo- cuses on geometry, we only compare with (Chen et al., 2018; Liu et al., 2022b) in geometrical as- pects, which is the same comparison way used in (Mittal et al., 2022). We compare CLIP-Forge (Sanghi et al., 2022) and DreamField (Jain et al., 2021) to show the gap between exploiting the 2D-text model and training over 3D-text pairs. Since DreamField learns a neural radiance field over a single text with a lot of sampling, its training time too long to test in the scale of our test set (dctails Appendix B.4). We only compare its qualitative results. In order benchmark Text = Point Cloud task, we turn models which output 3D voxels into point clouds by sampling 2,048 points on their output voxels. Shape
a cheir with emmreets
a classic ottoman
table lamp
a soft pilow
a round table with a single pole support and round base
Figure 4: Tert = Point Cloud by Neural Shape Compiler. A text prompt may translate to multiple shapes.
6

---

Published in Transactions on Machine Learning Research (03/2023)
a table with circular top, four legs connected with each other by slats crossing as X
swivel chair, five wheels, office chair with armrest open frame
open modern style chair, armrests open frame, solid back is slightly curved
Reference Shape Compiler
Shape IMLE CWGAN
CLIP Forge
Dream Field
Figure 5: Text = Point Cloud. Neural Shape Compiler can generate corresponding point clouds to the text prompt with structural details, while the basclines generate inaccurate structures or fewer alignments with the text prompt. We visualize Point Clouds via Misuba (Jakob et al., 2022)
Compiler is trained with all hetcrogencous data and tasks described Scction 4 show the benefits from jointly training over all tasks.
To benchmark Text = Point Cloud task, we measure generative performance from multiple angles: quality, fidelity, and diversity. (1) For measuring quality, use Minimal Matching Distance (MMD) (Achlioptas et al., 2018) to check if the generated shape distributions are close to ground-truth shape distributions by computing the distance between the set of all our generated point clouds and the set of all ground-truth point clouds. We use Chamfer distance for each pair of point clouds in computing MMD. (2) For measuring fidelity, we compute the minimal Chamfer distance between all generated point clouds and the corresponding ground truth shape of the input text. We report the Average Minimal Distance (AMD) over all the test texts. (3) For mcasuring diversity, 2020), which computes the difference between all the generated point clouds of the same text inputs. For each generated point cloud P., we compute its average Chamfer distance CDp, to other k - 1 generated point clouds P/ Pijzss and compute the average: TMD = 1 CD.,, Given an input text, every model finalizes 48 normalized point clouds of 2,048 points.
Evaluation Protocol:
adopt Total Mutual Difference (TMD) (Wu et al.,
Results: According to Table 1, Shape Compiler variants gen- erally outperform CLIP-Forge (Sanghi al., 2022), CWGAN (Chen et al., 2018), and Shape IMLE Diversified (Liu et al., 2022b), especially in AMD (fideiity, which demonstrates our proposed method can generate shapes that more closely match the input texts which contain levels of giometrical details. This consistent with the qualitative results in Figure 5, where all compared baselines cannot: handle the text prompt containing many structural details well. One potential drawback in (Chen et al., 2018; Liu et al., 2022b) is the use of 3D convolutional layers, which make their mod- cls tend to overfit training data distribution. This is also shown in Figure 5, where their results are less correlated with the input texts. Shape IMLE outperforms CWGAN, consistent with the discovery in (Liu al., 2022b). However, inference much slower than Shape Compiler due to the use of 3D convolutional layers and implementations, whereas
Dotaset
Method
MMD
AMD
TMD
CWGAN
22.16
27.32
0.67
Shape IMLE
12.37
14.92
1.83
ShapeGlos
CLIP-Forge
36.43
61.5
2.45
Shape Compiler Limited
6.19
11.27
1.85
Shape Compiler
5.7
7.31
2.8
CWGAN
10.42
18.21
0.79
Shape IMLE
15.34
2.34
Text2shape
CI.IP-Forge
5.16
19.69
1.34
Shape Compiler Limised
6.21
14.21
1.58
Shape Compiler
4.53
11.66
3.07
CWGAN
0.52
Shape IMLE
6.13
13.67
1.42
ABO
CLIP-Forge
7.89
24.15
1.28
Shape Compller Limited
4.93
8.33
0.67
Shape Compiler
4.76
7.77
1.34
Table 1: Tert = Point Cloud. MMD (qual- ity), AMD (fidelity), and TMD (diversity) are multiplied by 103, 10 and 10², respectively.
7

---

Published in Transactions on Machine Learning Research (03/2023)
method infers low-dimensional codes and decodes them through 1D residual decoder. In our same single GPU, Shape IMLE takes 1003.23 seconds to generate 48 shapes, while Shape Compiler takes only 14.69 seconds. Shape Compiler consistently outperforms Shape Compiler Limited indicating the proposed framework benefits from the joint training on all heterogencous data and lasks. However, due to the use of point cloud represenlation, the inherent difficulty of 3D shape generation with structure dctails, and the introduction of Gumbel noise, our framework produces rough boundaries, uneven surfaces, and irrelevant shapes. There is lot of room for research in the proposed framework and Test = Point Cloud task.
Despite the success of using CLIP in 3D shape generation with text prompt containing sim- ple structure descriptions (Sanghi et al.; 2022; Jain al., 2021), CLIP-Forge shows undesirable AMD and MMD scores in both ShapeGlot and ABO datasets, and Dream fails to gener- ate structures corresponding to the text prompt, although they are perceptually fine. Addition- ally, our experimental results suggest that the text embedding of CLIP models (Radford et al., 2021) is Not able to reflect differ- ences in small changes in the text, which may lead to large changes in structure. Ta- ble 2 shows several examples. The texts in the second column are similar to those in the first but will lead to significant structural changes. However, they have very high similarity scores in CLIP text embeddings, almost 1, which is an upper bound for cosine similarity. This may illustrate why CLIP-Forge (Sanghi el al., 2022) achieved low AMD scores (Table 1), where our 3D-text datasets contain complex structural descriptions as shown in Appendix A. CLIP text embeddings cannot distinguish text with struetural details well, which hurts the performance of lext-guided 3D shape generation. The clip model we used here is ViT-B 177-B/33, and other CLIP models show similar trends.
Textl
Text
Similarity
chair with armrests chair with armrests
chair with armrests
0.983
chair without armreest
0.988
swivel chair with ammrests
swivel chair with no Ammrests
0.979
a swivel chair with armrests
swivel chair without armrests
0.986
choir with armrests a chair with armreets
chair with
and carved back
0.962
a chair with no armrests and curved back
0.965
swivel chair with amorests swivel chulr with
and ourved back
swivel cloir with aturesle a swivel choir with no armresta and curved back
0.982
table with cirenier top
table with square top
0:902
a table with civculue top
a table with rectangular lap
0.938
chuir with twn lops
scool chair with thrne
0:902
couch with twu seals
couch with these seals
0.986
Table 2:
The rightmost column is the cosine similarity between the CLIP text embeddings (Radford et al., 2021) f1-f2 of the left two texts. Cosine Similarity = max(II (lll/ll
where fa and fz are the CLIP embodding of Textl and Text2, respectively. is le-6 to avoid division by zero.
Also, we found the cosine similarity between the CLIP embedding of rendered 3D shape images and the CLIP embedding of text cannot accurately measure the degree of alignment between the text prompt and 3D geometry when the text prompt and 3D geometry contain structural details.
plain round surface on top and having framed legs with unique design sliver round sheped table with four rod crossing
a plain rectangular surface on tap and having framed legs with unique design silver rectangle shaped table with four rod crossing legs rectangular table top and legs that shaped
table top legs sheped meet center
center
clip score: 0.318115
clip score: 0.327148
clip_ sccoe:: 0.319092
silprscore: 0.321533
dip_score: 0.328125
clip_score: 0.299561
clip_score 0.324219
clip_score: 0.332764
clip, score: 0.333496
sip_rccae: 0.322266
clip_ score: 0.329580
clip scare: 0.333496
Figure 6: Render the shape with six different camera positions and two different point lights by Pytorch3D Ravi et al. (2020). The texts shown at the top left are the shape-associated texts provided in our datasets, which only contain Round or Circular. We modified them into Rectangle or Rectangular as shown in the top right. The number above each image is the average cosine similarity between each CLIP image embedding and all corresponding CLIP text embedding.
8

---

Published in Transactions on Machine Learning Research (03/2023)
We render 3D shapes into images with six camera positions and two different point lights with PyTorch3D (Ravi et al., 2020), shown in Figure 6. Then, we obtain each image embedding by using the CLIP image branch encoder. Each shape has several text descriptions. We tokenize those texts with the tokenizer provided by CLIP and obtain our text embodding by feeding them into the CLIP text encoder. In Figure 6, the score above each shape is the average cosine similarity between the image and each text shown al the top. We select shapes with text containing Round or Circular and not including Rectangle and Rectangular. We compute the cosine similarity between the embedding of the rendered images and two different sets of text embedding: (1) the original ground-truth texts which contain Round or Circular; (2) we replace Round or Circular into Rectangle and Rectangular. An example of replacement is shown in Figure 6 where the right figure top texts are the replaced ones. We compute the similarity scores over 1,358 shapes with a total of 6,697 texls. As described above, we have six images for each shape, resulting in 40, 182 scores. We calculate the mean and standard deviation of all scores for the two sets of text: (1) 0.2925 + 0.02173 and (2) 0.2917 + 0.02211. Although there are huge differences between the two sets of texts, their average cosine similarity with the CLIP image embedding is very close. This experiment further supports CLIP is NOT suitable to deal with text containing geometric details. 4.2 Point Cloud = Text
the one with 4 legs; the one the back back the back;
circular table. it has a unique look. it has a circular stand;
3DEncoder-LSTM
a table which has a circular top and the base with the legs attached to circular base;
the chair with arms;
three open spaces in the back;
It looks like It is a seat with two slats in the back; 2 horizontal bars has extra bar connecting wire legs;
lags, sits lowest to ground, pyramid around back; one piece legs and weird shape;
back;
has a slatted back;
two horizontal slats, no arms, it has lege;
open back chair has silts not an angle; square with design on back;
a rectangular table with laminated table top and four legs; this four plain table with spindle legs;
the chair ient rounded;
Neural Shape Compiler
Neural Shape Compiler
Figure 7: Point Cloud =
Texl One description per sentence. The shown shapes are from test sets, and
Neural Shape Compiler tell their structures well.
Neural Shape Compiler performs Point Cloud = Text tasks that describe given shape point clouds as shown in Figure 7. To compare, we followed (Chen et al., 2021) and implemented a 3DEncoder-LSTM model as our Aeweedd---eeeeiinnneeeee
baseline by replacing the image encoder of adopt similar encoder structure as PointVQVAB and set the output feature to be 512 dimension, the same value as our intermediate representations. An LSTM model will then be trained over the 512 output features to predict words gradually, and an end word will be the last token. For each shape, all methods will generale 48 descriptions to compare.
et al., 2015) with a 3D encoder. We
We measure the qual- ity of the generated shape descriptions with our annotations in validation sets. Here we adopt the common metries used in text generations tasks (Vinyals et al., 2015; Koncel-Kedziorsk et al., 2019) including BLFUU (4-gram) (Pap- incni et al., 2002), CIDER (Vedantam et al., 2015), and ROUGE (Lin, 2004). We also adopt Dist-1 and D in (Li et al., 2015) to measure the diversity of the generated results: Dist-1 and Dist-2 are the results of distinct unigrams and bigrams divided by the total number of tokens (maximum 100 in our table).
Evaluation Protocol:
Dataset
Method
BLEU-11 1 CIDER
ROUGE 1 Dist-1 1 Diss-2
3DEncoder-LSTM
2.78
2,93
18.61
0.016
0.017
ShapoClot
Shape Compiler Limited
2,96
2.15
21.87
1.174
14.146
Shape Compller
3.19
2.35
22.21
1.193
14.784
Ground-
6.908
S7.92
3DRucodes-SS1
1.01
0.84
21.18
0,127
0.042
Texi2Stape
Shape Compiler Limiled
2.96
2.21
25.97
0.792
9.961
Shape Compiiee
4.25
2.96
27.01
0.921
11.343
Ground-
5.906
$1.05
3DEneoder-LLTTM
0.01
1.39
2.98
0.0G1
0.047
ABO
Shape Compiler Limited
8.95
108.31
42
3.605
23.165
Shape Compiler
7.98
111.22
26.47
1.27
13.43
Ground- Truch
18.294
49,097
Table 3: Point Cloud = Tert BLEU, CIDER, and ROUGE
measure quality. Dist-1 and Dist-2 measure diversity.
Results: According to Dist-1 and Dist-2 in Table 3, Shape Compiler generates much more
9

---

Published in Transactions on Machine Learning Research (03/2023)
Neural Shape Compiler
Neural Program Generator
draweTop'' Rost
drw<Tepp Reci P-0-11 G4PAR)
forjuc, Tranel (1.00..0000
fori-, Tiress un 13)
1- 6-
Chair
drow/Lag, 1out/ 1-)-12,5, www.t-uuu.,..... 0+101.20
drawetaase' Square Puj 10.00 G-
drew/Hari Bar, 'Gutr, 2,06-1122--400000 GHE)
sup, Paj-42-11 G-H4-41
drawd draw Back, /Cubr, Pu G.(12,2 theta.i0))
P= 9-C1ARII
Cabinet
(orl =2, Thore/ un03..00
forfic, Trane, utlgg20110
GHERAUI)
Tromi, LS--/.0) 12w.0.1--1111--1-11111100 1io--ttt--- G-232.10
(urt-a, Trans', un(bC.0)
drow Layer" Pwt4..00=',, G+(LA/0)
Forlic, 'Trarr, U.BL0,110
0-8.7201
Figure 8: Poinl Cloud- = Program. Neural Shape Compiler can well infer programs for the shown shapes of unseen categories and reconstruct shapes in a creative manner (e.g., represent the semicircle leg with bars).
diverse captions than 3DEMcoderLLSTM (Vinyals et al., 2015; Chen et al., 2021) due to the help of involving Gumbel noise over the learned priors in the inference. This is also shown in Figure 7 where Shape Compiler can describe given shapes in multi-angles. Also, our framework achieves generally higher BLEU, CIDER, and ROUGE scores than 3DLMcoder-LSTM. By inspecting the predictions, we found that 3DEncoeerLSTTM tends to predict the frequent words in datasets (e.g,, back and legs), and the output descriptions are less meaningful in semantics. However, those repetítive words can help them achieve higher BLFU,, CIDER and ROUGE scores in Text2Shape and ShapeGlot. Therefore, evaluating Point Cloud = Tezt performance with both quality and diversity metrics is essential. Furthermore, the ground-truth descriptions differ significantly from our predictions on diversity metries, suggesting that the proposed method still has huge gaps in achieving human-level shape captioning.
4.3
Point Cloud = Program
Shape Compiler performs transformation: Point Clouds = Program, which can help us understand how the point clouds are assembled by parts and regularities. For example, in Figure 8, we can tell how to decompose the point clouds into primitives via the symbolic words in the program and also find the mapping between points and the corresponding primitive via executing the program. Furthermore, the symmetry relationship between those racks of the cabinet is implicitly encoded in the FOR statements. The discovery of parts and relationships can help us design robots better interacting with 3D objects.
Neural Program Generator (Tian et al., 2019) is our most direct baseline. They used a two-layered LSTM to gradually generate program blocks and programs inside each block form the final shape of programs. We will not adopt the guided adaptation used in (Tian et al., 2019) in our experiments because it needs extra training loops and prevents the practical use of program generation (further discussion is included in Appendix D). We also compare with CSGNet (Sharma et al., 2018), which applies boolean operations shape primitives and assemble shape recursively.
Metric
Method
Chair
Txble
Bed
Sofa
Cabinet
Bench
Avg
CSGNet (Sharma et al., 2018)
0.365
0.406
Program Generator (Tian et al., 2019)
0.438
0.517
0.254
0.324
0.304
0.216
0.342
loU1 f
Shape Compiler Linited
0.429
0.539
0.27
0.31
0.504
0.314
0.394
Shape Compiler
0.492
0.634
0.252
0.432
0.51
0.451
0.462
CSGNet. (Sharma al., 2018)
7.73
7.24
Progrem Generator (Tian et al., 2019)
1.64
1.97
4.78
3.14
2.95
2.71
2.87
CD 4
Shape Compiler Limited
1.53
1.21
4.51
2.84
1.58
1.57
2.21
Shape Coupiler
1.17
0.83
4.39
2.52
1.49
1.38
1.96
Table 4: Point Cloud - Program. CD multiplied by 10
Avg denotes average number among categorics.
10

---

Published in Transactions on Machine Learning Research (03/2023)
Evaluation Protocol: Shape Program (Tian al., 2019), CSGNet (Sharma et al., 2018), and our models will predict programs under specific domain languages. By executing the output programs, we can obtain volumetric representation shapes; then, we can compute IoU based on the output voxels. Also, we sample points over the surface of output voxels for computing Chamfer distance. Due to the benefits of being a probabilistic method, our framework can predict many programs given a single point cloud. We here generate 48 programs for comparisons, the same number we used our text tasks. Ablation studies of the generated program numbers are included in Appendix D.
Result: According to Table 4, Shape Compiler outperforms all the methods due to the benefits of multimodal learning on all tasks and data. It also suggests our framework can effectively take advantage of task-unrelated heterogeneous data. Figure 8 shows two positive examples of our framework that better handles novel data than the baseline. Both cases show that the compared baseline mainly relies on memorizing the training table data, while Shape Compiler successfully predicts the rough structure and the regularities. Regarding the scores, Shape Compiler Limited achieved comparable performance with Shape Generator, while the significant performance gap in cabinet. and bench calegories shows our framework has stronger generalizability. Due to the limitation of the current program grammar, our method cannot handle complex structures for now. We raisc attention to the next-level shape program, and some further discussion is included in Appendix D.
4.4 Partial Point Cloud
Complete Point Cloud
square back with hole in seat is tall, bar stool with a square top? the legs have pointy legs it has a circle back and seat
Ground-trutt
tall chair with thin legs it has no slats on the back side stool with side curved brass edges tall, skinny, long high legs modern kitchen counter bar stool with no space between back and the legs
Method
Chaic
Aitpline
Car
Point (Yang et al., 2019)
6.95
1.07
3.97
PVD (Zhon et al., 2021)
7.34
1.19
3.83
Shape Compiler Limited
1.82
Shape Compller
6.9
1.58
3.81
Figure 9: The success of shape completion boosts shape structural understandings. We use Neural Shape Compiler to do Point Cloud = Text for both the partial chair and the complete chair. The ground-truth shape is on the top right.
Table 5: Portial Point Cloud = Complete Point Cloud. Numbers are Chamfer distance & with mul- tiplying 102.
In this section, we extend Neural Shape Compiler to conduct Partial Point Cloud = Complete Point Cloud (shape completion) task by projecting both partial and full point clouds into discrete codes and performing conditional generation thereon. We followed the benchmark provided in (Zhang et al., 2018; Zhou al., 2021) and the detailed data preparation is listed in Appendíx A.3. Different from the way used in Zhou et al. (2021), we train models on all training shapes in class-agnostic way. Results in Table 5 show Shape Compiler achieved great performance compared with the previous method. Also, PVD (Zhou et al., 2021) is slower than Shape Compiler because they involve very long steps (1,000) in the diffusion process.
Additionally, with the help of Neural Shape Compiler, we may better share the progress across different tasks. In Figure 9, we individually perform Point Cloud = Texl over the partial chair and the complete chair. The ground-truth chair is on the right. For the partial chair, the captioning results include hallucinated descriptions, like hole in the back and circle back, and also accurate descriptions of the partial chair but not precisely related to the ground-truth one, like poinly legs. Compared to it, the descriptions for the completion result are more related to our ground-truth point cloud.
11

---

Published in Transactions on Machine Learning Research (03/2023)
5 Limitations and Future Works
This paper proposed Neural Shape Compiler, a unified framework to perform multimodal inference and achieved great performance Text = Point Cloud, Point Cloud = Teat, Point Cloud = Program, and Point Cloud Completion tasks. However, there are still a lot of things to be considered:
1. Similar to (Chen et al., 2018; Liu et al., 2022b; Mittal et al., 2022), our method requires large-scale paired data, where the 3D-text pairs are hard to obtain. This situation could be alleviated by the dovelopment of large-scale 3D-lext datasels and recovery of 3D data from 2D images (Gkioxari el al., 2022) and videos (Qian et al., 2022) that contain paired text information.
2. Our framework is extensible, where we showed a case in shape completion task (Table 5). Similarly, we can project images (Van Den Oord et al., 2017), factile information (Gao el. al., 2022), and other modalities into the discrete representation and learn over them via a unified framework for various downstream tasks, such as single image 3D reconstruction (Zhang et al., 2018), multiview reconstruction (Ji et al., 2017) and modti-modal grasping (Calandra et al., 2018). On the other hand, would be interesting to study extending diffusion-based frameworks handle various types of generative modeling, since the current diffusion models usually focus on one type of transformation Rombach et al. (2022); Gong al. (2022).
3. The proposed framework currently camnot well handle out-of-distribution geometry, such as the dragon in (Mor- reale et al., 2022) and the Hilbert eube in (Liu et al., 2020a). Figure 10 shows the results of Point Cloud = Tezt and Point Cloud = Program over the Hilbert cube by Shape Compiler. The current programs can only tell the coarse geometry, and captions are about similar objects in our training data, such as cubes and cabinets.
Captioning
toltable storage cubes (6 high gloss 2 chawer bediidda cabinet large square table single piace
PolntCloud fo Porttooud-to-Pogrrm
deaeflay Tall Hidi D-DBU)
Squaee
Tu]a Trame a-0A
Hilbert Cube
Figure 10:
Point Cloud + Teat and Point Cloud = results over the input Hilbert cube point clouds by Shape Compiler. One description per sentence.
Program
4. Our approach currently adopts the domain-specifie language (DST.) pro- posed in (Tian et al., 2019) one of the input and target shape abstractions. However, the DSL's grammar and synlax are not powerful enough to handle various complex shapes. For example, it fails to perform accu- rate reconstruction or completion for 3D shapes with geometrical details (Chaudhuri et al., 2020) and detect selfssymmetrics possessed by individual parts (Mitra et al., 2006). Since the progress of shape programs is orthogonal to the proposed framework, the development of shape programs can further benefit our framework. Therefore, this paper also raises attention to developing the next level of shape programs.
5. Neural Shape Compiler still generates outputs that are mismatched to the input conditions. For example, it may generate shapes unsatisfied with the input text in Tezt = Point Cloud task and captions wrongly describe input point clouds in Point Cloud- = Text task.
6. Our current framework and data focus on correctly modeling connections between 3D geometry and text. However, less attention has been paid to color, texture, and material, which will be investigated in the future with the support of appropriate data.
7. DALL-E (Ramesh al., 2021) ranks their outputs of (Teal, Image) based on the CLIP embedding similarity scores (Radford al., 2021). However, the experiments conducted in Section 4.1, the CLIP similarity score cannot properly reflect the alignment between 3D shape and text if the text contains geometric details. How to align 3D objects and text is fundamental problem we should study in the future.
8. Neural Shape Compiler currently compiles different shape abstractions. Compared to this, a further direction is to compile operations on one shape abstraction into practical steps in other abstractions and change accordingly. For example, we add "armless" into a text description of chair with two arms. This change be effectively compiled into operations rrmooe two arm ports and their symmelry relationship' in
12

---

Published in Transactions on Machine Learning Research (03/2023)
the shape hierarchy and delete all points belonging to the two arms' in point clouds. We leave the shape editing direction to future study.
6 Conclusion
We proposed Neural Shape Compiler to transform between three shape abstractions: Text, Point Cloud, and Program. With the help of PointVQVABB it achieved great performance in Teat = Point Cloud, Point Cloud = Text, Point Cloud = Program, and PointCloud Complction tasks via a unified and extendable framework. Our experiments show that Neural Shape Compiler benefits from joint training on all heterogeneous data and tasks. Despite showing promises, Shape Compiler has limitations and draws attention to several related directions for future rescarch (Section 5). Furthermore, we study CLIP embeddings in tcxt-guided shape generation and find that is unsuitable for use once the text contains geometric details. We hope that Neural Shape Compiler can serve as effective framework for connecting different shape abstractions, and the studies in this paper facilitate the progress in 3D multimodal learning research.
7 Acknowledgement
This work is supported by fellowship from UMich Rackham Graduate School, two grants from LG Al Research, and Grant No. 1453651 from NSF. We thank Mohamed El Banani and Janpreet Singh for helpful discussions on text-to-3D and the help in proposing the CLIP text similarity experiment. We thank Mohamed Banani for his helpful comments on our writing. We thank Chris Rockwell for the helpful discussion at the early stage of this project, especially in PVán We thank Joyce Zhang and Yutong Tan for parts of the aesthetic design and suggestions. We thank Kaiyi Li for the helpful discussion on compiler concepts and for helping compile several renderers in Apple-M1-Mas chip. Tiange especially thanks Minghua Liu, Zhiwei Jia, Weijian Xu, Xiaoshuai Zhang, Yuheng Zhi, Zhankui He, Zexue He, Kaichun Mo, Eric Yi, and Jiajun Wu for helpful conversations back in 2020.
References
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018.
Panos Achlioptas, Judy Fan, Robert XD Hawkins, Noah D Goodman, and Leonidas Guibas. Shapeglot: Learning language for shape differentiation. arXiv preprint Xix::11905 02925, 2019.
Firikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Tuce Van Gool. Soft-to-hard vector quantization for end-to-end learned compression of images and neural networks. arXiv preprint arXiv: (xXi:1704.00635 3, 2017.
Alfred Aho and Jeffrey Ullman. Abstractions, their algorithms, and their compilers. Communications of the ACM, 65 91, 2022.
Alfred V Aho, Monica S Lam, Ravi Sethi, and Jeffrey D Ullman. Compilers: principles, techniques, & tools. Pearson Education India, 2007.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint vrXi:22204.14190 2022.
Andrew W Appel. Modern compiler implementation C. Cambridge university press, 2004.
Song Bai, Xiang Bai, Zhichao Zhou, Zhaoxiang Zhang, and Longin Jan Latecki. Gift: A real-time and scalable 3d shape search engine. In Proceedings of the IEEE conference on computer vision and poltern recognition, pp. 5023-5032 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. Xiv preprint 1308.33333343 2013.
Dimitri P Bertsekas and David A Castanon. The auction algorithm for the transportation problem. Annals of Operations Research, 20(1):67-96, 1989.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Michael M Bronstein and Iasonas Kokkinos. Scale-invariant heat kernel signatures for non-rigid shape recognition. In 2010 IEEE computer sociely conference on compater vision and pallern recognition, pp. 1704-1711. IEEE, 2010.
Gülen Çagdas, A shape grammar: the language of traditional turkish houses. Environment and Planning B: Planning and Design, 23(4):443 464, 1996.
Ruojin Cai, Guandao Yang, Hadar Averbuch- Elor, Zekun Hao, Serge Belongie, Noah Suavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision, pp. 364-381. Springer, 2020.
Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H Adelson, and Sergey Levine. More than a foeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300 3307, 2018.
Peter Calingaert. Assemblers, compiters, and program translation. WH Freeman & Co., 1979.
Dan Cascaval, Mira Shalah, Phillip Quinn, Rastislav Bodik, Mancesh Agrawala, and Adriana Schulz. Differentiable 3d cad programs for bidirectional editing. In Computer Graphics Forum, volume 41, pp. 309-323, Wiley Online Library, 2022.
Angel Chang, Manolis Savva, and Christopher D Manning, Learning spatial knowledge for text to 3d scene generation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2028 2038, 2014.
Augel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprind arXim:111 151200001,, 2015.
Siddhartha Chaudluri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao Zhang. Learning generative models of 3d structures. In Computer Gruphics Forum, volume 39, pp. 643 666. Wiley Online Library, 2020.
Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian conference on computer vision, pp. 100 116. Springer, 2018
Zhenyu Chen, Ali Gholami, Matthias NieBuer, and Angel X Chang, Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 3193 3203, 2021.
An-Chieh Cheng, Xueting Li, Sifei Liu, Min Sun, and Ming-Hsuan Yang. Autoregressive 3d shape generation via canonical mapping- arXin preprint arXin:2204.0 01955, 2022.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiiv preprint orXiv:1904.105m0 2019.
Adam Coates and Andrew Ng. Selecting receptive fields in deep networks. Advances in neural information processing systems, 24, 2011.
Jasmine Collins, Shubham Goel, Achleshwar Luthra, Leon Xu, Kenan Deng, Xi Zhang, Tomas F Yago Vicente, Himanshu Arora, Thomas Dideriksen, Matthieu Guillaumin, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. arXiv preprint Xiv221 10.06199, 2021.
Theo Deprelle, Thibault Groucix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and malching. Advances in Neural Information Processing Systems, 32, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ar preprint arXiv:1810.04806, 2018.
Haoqiang Fan, Hao Su, and Leonidas J Guibas, A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605 666,, 2017.
Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenu: Group-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE conference on computer vision and pallern recogrition, pp. 2018.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. Shapecrafter: recursive texl-conditioned 3d shape generation model. arXiv preprinl arXio:2207.04440 2022.
Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. arXiv preprint orx:222004 02232,9 2022.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
4857-4866, 2020.
Georgia Gkioxari, Nikhila Ravi, and Justin Johnson. Learning 3d object shape and layout without 3d supervision. In Proocedings of the LEEE/CVF Conference Computer Vision and Pattern Recegnition, pp. 1695-1704, 2022.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text
generation with diffusion models. arXiv preprinl arXiv:2210.06 08933,, 2022.
T Groueix, M Fisher, VG Kim, BC Russell, and M Aubry. Atlasnet: papier-máché mâché approach learning 3d surface generation. arxiv 2018. Xrv preprint or Xrv: 05384. 1802.
Armin Gruen and Devrim Akca. Least squares 3d surface and curve matching. ISPRS Journal of Photogrammetry and Remote Sensing, 59(3):151-177 174, 2005.
Zhizhong Han, Honglei Lu, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen. 3d2seqviews: Aggregating sequential views for 3d global feature learning by cnn with hierarchical attention aggregation. IEFE Transactions on Image Processing, 28(8):3986- -3986-3999, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pp. 1026-1032 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognattin,, pp. 770 778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961 2969, 2017.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industriall-steenth natural language processing in python. 2020.
Shi-Min Hu, Zheng-Ning Liu, Meng- Hao Guo, Jun-Xiong Cai, Jiahui Huang, Tai-Jiang Mu, and Ralph Martin. Subdivision-based mesh convolution networks. ACM Transactions on Grophics (TOG), 41(3):1-16, 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learnóng, pp. 448-456. PMLR, 2015.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint 2xx::21077.1478 1955, 2021.
Ajay Jain, Ben Mildenhall, Jonathan Barron, Pieter Abbeel, and Ben Poole. Zero-shol lext-guided object generation with dream fields. arXiv preprint arXiv:21 12,045,, 2021.
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vieini. Dr.jit: A just-in-time compiler for differentiable rendering. Transactions on Grophics (Proceedings of SIGGRAPH), 41(4), July 2022. doi: 0..1155/3222223353000
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXi::611.00114, 2016.
arXiv preprint
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang, Surfacenet: An eud-to-end 3d neural network for multiview stereopsis. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2307-2315, 2017.
R Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy Mitra, and Daniel Ritchie. Shapeassembly: Learning to generate programs for 3d shape structure synthesis. ACM Transactions on Graphics (TOG), 39(6):1 2020.
15

---

Published in Transactions on Machine Learning Research (03/2023)
It Kenny Jones, David Charalan, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. Shapemod: macro operation discovery for 3d shape programs. ACM Transactions on Graphics (TOG), 40(4):1-16, 2021.
Diederik P Kingmna and Max Welling, Auto-encoding variational bayes. arXiv preprint arXi::3322.6114 2013.
Rik
Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannanch Hajishirzi. Text generation from knowledge graphs with graph transformers. arXiv preprint KKix:1944.02244 2019.
Alon Lahav and Ayellet Tal. Meshwalker: Deep mesh understanding by random walks. ACM Transactions Gruphics (TOG), 39(6):1 13, 2020.
Chris Lattner and Vikram Adve. Llvm: compilation framework for lifelong program analysis & transformation. In
International Sympostum on Code Generation and Optimization, 2004- CGO 2004-, pp- 75 86. IEEE, 2004.
Aoxue Li, Tiange Luo, Zhiwu Lu, Tao Xiang, and Liwei Wang. Large-scale few-shot learning; Knowlodge transfer with class hierarchy. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pp. 7212-7220, 2019.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-prommotng objective function for neural conversation models. arXiv preprint 1rXio:1510 151000355, 2015.
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions Grophics (TOG), 36(4):1-14, 2017.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Y Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arxiw:2211.1 10440, 2022.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74 81, 2004.
Hsueh-Ti Derek Liu, Vladimir G Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. Neural subdivision. arxw preprint axx::2005501811 2020a.
Minghua Liu, La Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11596 11603, 2020b.
Zhengzhe Liu, Peng Dai, Ruihui Lí, Xiaojuan Qi, and Chi- thig Fu. Ias: Image as stetting stone for text-guided 3d shape generation. Xiv preprint or arx:w:209..0 04 145, 2022a.
Zhengzhe Lin, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. arXio preprint arXiv:220.. 14622, 2022b.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32, 2019.
arXiv preprint Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts.
Xrxi::1608. 039883, 2016.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 2837 2845, 2021.
Tiange Luo, Tianle Cai, Mengxiao Zhang, Siyu Chen, Di He, and Liwei Wang. Defective convolutional networks. ar preprint XXXx:1911000432 2019.
Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. Learning to group: A bottom-up framework for 3d part discovery in unseen categories. arXio preprint arXiv:2002.00 064 2020.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances neural information processing systems, 29, 2016.
Daniel Maturana and Sebastian Scherer. Voxuet: A 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 922 928. TEFE, 2015.
16

---

Published in Transactions on Machine Learning Research (03/2023)
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pollern Recograition, pp. 4460 4470, 2019.
Niloy J Mitra and An Nguyen. Estimating surface normals in noisy point cloud data. In Procesdings of the wineleenth anauall symposium on Computational geometry, pp. 322 328, 2003.
Niloy Mitra, Leonidas J Guibas, and Mark Pauly, Partial and approximate symmetry detection for 3d geometry. ACM Transactions on Graphics (TOG), 25(3):560 568, 2006.
Niloy J Mitra, Leonidas Guibas, and Mark Pauly, Symmetrization. ACM Transactions on Gruphics (TOG), 26(3):
63 es, 2007.
Paritosh Mittal, Yhn-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pollern Recogrilion, pp. 306-33 315, 2022.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. arXiv preprint arXi::1908.000 00575, 2019.
Luca Morreale, Noam Aigerman, Paul Guerrero, Vladimir G Kim, and Niloy J Mitra. Neural convolutional surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19333-19342, 2022.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th indernational conference on machine learningg (ICML-11), pp. 689 696, 2011.
Thu Nguyen- Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7588 7597, 2019.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint 2rxi::2212.08751, 2022.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, An- drew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.0 03499,, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Blew: a method for automatic evaluation machine translation. In Proceedings of the 40th annuall meeting of the Association for Compulational Linguistics, pp. 311 318, 2002.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165 174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zemingg Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 2085 2094, 2021.
Mark Pauly, Niloy J Mitra, Johannes Wallner, Helmut Poltmann, and Leonidas J Guibas. Discovering structural regularity in 3d geometry. In ACM SIGGRAPH 2008 popers, pp. 1-11. 2008.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXio preprint 2et.222999.888. 2022.
Charles R Qi, Hao Su, Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 652 660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet+ Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems, pp. 5099-5100000 2017b.
17

---

Published in Transactions on Machine Learning Research (03/2023)
Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F Fouhey. Underslanding 3d object arliculation in internet videos. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition, pp. 1599-1609, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Mochine Learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, arXiv preprint orXiv:8204.06125 2022.
Nikhila Ravi, Jeremy Reizenstein, David Novotay, Taylor Gordon, Wan- Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint orXiv:2007.085) 2020.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advonces in neural information processing systems, 32, 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint 265:2200500615, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 40(2):99-121, 2000.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Chasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXio preprint Xrx:222005. 1 11487, 2022.
Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 693 700. JMLR Workshop and Conference Proceedings, 2010.
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Y Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 18603 18613, 2022.
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.0700 2015.
Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. Csgnet: Neural shape parser for constructive solid geometry. In Proceedings of the TEFE Conference on Computer Vision and Pattern Recognstion, pp. 5515-5523, 2018
Lucas Theis, Wenche Shi, Andrew Cuminggamm, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXin preprint arXiv: 00395, 2017.
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freemann, Joshua B Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. arXio preprint arXie:1901.0285 02875, 2019.
Arash Vahdat and Jan Kautz, Nvae: deep hierarchical variational autoencoder. Advonces Neural Information Processing Systems, 33:19667-19679, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Fspeholt, Oriol Vinyals, Alex Graves et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
18

---

Published in Transactions on Machine Learning Research (03/2023)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference computer vision and pattern recognation, pp. 3156 3164, 2015.
Cheng Wang, Ming Cheng, Ferdous Sohel, Mohammed Bennamoun, and Jonathan Li. Normalnet: A voxel-based for 3d object classification and retrieval. Neurocomputing, 323:139-147, 2019.
Yanzhen Wang, Kai Xu, Jun Li, Hao Zhang, Ariel Shamir, Ligang Lin, Zhiquan Cheng, and Yueshan Xiong. Symmetry hierarchy of man-made objects. In Computer graphics forum, volume 30, pp- 287-296. Wiley Online Library, 2011.
Niklaus Wirth, Niklaus Wirth, Niklaus Wirth, Suisse Informaticien, and Niklaus Wirth. Compiler construction, volume 1. Citeseer, 1996.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d senerative-adversaae modeling. Advances in neural information processing systems, 29, 2016.
Rundi Wu, Xuelin Chen, Yixin Zhuang, and Baoquan Chen. Multimodal shape completion via conditional generative adversarial networks. In European Conference on Computer Vision, pp. 281-296. Springer, 2020.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912 1920, 2015.
Kai Xu, Hao Zhang, Wei Jiang, Ramsay Dyer, Zhiquan Cheng, Ligang Liu, and Baoquan Chen. Multi-scale partial intrinsic symmetry detection. ACM Transactions On Graphics (TOG), 31(6):1-11, 2012.
Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang, Shapeformer: caassoomer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Yu Liu, Serge Belongie, and Bharath Haribaran. Pointflow: 3d point cloud generation with continuous normalizing flows. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4541-4550) 2019.
Alon Zakai. Emscripten: jlvn-to--avascript compiler. In Proceedings of the ACM international conference companioo on Object oriented programming systems languages and applications companion, pp. 301 312, 2011.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. Xix preprint Xrv:221 12.00213, 2021.
Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Josh Tenenbaum, Bill Freeman, and Jiajun Wu. Learning to reconstruct shapes from unseen classes. Advances in neural information processing systems, 31, 2018.
Linqi Zhou, Yilun Du and Jiajun Wu. 3d shape generation and completion through poxet-voxel diffuusion. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5826-5835, 2021.
19

---

Published in Transactions on Machine Learning Research (03/2023)
Appendix
Table of Contents
A Data Collection
20
A.1 3D Shape Assets
20
A.2 Shape-Text Pairs
................................
20
A.3 PartaalShpeeCCompleeshaap Pairs
B Training & Model Details
22
B.1 PointVQVAE
22
B.2 ShapeCode Transformer
23
B.3 ProgramVQVAB
21
B.4 Baselines Implementation Details
24
PointVQVAE Discussions
C.1 Analysis of Learned Codebook
25
C.2 Empirical Studies
26
D Point Cloud =
Program Discussions
27
More Results
A Data Collection
A.1 3D Shape Assets
As mentioned in Section 4, we collect a total of 140,419 shapes from ShapeNet (Chang et al., 2015), ABO dataset (Collins et al., 2021), and Shape Program (Tian et al., 2019). For ShapeNet, we use the most recent release version (ShapeNetCore..22 v2), which contains 52,472 shapes from 55 categories. ABO dataset contains 7,947 shapes from 98 categories and has nine overlapped shape calegories with ShapeNet. We adopt the Chair and 10 Table templates in Shape Program (Tian et al., 2019) to synthesize 40,000 shapes in each category. We do not distinguish shape categories during training for both PointVQVAE and ShapeCode Transformer. We will normallze the input point clouds into unit ball for addressing dataset gaps.
A.2 Shape-Tens Text Pairs
We collect 107,371 (Point Cloud, Structure-Relater Texf) pairs by adjusting and annotate the current datasets, including Text2Shape (Chen et al., 2018), ShapeGlot (Achlioptas et al., 2019), and ABO (Collins et al., 2021) datasets. We take tables from Text2Shape, chairs from ShapeGlot, and all shapes from the ABO dataset. Therefore, our dataset consists of furniture, most which are tables and chairs. We list our considerations and data collection dctails below.
Text2Shape: (Chen et al., 2018) provides descriptions for both chairs and tables. Through careful inspection, found there are large number of descriptions regarding artificial colors (e.g., red, blue), textures (e.g., green grids on the top), and materials (c.g., wooden, steel). Since this work aims to focus on the 3D gcometry and will not predict any colors for output point clouds, we delete those artificial lexture contents but remain the geometrical ones. Also, compared to Text2Shape, ShapeGlot (Achlioptas et al., 2019) provides much more geometrical descriptions for chairs, where the chairs have a very high overlap with chairs in Text2Shape. We thus discard the chair descriptions and only adopt the table descriptions of Text2Shape our data. For a few data which lack structural descriptions, will annotate them manually. Some examples are in Figure 11.
20

---

Published in Transactions on Machine Learning Research (03/2023)
modemm, round shape, table with four lege low table with circular top
round table with four legs
2 tor rectangular table with frame and top and shalf two layer table, rectangle shape table rectangule shape. have footrest which used for placing things lags
low heighten round table with three legs shape
table
lege
coffee table roctangulat
e-shaped striped outsidee
oval with two curvy legs nalled to a square base the top is sitiigg on two pieces shaped like
round table has a strong round log round top poodle leg turring table table with circular shape, has large leg
office table consiats of tables put together
enclosing space the middle
table, office table, drawer are include and old- fashioned
table
single pillar legged shaped
shaped
zig leg, small rectangular stand
drawers hendles on each side.
Figure 11: Random examples in adjusted Text2Shape dataset. One deeccrptton per sentence.
ShapeGlot: (Achlioptas et al., 2019) provides many meaningful structural descriptions for chairs. However, the descriptions are about triple relationships. It provided users with two shapes and comparative description and asked users to choose which shape most satisfied the description, i.e. (Shape A, Shape B, Tenl). We transfer the triples into doubles by assigning the text to the target shape according to annotations. However, some texts lose their meanings if they do not have the control shape. Furthermore, we found single text ShapeGlot usually not very informative that can plot the target shape accurately. In the end, we filter out the less-meaningful texts and concatenate several shapes' texts together superlative words, like longest, because they usually cannot hold across the entire dataset. For few data which lack structural descriptions, we will annotate them manually. Some examples are in Figure 12.
description. Also, remove
looks like a chair that would rec tig cushy chair eits on ground
the chair that has thick vertical slats hass five veriice burss the seat
no lege wheels, karthoy type, thick overshufled
Full chair with square back, This chair
Square
Email back sofa, the buildistt chair,
soat and back variccll slats, flat arms slatted back outside chaic vertical slats
vortical sitt on back, and back made vertical slats
this cheir has back and seeil made of fet vertical burs
the arma do not appear munded
loga, hole the back 4 logo and opon back, thin logs dook chair
back rezt sepersted from the seet
looks like seet in public school with the
weighs the most, arms
looks like barbers chair. square bottom the oree with one single leg to the floor the with ammeetss back, one aquare on Resembtes salon with square
rounded anm rests, but with only one chair pad rounded amms, not entra elits on legs curved curved backs, no extrn brazing on legs Rounded armeett chair support connected to logs, top, normal skinny logo.
"all basckside with lines going up, the one with
the slits going up and down
cirouler
high back and it's curved the top wood panels
arma top,
loga
has no arms vericcal planks
Figure 12: Random examples in adjusted ShapeGlot datasct. One description per sentence.
21

---

Published in Transactions on Machine Learning Research (03/2023)
(Collins et al., 2021) provides high-quality CAD models and descriptions in different languages for amazon products. We only use the description of the 'item name" under the language tag 'en US: To add structure-related texts to ABO shapes, we collect more descriptions by manually annotating. Also, since ABO consists of amazon products, some original descriptions contain brand names, such as "amazon basics" and 'ravenna', which are not. connected lo shape geomelry. Also, some descriptions contain product dimensions like 16.0 8.2 x 7.4. Since we normalize all point clouds into a unit ball, those dimensions are not connected to its scale. Therefore, we deleted the descriptions related to brand names and dimensions in the descriptions. Some examples are in Figure 13.
ABO:
whipstitch edge decorative throw pilow
midcentury stark 6 drawer dresser
dumbbell hand weights
classic oversized ottoman puff stool cushion, supported 4 legs
dual monitor stand height adjustable arm mount
living room chair with curved arnrests
Figure 13: Random examples in adjusted ABO dataset. One description per sentence.
A.3
Pairs
We used the data from GenRe (Zhang el al., 2018) which has 20 random view renderings of airplanes, cars, and chairs from ShapeNet, for a total of 20 x 9,646 training (point-clouds, point-clouds) pairs and 20 x 1,382 test (point-clouds, point-clouds) pairs. We sample 200 points from the above depth images as inputs, sample 2048 points from the corresponding 3D shapes as targets, and evaluate over all the provided 20 views. Unlike previous benchmarks (Zhou et al., 2021), which train model for single shape category, train every completion model with all training data.
B Training & Model Details
This section provides the training and model details for our Poin/VQVAF and ShapeCode Transformer used in our main paper, and the Program@QVAL we mentioned in Section 3.1. We implement and train our models with Porch (Paszke et al., 2019) and A40 GPUs.
B.1 PointVQVAE
For PointVQVAE, we adopt the cosine annealing strategy of SGDR. (Loshchilov & Hutter, 2016) (w/o restarts) as our optimization scheduler and set the initial learning rate 0.002 with 8 x 32 batch size and 200 epochs.
PointVQVAE encoder has two hierarchical layers (#T = 2). In the first layer, we sample 512 center points.
Then, we gather information from 64 points within Ry, = 0.1 ball around each center point with two fully connected layers [64, 512] and [512, 512]. After the first layer, our point cloud size will be reduced 512 from 10,000 (original input size). Similarly, in the second layer, we sample 128 center points, and do R7, 0.4
22

---

Published in Transactions on Machine Learning Research (03/2023)
ball query with 512 points. We also adopt two fully connected layers here with shapes [512, 512]. After the encoding phase, we will have 128 embeddings and look for the closest embedding in the codebook that has the size of [512, 512]. We concatenated the indexed embedding to form a [128, 512] embedding and sent it to the decoder. The structure of PointVQVAE decoder is shown in Figure 2, we have batch norm ID (Toffe & Szegedy, 2015) and ReLU (He et. al., 2015) following each convolutional layer. The output of the residual-like structure (Figure 2) has the shape Batch Size, 512, 128). We then applied a max-pool layer to the last dim of the output to obtain an embedding with shape [#Batch Size, 512, 1]. The max-pool layer can eliminate the effect caused by the concatenation order of the codebook embedding, similar to the use of max-pool layers in PointNet (Qi et al., 2017a). After the max-pool layer, we finally have two Convolutional 1D layers with 5122, 1024] and [1024, 2048 * 3] to predict 2,048 points. Batch norm 1D and ReLU are also used between the last two layers.
We use Chamfer Distance (CD) and Earth Mover's Distance (EMD) as the reconstruction loss to serve as one of the objectives of PointVQVAE. Specifically, we adopt the implementation of CD in Pytorch3D (Ravi et al., 2020) and implemented an EMD approxination via auction algorithm (Bertsekas & Castanon, 1989) with the help of codes provided in (Fan et al., 2017; Liu et al., 2020b). The EMD codes will be released with detailed report.
DEMD(PI,PE))
min
Σ sEPi
|| - (2x))2, & is a bijection
(2)
d:P, P
Dc min MEPs
= Σ
2 y112 +
Σ
min ||
y1l
(3)
ZEP
zEP.
PEP
(4)
The overall objective for optimizing PointVQVAE adopted the one used in (Van Den Oord et al., 2017) and is shown below, where sg represents stop gradient operations, Ze (p) refers to the output of our encoder, e is the nearest embedding in our codebook with the encodor output. The first two terms are the reconstruction loss to push the final output of Point VQVAE closer to input point clouds. The third term is Le loss to move the embedding in our codebook closer to the corresponding encoder output. The fourth one is the commitment loss to constrain those embodding growing.
Liotal = =DEND + +DDD + [|g][2.()] o + 11) - -ggeel1
B.2 ShapeCode Transformer
We also adopt the cosine annealing strategy as our optimization scheduler and set the initial learning rate 0.001 with x 24 batch size and 100 epochs. For generality, we adopt the simplest Transformer (Vaswani et al., 2017) with depth 5, wide 64, and 8 attention heads. We integrate full attention masks (Child et al., 2019) to force it to do autoregressively predictions (Van den Oord et al., 2016).
As mentioned in Section 3.2, we have different positional embedding for each type of data pair (i.e., task), which will be optimized during the training. Since we pad 0 at the leftmost of our data, the positional embedding for the conditional code will have an extra token. For example, we conduct (Teat, Point Cloud) transformation, we will have Ptent = 256 + 257 positional tokens for Text and Ppoint = 128 for Point Cloud. Therefore, we have a total of 257 + 128 385 positional tokens for (Terf, Point Cloud). For cach positional embedding, we have the embedding dimensional 512. As result, we have embedding size [385, 512) for the task (Text, Point Cloud). For each task, Neural Shape Compiler has different positional embeddings but the same process described above.
Also, ShapeCode Transformer has self-maintained embedding matrix for each type of code, as illustrated in Section 3.2. We also use (Text, Point Cloud) transformation as example, where Text code has Ntezt = 256 tokens and Point Cloud code has N Npoint 128. The embodding dimension is 512. Therefore, we have embedding matrix with size [Niest, 512), [Npoint 512], and [Nprogram: 512] for Text, Point Cloud, and Program, respectively. Regarding token length and vocabulary size for each shape abstraction, Croint has
23

---

Published in Transactions on Machine Learning Research (03/2023)
Np Npoint = 128 tokens with a vocabulary size of 512 via argmaxing from the PointVQVAE encoder output; Cleat has Ntext = 256 tokens of vocabulary size 49, 408, we pad between the last valid text token and the beginning of another token with a value between [49408, 49408 + 256]; Cerogram has N Nprogram = 240 tokens and a vocabulary size of 78.
The overall objective of ShapeCode Transformer is the summation of loss computing over each pair of data: (1) (Text, Point Cloud); (2) (Point Cloud, Text); (3) (Point Cloud, ShapeProgram); (4) (Partial PointCloud, Complete PointCloud). For each pair, we define the loss over every token with Cross-Entropy loss as shown below, where L indicates cross-entropy. Noting that, for program codes Curogram, we will turn negative integers into positive integers, like mapping [-20, 20) to [0, 40).
Σ f(ck 100,- .. .k-1))
B.3 ProgramVQVAE
Currently, Shape Programs (Tian et al., 2019) only has limited discrete parameters, and thus, we adopted the way of discretizing parameters into codes in our main paper. For example, we will map integers in [-20, 20) to [0, 40] to assign unique positive integers for each parameter. Although the current shape program can already cover a lot of 3D shapes, it may be upgraded to encode more detailed structures of 3D shapes that need to involve continuous parameters. For example, the statement Araw("Tpp' 'Square', P=(-1,,,0), G= in Figure 1 may have decimal parameters draw("Top', 'Square', P=-(11.2000..3 G=(2,5.7)). Furthermore, the primitive basis used in the shape program may be replaced with a functional basis (Mescheder et al., 2019; Genova et al., 2020). As mentioned in Section 3.1, we also designed VQVAE to transform shape programs into codes to fill the gap in handling continuous paramcters.
The design of Program VQVAE is pretty similar to our Poin&VQVAE, where we leverage the most basic concepts of vector quantization and adopt the straight-througe gradient estimator and the vanilla codebook objective used in (Van Den Oord et al., 2017) for optimizing. Given a statement, we will turn its statement type into a one-hot vector and concatenate it with its parameters as the input to Progra/QVAA For example, we will turn drop' 'Square', P=(-1,0.0), G=(2,5) into [3. 13,0 0, 2, 5, 0, 0] where the value is the represents the statement type 'draw'. The encoder and decoder will be two vanilla Transformer (Vaswani al., 2017) with depth 3 and MLP dimension 32, and the codebook is [512, 512], the same size as we used in PointVQVAL We use cross-entropy to compute the loss for predicting statement types and L2 loss for regressing statement parameters.
We synthesize 200,000 (Point Cloud, Program) pairs for training the VQVAE. On 10,000 validation data, it achieved 100% accuracy in predicting statement types and < 0.01 L2 distance in parameter regression. The results show the proposed PQVEE can well learn the distribution of our synthesized shape programs.
B.4 Baselines Implementation Details
This paper compared various baselines across different tasks to examine the performance of the proposed framework. We generally follow their provided codes and default configurations for each baseline, including training epochs, learning rates, and other hyper-parametors. We only modify the code when changes are needed to use our dataset. Some modifications are listed in this section.
Shape IMLE (Liu et al., 2022b) preprocesses all texts with Bert (Devlin et al., 2018) in its framework. We adopted the same way and used Bokeniier of the pre-trained flag -bert-base-uncasedd process all the texts in our dataset.
We adopted clip. tokenize and elip. eapppeecod__eete text to encode all test texts and input them into CLIP-Forge (Sanghi et al., 2022) for quantitative comparisons. We input the text prompt to DrcamField (Jain et al., 2021) by the provided command line in Gihub of Jain et al. (2021) for qualitative comparisons. In our environment, we need GPUs of 2048 Gb memory to run single text input with Dream
24

---

Published in Transactions on Machine Learning Research (03/2023)
(Jain et al., 2021). It takes more than eight days to finish the default 10,000 iterations for single text input. Therefore, it is impractical to test DreamField in our test set, which consists of 15,000 text-shape pairs.
CWGAN (Chen et al., 2018) provided processed data as templates. We follow their data formats to process our data and update each entry, including caption apption_tuples caption matches vocab size, cwwppcapp_eeeetee
and dataset_ size. We adopted spaCy (Honnibal et al., 2020) to tokenize texts as mentioned in Chen et al. (2018). (Chen et al., 2018) simply set the token number in a linear increase rule based on their data order, while we set our token number basod on our data order. We also disregarded descriptions with more than 96 tokens (Chen et al., 2018) did.
For AutoSDF (Mittal et al., 2022), their current codebase did not support training generation models conditioned in texts; we do not compare it for now. We will try to add their results once the code is updated.
For the point cloud completion baseline PVD (Zhou al., 2021), we jointly train their models with three data classes. This differs from their original paper, which trains separate models for each shape class.
C PointVQVAE Discussions
C.1 Analysis of Learned Codebook
This section provides some preliminary analysis of the learned codebook of PointVQVAE. The codebook is the interface to connect the output of PointVQVAE encoder and the input of PointVQVAE decoder. In our experiments, the codebook size is [512, 512], and we have 128 embedding for single input point cloud.
Figure 14: The right three shapes are reconstruction results of random codes and the left two shapes are reconstruction results of indexed codes illustrated in the text.
For each input point cloud, we will have 128 codes to index the embedding and feod it into the decoder later. We look at how many different embeddings of the codebook will be used for a single input point cloud. We compute the number with whole ShapeNet (Chang et al., 2015) shapes, where we auto-encode shape and compute the average ratio of unique codes. The result 44.28, which indicates point cloud code Cpoint will have multiple replicate code numbers. Does this indicate that the codebook learns the basic primitives (Deprelle al., 2019), selects the suitable ones for specific point cloud, then fuses them for reconstruction? If so, will those primitives be perceptually understandable to humans? To study this, we design indexed codes with all one nunber 2x,,.... 2z), like 1, 1I, length = 128. Those codes visualize the codebook's 's #o embedding. We feed the special codes into our decoder to reconstruct the corresponding embedding and show two examples at the left of Figure 14. Results show the embedding is pretty random to humans and may exist in certain symmetry structures. Since the reconstruction results are highly correlated to the decoder weights, the way we adopted above may not be the ideal solution. We will try to develop other met hods to understand how the 3D codebook works in the future. Besides, we also tried to randomly generate the codes and reconstruct them to see if the random samples from the latent discrcte space are meaningful. Results are shown in the right of Figure 14, where they are somewhat reasonable in the structure. We also show an example in Figure 15 where we interpolate two different shape codes and reconstruct the intermediate results.
25

---

Published in Transactions on Machine Learning Research (03/2023)
Interpolation
Source
Source 2
Figure 15: A random example of interpolating latent codes between two resource shapes of our proposed PointVQVAE.
C.2 Empirical Studies
In this section, we conduct empirical studies about PointVQVAE, including ablation studies and comparisons. We changed the number of points in the last layer of the encoder (the color points in Figure 3) from 1 to 128. When the point number equals 1, our encoder equals the standard encoder of PointNet++ (Qi et al., 2017b) which uses a global pooling layer. Table left shows such a design brings a catastrophic performance drop due to it being significantly difficult for the codebook to encode holistic point clouds instead of parts of the point clouds.
Variants
CD |
Variants
CD 4
PointVQVAE globalpool
16.63
[RT, = 0.1, RT. = 0.2
13.356
PointVQVAE final-16
10.26
[R,, -0.1, RT, = 0.4
6.877
PointVQVAE final-32
8.57
[Rr, =0.1, RT; = =0.6
8.283
Poin/VQVAF final-64
7.34
Rn 0.05, Rı; 0.1, Rıs 0.6]
6.513
Point VVVE final-128
6.93
RT 0.05, Rr; = 0.2, Rrg 0.6
7.808
POEE final-256
6.78
Table 6: Ablation studies over POVE. CD is multiplied by 10a. Left: X in PointVQVAE final-X means how many points sampled in the last layer of the encoder. "globalpool" indicates only having one point with the global receptive field. Right: studies the choices of Ry, and the number of downsampling processes in PointVQVAE (Section 3.1).
We also perform some ablation studies over the number of downsampling processes and choices of Rr; Section 3.1. We've tested several ablations about Rr, as shown in the right of Table 6, including both two and three times downsampling processes. In experiments, we chose [Ro, 0.1, Ry; 0.4] due to its good performance and light weight.
Besides, we compare with some point clouds based auto-encoders (Achlioptas et al., 2018; Yang et al., 2019; Luo & Hu, 2021; Cheng et al., 2022). For each shape, we sample 2048 points thereon and use Chamfer Distance (CD) to measure the difference between inputs and reconstruction results. We will report Chamfer Distance (cn(Pi,P)) as the summation of both two-direction P1 > P2 and P2- > Pt. EMD is not reported, because each method adopted the EMD implementation proposed in (Fan et al., 2017), which cannot guarantee an approximation of Earth mover's distance. In contrast, the used CD implementations are correct. The results are shown in Table The numbers under 'Oracle" mcan the lower bound of reconstruction, obtained by computing the distance between two different samplings from the same shape meshes. Compared to Poin/VQVAF Limited, Point achieved
Methors
CD L
I-GAN (Achlioptas et al. 2018)
9.23
Point Flow (Yang al., 2019)
7.76
ShapeGF (Cal et al., 2020)
5.97
DiffGen [Eao & Ho, 2021)
5.62
PointVQVAE Limited
PoldVQVAE
5.98
Oracle
3.09
Table 7: Autoencoding performance. CD is multiplied by 103.
26

---

Published in Transactions on Machine Learning Research (03/2023)
higher scores which mean training more 3D data help PointVQVAE model the 3D shape distribution and also not reduce its performance on the part of the training data.
Ground- Truth
PointVQVAE
Figure 16: Random reconstruction examples of Point PointVQVAL VQVAE.
D Point Cloud = Program Discussions
In this section, we mainly discuss two points around Point Cloud = Program: (1) the generated sample numbers used in our shape program experiments; (2) the guided adaptation used in (Tian et al., 2019).
choir
table
bed
sofa
cabinet
0.5
0.3
0.1
16
128
256
sample
Figure 17: Poinl Cloud
Program performance versus the shape generated sample numbers.
We adopt 48 as the sample number in Point Cloud = Program experiments because all other tasks adopt batch size 48. Here, we provide more thorough study of the sample nunbers. We test Shape Compiler performance in all six shape categories with sample numbers [1, 2, 4, 8, 16, 48, 96, 128, 256]. The results are shown in Figure 17. The results show that the proposed method achieved higher performance by incrcasing the generated sample numbers. This is also one of the benefits of the generation framework we adopted in our approach.
Shape Program (Tian et al., 2019) used guided adaptation finetuning their models to new test data. They achieved higher performance by using guided adaptation. However, in our experiments, we intended not to follow (Tian et al., 2019) perform guided adaptation. We choose this way because aim to
27

---

Published in Transactions on Machine Learning Research (03/2023)
take step further in shaping program generation for practical uses. In real-world applications, usually cannot, afford the extra training time for conducting 'guided adaptation"; sometimes, we even do not have enough computation to run adaptation (e.g,, edge computing). Also, generating shape programs without "guided adaptation' can enable us to use one class-agnostic model to process shapes from different categories instead of having class-specific models for each class. The proposed framework, Shape Compiler, only has one class-agnostic model for processing different shape categories. However, Tian et al. (2019) will neod train six different models to process the test shape if they guided adaptation. Combining the advantages of time-saving and class-agnostic, PointCloud-to-PPe to-Program could take step toward achieving real-world impact.
E
More Results
In this section, provide more generated samples by the proposed Neural Shape Compiler. Extra results are contained in the supplementary.
an shape legs croesed with each other
serving tray
Figure 18: Generated shapes by Neural Shape Compiler given different text prompts. Shape Compiler can generate multiple shapes given a text query.
a queen bed
chandelier
without back
a rug
Figure 19: Generated shapes by Neural Shape Compiler given different text prompts. Shape Compiler can generate multiple shapes given a text query.
28

---

Published in Transactions on Machine Learning Research (03/2023)
long tv table along with three rounded edge and off corners; very long table, looks like an antique anyone would long rectangular;
has one separate seat with legs and gap between the legs;
workout dumbbell hand weights;
legs, gap between the slat legs and back;
bishap modern king bed; tisbury nailhead trim queen bed;
geometric ceramic planter; rustic stoneware crosshatch indoor outdoor flower plant pot:
rectangular tv table with storage on both sides and other side bottom:
Figure 20: Point Cloud = Text results by Neural Shape Compiler given different point clouds. Shape Compiler can generate multiple captions given a point cloud query. One description per sentence.
Shape Compiler
a chair
CLiP-Forge
"scaair
Figure 21: Directly comparing the performance of Neural Shape Compiler with CLIP-Forge (Sanghi et al., 2022) results a simple text prompt.
Shape Compiler
square
CLIP-Forg Forge
Figure 22: Directly comparing the performance of Neural Shape Compiler with CLIP-Forge (Sanghi al.,
2022) results on a simple text prompt.
29

---

Published in Transactions on Machine Learning Research (03/2023)
round centre table which does not have any leg but circular support; special circle of half circles on both sides no legs:
Captioning
draw(Top): 'Groole' P-(1.0.0), G-B,111]
PointCloud-to-Program
old fashion table with drawer and unique design; an old fashion table with drawer beneath the table inserts;
Captioning
draw[Top', P+RAAAL G-(1.8.11)
Dointlluudtto-Prorrae
forl-22 'Trara' un18.0.3]
drow(Sideboarc,, CCub: Pa44A-7H4% Ga(11A8)
Figure 23: Point Cloud - Text and Point Cloud input point clouds. One description per sentence.
Program results by Neural Shape Compiler given the
30


Reviews:
[{"review_id": "WCAnTtWKRo", "reviewer": "Reviewer_qL43", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n+ The paper clearly argues a core hypothesis, namely that training a joint encoder-decoder architecture across different modalities improves on shape reconstruction & generation tasks.\n+ The model is well-motivated and well-crafted; The use of a vector-quantized latent space makes sense, the architecture of the probabilistic ShapeTransformer makes sense.\n+ Results are convincing, with cool generations and a clear improvement over baselines. \n+ I find the investigation into the properties of CLIP embeddings in Table 2 well-motivated and insightful.\n+ Great related work section.\n+ I find the discussion of limitations great. It is absolutely expected that the proposed method cannot appropriately describe the hilbert cube, but good to see and to introduce as a reasonable task. The piontcloud-2-program result is still convincing.\n+ Very thorough evaluation.\n\n## Weaknesses\n- I find that the grammar is a bit lacking from time to time - not a huge deal, but readibility could be improved.\n- I find that the notion of \"unified latent space\" is extremely confusing, please see \"Requested Changes\".\n- I find the structure of section 3.1 a bit lacking. I think it would be clearer to move some of the details on the point cloud decoder into this section, and then to have one paragraph per encoder-decoder type. \n\nAll in all, I believe that this paper is a valuable contribution to the field and a valuable reference. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a multi-modal auto-encoder model for 3D shapes.\n\nSpecifically, the authors propose a set of VQ-encoders to encode point clouds, programs, and text descriptions into three different kinds of VQ shape codes. \n\nThe authors then propose a ShapeTransformer to transform one kind of latent code into another kind of latent code.\n\nThree different decoders then decode the domain-specific latent codes into the appropriate representation.\n\nThe model is trained end-to-end, and the authors demonstrate not only that training on multi-modal data improves performance over training a single-domain auto-encoder, but also demonstrate significant qualitative and quantitative gains over a variety of baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## \"Unified\" shape codes\nOn the one hand side, the paper again and again highlights \"unified shape codes\", suggesting that one shape code can be decoded into all three representations. It turns out that this is *not* what unified means - in fact, I'm very confused about what \"unified\" is supposed to mean here. Instead, the paper talks about the \"ShapeCode Transformer\" as the model that *converts* one shape code into another. This is already confusing in the abstract, and then, Fig. 2 is confusing yet again: The different front-ends are all encoded into a \"ShapeCode\", but then, the \"ShapeCode Transformer\" outputs yet another \"ShapeCode\". Both the first and the second ShapeCode in Fig. 2 are suggested to be domain-agnostic (i.e., the same ShapeCode is used for all of Text, Point Cloud, and Program), but the text suggests \"ShapeCode Transformer ... transforms shape codes of one type into another\". Later, in 3.2, it's clarified that the ShapeCodes *are* different across different representations. \n\n1. The use of \"unified\" should be removed.\n2. Fig. 2 should visualize the three different kinds of shape codes, and the transformer should translate between them.\n\n## Bold numbers in tables\nIn table 1, 3, the best-performing method should be bolded.\n\n## Misc\nPage 2: permutation-equivalent --> permutation equivariant?\n4.1. \"Tons of sampling\" --> \"A lot of sampling\".", "broader_impact_concerns": "No concerns on ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XYNA0YtMWw", "reviewer": "Reviewer_eY29", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have to clarify that this paper is completely out of my expertise. I never published any paper in the 3D vision domain.\n\n## Strengths:\nThe work proposed a versatile framework to transform between point cloud, text, and programs. The authors conducted comprehensive experiments with clearly elaborated settings and comparisons.\n\n## Weaknesses\n1. Any quantitive ablation study on the number of downsampling processes and choices of $R_{T_i}$ in section 3.1?\n2. Are all results in experiments compared with fair model sizes and computation costs (FLOPs)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper builds a unified framework, called Neural Shape Compiler, to transform between shape abstractions: Text, Point Cloud, and Programs.\nThis framework includes a PointVQVAE (as the encoder and the decoder), and also a ShapeCode Transformer.\nOn diverse benchmarks, including Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, the proposed Neural Shape Compiler shows better performances.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could the authors more clearly elaborate on:\n1. The motivations and the unique contributions of your model design (PointVQVAE and ShapeCode Transformer)? Especially, compare with previous works: VQVAE, PointNet++, and Ramesh et al., 2021.\n2. Why do the authors follow the use of Chamfer distance (Fan et al., 2017) and Earth Mover’s distance (Rubner et al., 2000)?\n\nI believe Section 3 could be better motivated and organized, so that readers could follow the motivation why the authors choose these options.", "broader_impact_concerns": "To better show the broader impact of this work, the authors could consider explaining more on: 1) why it is necessary to build a unified framework to translate between pairs of shape abstractions; 2) why previous work failed in this unified framework.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tq6g7Zs80E", "reviewer": "Reviewer_basX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this work is that it presents a straightforward and reasonable model for multi-modal inference on 3D shape data. It successfully shows the benefit of training on multi-modal data, which is not always as easy as it sounds. It also presents some interesting analysis of CLIP embeddings, and suggests plausible reasons for why CLIP-based methods fail to produce good fine-grained shape estimation (because the CLIP embeddings do not reflect fine-grained shape information).\n\n\n\n\n\nThe writing of the paper is a weak point. It sometimes uses informal or imprecise language, which often makes things ambiguous, and even confusing (at least for me). Besides typos and bad sentences, this kind of writing is problematic when it suggests something novel is happening when it's not. The first example of this is the name of the paper: \"Neural Shape Compiler\". There is no compiler here. The paper seems to argue that LLVM (Lattner & Adve, 2004) can be interpreted as an autoencoder, and therefore autoenders can be interpreted as compilers. This does not work for me. The LLVM paper never mentions autoencoders, and never even mentions decoders. The current work presents a fairly standard multi-modal autoencoder, where the multiple modes are sent to a shared latent space, and specialized decoders translate to target modes. The \"compiled shape codes\" simply refer to the output of the Transformer in the middle of the model. Much better references exist than LLVM for motivating this approach -- and several are even cited in the \"Multimodal Learning\" section. It feels like \"compiler\" is used to differentiate the approach from related work, while in fact this particular differentiation does not exist.\n\n\n\nHere are a few more examples of ambiguous or imprecise writing.\n\n\n\n\"ShapeCode Transformer communicates with all encoders and decoders via discrete code and handles all types of transformations\"\n\n\n\nThis does not seem to be true. For example it doesn't seem like any geometric transformations are handled. (How about affine? How about translating the whole object left or right?)\n\n\n\n\"In each turn T_i, we first do farthest point sampling over input point clouds\"\n\n\n\nIt is unclear what is meant by a \"turn\" here, and it seems turns are never mentioned again.\n\n\n\n\"This paper studied how to translate between Text ⇐⇒ Point Cloud ⇐⇒ Program.\"\n\nThe citations preceding this quote do not do this, so I'm guessing this is supposed to refer to the current paper (\"studied\" should be \"studies\"?).\n\n\n\n\"perform compilation between the three shape abstractions with neural networks (Rumelhart et al., 1986)\"\n\n\n\nThis reference seems to be mixed up. That Rumelhart paper does not involve any compilation or shape abstractions.\n\n\n\nThe paper repeatedly says it does not use positional encodings: \"The decoder then combines all the 3D part codes to reconstruct whole point clouds in a permutation-equivalent manner.\" ... \"we discard the position / coordinate information of the final points of the encoder\" ... \"we do not have any positional information within the source codes\"\n\nThen the paper says it uses two types of positional encodings: \"Our solution is to encode our pairs of tokens with different positional embedding, i.e., we have two different positional encoding Pφtext point ([Cpoint, Ctext]) and Pφprogram point ([Cpoint, Cprogram]).\"\n\nWhat exactly is happening here? \n\n\n\nIt seems that an important technical distinction between this work and other multi-modal autoencoders/translators is the use of a transformer at the bottleneck. The details of this module are unclear to me. What exactly are the tokens? It seems like the input and output is treated as a long sequence, and the data itself has no positional embeddings, so how does the model know when the \"pointcloud\" part of the sequence should end and the \"text\" part should begin? How many tokens are there per modality -- is it a fixed number, or does it depend on the size of the raw data?\n\n\n\n\"In training, we pad 0 at the left of data (e.g., [0, ct1 , · · · , ctNtext , · · · , cpNpoint ]) and use full attention masks\"\n\n\n\nWhat does a single zero contribute here? Why is this done, and why is it important?\n\n\n\n\"we turn shape programs into codes with statement types in the first place\"\n\n\n\nWhat does \"in the first place\" mean here? (Is this a specific kind of statement type?)\n\n\n\n\"Since DreamField learns a neural radiance field over a single text with tons of sampling, its training time is too long to test in the scale of our test set\"\n\n\n\nWhat is \"tons of sampling\"? What is \"too long\" exactly?\n\n\n\n\"Table 2\" is interesting but is never referenced in the text.\n\n\n\n\"Our current framework presents some positive signals\"\n\n\n\nWhat exactly are positive vs negative \"signals\" here?\n\n\n\n\"This situation could be alleviated by the proposed of large-scale 3D dataset\"\n\n\n\nUp until this part of the paper, it did not sound like a dataset was a contribution. Maybe this sentence is from an older draft, which included a dataset contribution?\n\n\n\n\"Our current framework and data focus on correctly modeling connections between 3D geometry and text. Nevertheless, less attention has been paid to colors, textures, and materials\"\n\n\n\nI don't understand the use of \"nevertheless\" here. It seems like \"and\" would work better, which is almost opposite in meaning.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a multimodal autoencoder for text, pointclouds, and shape-generating programs. The main idea is to encode text, pointclouds, and programs into a shared latent space, and then use domain-specific decoders to produce text, pointclouds, and programs as output. The model uses a discrete latent space like VQVAE. Like other multimodal autoencoders, it is trained with paired multimodal data, gathered from several datasets. The model benefits from the multi-modal training, and outperforms models which handle fewer modalities at once.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great to correct or disambiguate the specific sentences I pointed to, and others of the same style if possible.  ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the paper's claims to be clearly made and well-supported by evidence. The proposed architecture is not especially surprising or novel. However, the evaluation is thorough, and the results should be of interest to a significant segment of the ML community. The authors also improved the paper significantly during the revision process, and the reviewers appreciated this. 

Given all this, I am recommending that the paper be accepted in its current form. 
Decision: Accept (poster)