Unsupervised Learning of Shape Programs with Repeatable Implicit Parts
Authors: Boyang Deng
Sumith Kulal
Zhengyang Dong
Congyue Deng
Yonglong Tian
Jiajun Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: shape programs
structured shape representation
unsupervised shape decomposition

Abstract: Shape programs encode shape structures by representing object parts as subroutines and constructing the overall shape by composing these subroutines. This usually involves the reuse of subroutines for repeatable parts, enabling the modeling of correlations among shape elements such as geometric similarity. However, existing learning-based shape programs suffer from limited representation capacity, because they use coarse geometry representations such as geometric primitives and low-resolution voxel grids. Further, their training requires manually annotated ground-truth programs, which are expensive to attain. We address these limitations by proposing Shape Programs with Repeatable Implicit Parts (ProGRIP). Using implicit functions to represent parts, ProGRIP greatly boosts the representation capacity of shape programs while preserving the higher-level structure of repetitions and symmetry. Meanwhile, we free ProGRIP from any inaccessible supervised training via devising a matching-based unsupervised training objective. Our empirical studies show that ProGRIP outperforms existing structured representations in both shape reconstruction fidelity and segmentation accuracy of semantic parts.

Unsupervised Learning of Shape Programs with Repeatable Implicit Parts
Boyang Deng
Sumith Kulal
Zhengyang Dong'
Congyue Deng'
Yonglong Tian
Jiajun Wu!
Stanford University, MIT, equal contributions
Abstract
Shape programs encode shape structures by representing object parts as subroutines and constructing the overall shape by composing these subroutines. This usually involves the reuse of subroutines for repeatable parts, enabling the modeling correlations among shape elements such as geometric similarity. However, ex- isting learning-based shape programs suffer from limited representation capacity, because they coarse geometry representations such as geometric primitives and low-resolution voxel grids. Further, their training requires manually annotated ground sound programs, which are expensive to attain. We address these limitations by proposing Shape Programs with Repeatable Implicit Parts (ProGRIP). Using implicit functions represent parts, ProGRIP greatly boosts the representation ca- pacity of shape programs while preserving the higher-level structure of repetitions and symmetry. Meanwhile, we free ProGRIP from any inaccessible supervised training via devising matching- bbaeed unsupervised training objective. Our empir- ical studies show that ProGRIP outperforms existing structured representations both shape reconstruction fidelity and segmentation accuracy of semantic parts.
Introduction
Representing geometry and structure simultaneously has long been one of the central pursuits of 3D shape modeling algorithms. The capability of preserving geometric details indicates the sufficient shape capacity of an algorithm. Mcanwhile, the prediction of plausible structures provides answers to quests for more intelligent shape understanding What are the parts constituting an object? How they assembled? What are the relationships among them?
Primitive-basse representations have been extensively studied address such queries. Specifically, numerous shape primitives are designed to construct objects in bottom-up manner, including the use of oriented boxes [57], ellipsoids [20], convex polytopes [12, 9], superquadrics [43], and invertible implicit functions [45]. Agnostic to the exact primitive used, these models all predict primitives independently, where each primitive accounts for a unique part of the object. Yet, more common in real world are recurrences of similar even identical geometric parts, e.g., legs of chair. Consequently, the absence of recurring parts modeling handicaps the advances of these methods towards human-leve understanding of object shapes.
To enable the understanding of shape regularities including the repetition and symmetry of parts, few recent papers have proposed to cast shape reconstruction as prediction and execution of neural shape programs [55, 25]. The designed domain specific language (DSL) of shape programs supports the prediction of repeatable parts. These methods however suffer from two major limitations. First, they often fail to achieve accurate geometry reconstruction due to the use of coarse geometric
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Spart cymes
<pose trars- trars-'up"/>
execute
</part>
generate
geamelyym
<poze "ytate-"--a" "saft="left"/>> <pose zotato "goz"" txons "right"/> </part>
execute
Figure 1: Our method represents an object as a shape program with repeatable implicit parts (ProGRIP). The program has two levels: the top level defines set of repeatable parts (as latent vector 24) and the bottom level defines all occurrences of each part with varying poses. The joint predictions, Le., posed parts, executed posed implicit functions. Both the generation and the execution of ProGRIP are invariant to the order of predictions at both levels. ProGRIP can be learned without any annotations using our proposed matching based unsupervised training objective.
representations, such as primitives or wwwrresolution voxel grids. Second, they often rely on hand- crafted synthetic datasets with ground-truth programs for training or pre-training [55], which prevents them from scaling up to various, unseen object categories one has to manually annotate shape programs for each new category.
In this work, we address these two fundamental limitations by proposing shape programs with repeatable implicit parts (ProGRIP, Fig. 1), and method that learns ProGRIP without supervision, illustrated in Fig. 4. As in earlier studies on shape programs, ProGRIP comprises of repeatable parts, rendering the modeling of recurring structures straightforward: unlike them, we now significantly enhance the quality of reconstructed shapes by introducing implicit functions as part representations. Compared with low resolution voxel- based representations, neural implicit representations have recently demonstrated strikingly high fidelity in shape and part modeling [37, 21J. Reformulating shape parts in our programs as neural implicit functions leads detail preeeerir reconstruction.
While earlier studies on shape programs require annotated programs for training. we propose an unsupervised learning objective, matching the oriented bounding boxes of predicted repeatable parts to non-repeatable box-based shape decomposition [64], which allows learning from unannotated shapes. Unlike the reconstruction loss, which confines each part to fitting local shape in the vicinity of its initialized location, our objective encourages repeatable parts to match to object shape elements with similar geometry, even if far from their locations. This leads to unsupervised acquisition of meaningful part decomposition, enabling the learning of ProGRIP for any object category.
We conduct extensive experiments on ShapeNet [6] to validate the effectiveness of ProGRIP on shape reconstruction and semantic part segmentation. Our results demonstrate that ProGRIP consistently preserves higher geometric fidelity than state-of-the-art structured representations all categories. Meanwhile, achieves competitive semantic part segmentation accuracy.
In summary, our key contributions are three-fold:
We propose ProGRIP, introducing implicit functions to represent repeatable parts in shape programs, greatly enhancing their representation capacity of shape programs.
We develop an unsupervised training objective for learning ProGRIP, relaxing prior require- ments of manually designed ground-truth programs.
We demonstrate the power of ProGRIP representation and learning objective on shape reconstruction, part segmentation, and shape compactness, outperforming statc-of-the-art structured representations on multiple categories.
2
Related Work
Our method aims to construct primitive-basee shape representations with recurring geometric struc- tures. It builds upon neural shape programs and implements primitives as neural implicit functions. Thus, we review the relevant literature in the following three fields:
Primitive-based shape modeling. Most artificial objects naturally exhibit part hierarchy and structural regularities [40]. Primitive-based shape modeling abstraction a decent way of modeling

---

Encoding network
Geometry level
Pose level
scale
part Intent
pointeloud
global feature
geometry trasfomne
scale
part latent
pose transformer
(R Ny (,-1: M vf luples
enonder
X
scale
part Intent
(scale part latent x/)
tuples
Figure 2: ProGRIP Generation. Given a pointcloud, an auto-encoding architecture generates ProGRIP composed of 2 levels of predictions. At the geometry level, our model predicts a set of (8,, z:a-iairs as the scales and deep latents of repeatable parts (middle); the pose level, our model predicts a set of (ti.j, Ri,j (e..))-tripets as translations, rotations, and existence probabilities (right). Transformers are used at both levels for permutation invariant predictions.
such relations. Some recent works have extended the classic idea of generalized cylinders [48, 3] or geons [2] by describing shapes as a collection of geometric primitives [57, 66, 49, 55, 25, 44, 32, 31, 28, 27,52, 52, 50, 19, 60, 63, 65]. In particular, Tulsiani et al. [57] and Zou et al. [66] abstracted shapes by minimizing the difference between raw shapes and reconstructions using pre-defined primitives, in unsupervised fashion. In Sharma et al. [49] and Du et al. [15], 3D CAD programs inferred from the perceptual input of shapes. In addition to part decomposition, inter-part relations are also widely discussed, Tian et al. [55] defined programs in terms of semantically meaningful parts; Mo et al. [39] built relational graphs upon parts to represent shape structures; Xu et al. [62] and Ritchie et al. [47] studied constructive shape modeling processes. However, these methods usually require detailed structural annotations on real or synthetic data, preventing them from being applied to various unlabeled shape categories.
Visual program induction. Programs have been commonly used to capture regularities (e.g. repre- senting repetitions by loops) and have been combined with deep models to describe various visual data, such as 2D graphics [16, 49, 35]. 3D shapes [55, 49, 25, 27], scenes [34, 36, 33, 18], and videos [30]. Most of these works [16, 55, 18, 17, 27, 59] follow the paradigm of analysis by synthesis: recognition network takes input visual data and outputs programs, which are rendered back to visual data to compare with the original input; such reconstruction error is back-propagated to optimize the recognition network. The rendering can be implemented as either a graphics engine [16] or learned neural models [55]. Our model shares similar spirits with these previous arts, but learns neural implicit occupancy decoder to render programs into 3D shapes. Additionally, ShapeMOD [26] assumes shape programs as input and proposes to discover macro 'options' to compress them; in contrast, our model aims infer shape programs and learn to encode parts simultaneously using implicit functions. Also relevant is DeepCAD [61] which utilizes predefined CAD commands to compose 3D shape programs while our model is not restricted to any predefined shape elements and learns the programs from scratch.
Neural implicit representations. Neural implicit representations define surfaces of geometry as zero level-sets of volumetric fields parameterized by neural networks [42, 37, 7, 23]. Acquirable from raw shape data [1, 22, 1 11], neural implicits show excellent performances in capturing
frequeryy details at any level [38, 51, 53], even for large-scale scenes [51, 24, 5J. With latent space encoding conditioned on local features [10, 24, 46, 5]. poses [14], or parts [13, 41J, neural implicits can naturally incorporate geometric and structural priors into its shape representation. A few recent works also reveal category-level structural similarities through implicit reconstruction, such as Genova et al. [20, 21] (via patch-correspe (oruchhcorrssondenccss and Chen et al. [8] (via part (co-) (co-)segmentation) or cross- category geometric similarities [56], with zero minimal supervision. Recently, Neural Parts [45] also models 3D shapes based on implicit shape elements. Different from our model, Neural Parts treats each element independently hence ignores essential geometric self-similartic in the modeling.
We summarize the key differences between ProGRIP and most relevant prior methods below and empirically compare ProGRIP against them in Sec. 5.
CubeSeg [64] is part-based representation that models shapes as a collection of oriented boxes. Each box in the collection is regarded an independent element with its own shape
3

---

Query point canonicalization
Occupancy decoder
Query
implicit part decoder
Occupancy probubility
scaling
point
multiply
inverse transformaation
part latent
existence
Figure 3: Posed Implicit Functions. We execute each posed part (i.e., (8,, Zi, bi.j. Ra, 0z,5) tuple) as a posed implicit function. A posed implicit function constructs an occupancy function ot, to answer point queries æ. For each query point æ, we first canonicalise it using (s,, ti,, Ra,5), then predict its occupancy conditioned on part latent 21. and finally mask it by binarised existance 81.--
and pose. Contrarily, ProGRIP predicts shape and pose separately and allows the same shape to have multiple copies in various poses. This enables the understanding of self similarity within a shape.
Shape2Prog [55] predicts programs with repeatable shapes voxel grids. ProGRIP differs from Shape2Prog in using a higher fidelity neural implicit shape representation and learning programs without supervised pretraining.
BSP-Net [9] is an implicit representation targeting structured shape understanding. Similar to CubeSeg, it outputs independent shape elements, while ProGRIP recognizes shape similarity among elements.
ProGRIP
We model shapes using shape programs with repeatable implicit parts, or ProGRIP (Fig. 1). Given an object, we first extract its structural regularity especially repeating geometry via shape program generator. The generator outputs shape programs with two levels of permutation invariant predictions, parts at the geometry level and poses at the pose level (Sec. 3.1 and Fig. 2). The compound set of these predictions are posed parts, which we use to construct object shapes. Each posed part is essentially posed implicit function (Sec. 3.2 and Fig. 3). Composing all posed parts gives us the
overall object shape, which is essentially a spatial occupancy function R + [0, 1] with its
surface boundary defined as 00 {a R3 O(z) 0.5).
3.1 Generating ProGRIP
Unlike the sequential programs Tian et al. [55], we introduce parallelism into our program, allowing repeatable parts to be executed simultaneously without a specified order. Specifically, our program generator takes as input a point cloud X TR. Px3 with P points and interprets as ProGRIP using auto-encoding architecture (Fig. 2). A ProGRIP consists of two-level hierarchy. The top level, namely geometry level, defines a set of repeatable parts as shape building blocks, e.g. the shape of leg a chair. The bottom level, namely pose level, predicts set of poses for each repeatable part from the geometry level, e.g., four copies of the leg shape in four different poses on a chair. Within pose set the locations and orientations of all occurrences of the corresponding part in the object (Fig. 1). All predictions at the pose level, joint by the prediction from its parent part, specify all element posed parts (e.g., four legs of table) that constitute the object shape. Formally, by slight abuse of notations, we define the hierarchy:
{(5;, zi = L....,N,, where 8; E 3R3 describes the axis-aligned scales of the i-th repeatable part, z; E de encodes the d-dimensional deep latent of this part, and N is the number of parts at this level.
Geometry Level:
2. Pose Level: {(ta,) R.,; 81,5) li 1,..., M) for the i-th part, where h.jj R.jj and 8 are the translation, rotation, and existence probability of the j-th posed occurrence of this part. Note that during program generation, we predict continuous probability as 6. to allow back propagropantionn of gradients, while during program execution, binarise 8,, into

---

(a) Commonly used reconstruction loss.
Proposed matching loss
Figure 4: A Demo of Unsupervised Matching Loss. On a task of fitting repeatable parts to a target shape, starting from the same initialization, the reconstruction loss (left) confines each posed parts to their initial local neighborhood and consequently prevents better parts arrangements. Conversely, our matching loss (right) match posed parts to local geometry of targets by shapes, thus rescue the part arrangement from the suboptimal local minima in reconstruction loss.
d.j = 1(61, 2 0.5) and remove occurrences with 8 = O to allow varying numbers of occurrences of the i-th part.
In adaptation to the program parallelism, the predictions at both the geometry level and the pose level are orderless sets. We employ two nested transformers [58] our architecture to facilitate permutation invariance at both prediction levels. In particular, the geometry transformer takes in the encoded global feature for the object and predicts the the parts including the scale 8, and a deep part latent 2,. Then, we feed each 2; pose transformer to predict the (R. ji.. 8t,)) triplets for each posed occurrence. Combining the pose predictions and their parent part prediction produces a set of {ss,, 3i, و. Ri } tuples, each defining a posed part.
3.2 Executing ProGRIP
We execute ProGRIP by executing the predicted posed parts as posed implicit functions [37, 42, 7]. Instead of simple polygonal primitives with only few shape parameters, we propose implicit function PP( z) TR3 + [o, 1] to represent part. It takes in any point query æ and predicts the likelihood of æ being inside of a part.
the occupancy decoder parameterized by deep neural network and shared by all parts. z is the part latent from the geometry level. We condition P on obtain specific implicit function for part. For a posed part parameterised by the 5-tuple (st, Zz. ti.j, R 81,5), we construct its posed implicit function 0ú,
01,((x) = &.,P ( ((TT) (xx t(.j)) /s: 2i
Note that we binarize deg to dz.. As in Fig, 3, given query point æ, we first canonicalize its coordinates to x (R. (r)))/s; //,- Then we feed x' to P together with the shape code z,. The binarized existence probability òz, is finally multiplied to the predicted occupancy, masking out the non-existing primitives. The eventual output of the program an object shape as the union of all posed parts. The object occupancy function can therefore be written as
(xz) = max max (zz)
(2)
JE[M]
Unsupervised Learning of ProGRIP
As our shape modeling consists of two steps, i.c., the generation and execution ProGRIP, we first learn the ProGRIP generator and then optimize our posed implicit functions. Both optimizations are done without any supervision.
4.1 Learning Generate ProGRIP
Prior works generating shape programs often rely hand-crafted synthetic programs to bootstrap the learning [55]. manually creating programs for numerous object categories prohibitive, this significantly restricts large-scale applications of these works. To circumvent this limitation, would
5

---

resort to unsupervised training objectives. A trivial design of such objectives is reconstruction losses. Particularly, we only supervise the composition of all elements by the overall reconstruction quality, Despite its decent effectiveness in prior works [12, 9, 57, 64], we show in one demonstrative case (Fig. 4) that it's not applicable when we have repeatable parts. Reconstruction loss encourages each posed part to evolve to the local geometric structure closest to its initialized location. When parts are repeatable, different copies of the same part may appear in irrelevant locations, c.g., one near a chair leg while another close to the chair seat. The local minimal of reconstruction loss consequently traps these copies to its initially assigned neighborhood and suppresses the transformation that would lead to more meaningful assignment of these copies. To address this issue, we devise matching -based unsupervised objective that supervises a posed part using a local geometry of the object with a similar shape, even if it's far from the current location of this posed part. Practically, we match our predicted posed repeatable parts to a rep-raapee primitive decomposition of the object, where each primitive serves as local geometry descriptor.
We acquire the non-repeatable primitive decomposition using the unsupervised box-based algorithm in Yang and Chen [64]] Namely, given pointcloud X, it predicts a set of independent oriented bounding boxes {3x, Eg. Rgg 5x)) I k = I....... MM. Here 8g is binary value indicating if the k-th box exists (1 for present and for absent). If the number of valid boxes is less than N NM (i.e., the number of posed parts predicted by our ProGRIP), we pad the remaining entries with all zero. As the subset 4-tuple {(Fk, 12z Rg: 6x)} from the 5-tuple specifying posed part also defines its oriented bounding box, we optimize our predicted primitives by fitting these boxes to {($ Ey. Rg. 3x))- Specifically, we use box scale loss L., as well two auxiliary losses for the box vertices (C.) and existence (Le) to supervise the fitting of boxes. The overall loss is a weighted combination f. of these three losses. We name it matching loss. Inspired by Carion et al. [4], we base the computation of all loss terms on a permutation p of NM elements, where p(k) is the k-th element in the permutation. Without loss of generality, we define L. Lu, and Le for the k-th predicted posed part as the following:
Scale loss L.. We first supervise the shape of the posed part using the target local geometry as we seek to find the repeated shape elements in object. In the context of oriented boxes, we fit the axis-aligned scales of our predicted boxes to their assigned target boxes from the non-repcatable decomposition. We implement it the complementary of Intersection uninnnnionn between two axis-aligned origin-centered boxes defined by scales 85 and & (pt))
C. (k) = 1 1 [1 I--IoU(Sk,)) pp(k)))
Vertices loss L. We also require the matched predicted box and the target box to have the same location and orientation by minimizing the ( didisstecce between the box vertices after posing. Formally, we define the box vertices generated using (Ra, tk, 8k) as B₁ and similarly B P()) for (Ri, Et, 8x). Consequently.
(k) 1 (0)00)=1) BB B B1(
(4)
Existence loss Le. As we are matching two sets with possibly different number of elements, we synchronize the existence of each element appearing in the two sets. To enforce such a coherency, we minimize the cross-entropy for the existence probabilities between each pair of matched prediction and target bounding boxes:
Le (k) = H(5e, pp()).
(5)
Therefore the matching loss L. of an object is defined as
NM
f. Σ A.L.(K) + ) + (k)..
(6)
During optimization, we select the optimal permutation p² among all permutations SNM to compute f. and apply gradient descent. Particularly, at each step, we search for p" arg min min L... using Hungarian algorithm [29]. In our experiments, we use X, = 1, A, = 0.2, and X = 0.8 for all categories.
Here, we provide an intuitive understanding of our optimization process. First, our objective anchors one copy of an initialized shape to nearby part on the object with similar scales, e.g., leg of

---

Method
IoU 1
F-Score@0.01
Method
mloU f
Chair
Table
Airplane
Chair
Table
Airplane
Chuir
Table
Airplane
Shape2Prog [55]
0.365
0.345
N/A
20.47
25.76
NIA
Shape2Prog [55]
0.549
0.743
N/A
CubeSeg [64]
0.315
0.238
0.360
25.56
30.38
42.53
CubeSeg [64]
0.737
0.895
0.688
BSP-Net [9]
0.541
0.121
0.172
12.91
36.18
16.91
BSP-Net [9]
0.689
0.859
0.721
ProGRIP (ours)
0.620
0.656
0.680
57.98
72.30
76.84
ProGRIP (ours)
0.857
0.757
ShapeNet Reconstruction
Unsupersised Part Segmentation
Table Performance on ShapeNet. To quantitatively evaluate ProGRIP, we report the reconstruc- tion quality using IoU and F-Score, and unsupervised part segmetnation performance using mloU. For reconstructions, ProGRIP outperforms prior approaches by a margin. For unsupervised part segmentations, ProGRIP tops 2 out of 3 categories. The results of Shape2Prog [55] on airplanes are not available, as the domain specific language for airplanes is not defined their original paper.
chair, due the joint impact of f. and L.. Then, as the gradient on scales will be propagated to other copies of the same shape, these copies will converge to leg shapes as well even if they are not matched to legs initially. In later rounds of optimization, these copies' poses will also converge to the nearest legs the chair because of the higher weight on Ls. In the end, ProGRIP learns the recurring leg shapes on a chair that are posed correctly.
4.2 Learning to Execute ProGRIP
After the optimization in Sec. 4.1, ProGRIP can predict the repeatable oriented bounding boxes of each shape element. We further train ProGRIP to refine the shape element within each box as posed implicit functions. As defined in Sec. 3.2, the outcome of executing ProGRIP is an object occupancy function O. To optimize this function, we uniformly sample points æ within the padded bounding box of an object and collect the ground truth occupancy for each points (xx), where represents inside points and 0 outside. We further augment the samples by sampling points near the surface and sampling the point cloud of the object. Note that we define the occupancy of on-surface points x 80 as (xx) = 0.5. The loss C. for a set of C' samples from an object is averaged cross entropy:
Σ H(0()), C.
0
x
Experiments
We conduct all our experiments using the ShapeNet [6] dataset following ShapeNet Terms of Use. We quantitatively evaluate the performance of our model on the reconstruction (Sec. 5.1) and part segmentation (Sec. 5.2) tasks. Additionally, we dive deep into our method via investigating the impact of our unsupervised matching loss (Sec. 5.3). We also showcase applications of ProGRIP in shape editing and compact shape representations our supplementary material, along with failure cases.
Metrics. We quantitatively evaluate our method on its reconstruction quality and segmentation accuracy. For reconstruction, we use two quantitative criteria: ! The volume-basee Intersection-oven Union (IoU), where we uniformly sample points and evaluate the IoU of our predicted occupancies and the ground truth occupancies. Note that this metric is focused on the accuracy of the volume reconstruction. 2 The surface-bar based F-Score, where we follow the protocol defined in Tatarchenko et al. [54] to compute the "percentage of surface that is correctly reconstructed". It is claimed to be the best metric to assess the fidelity of the reconstruction, with concentration on the surface. For segmentation, we use the mean per-label of our predicted segmentation and the ground truth labels. As our segmentation task unsupervised, to compute the mloU during evaluation, we assign the ground truth label to each predicted label by majority voting, i.c., picking the ground truth label with the highest correlation in the training split.
Pnarrectioo-vve--nniii
(mloU) to measure the alignment
Baselines. We compare against representative state-of state-of-the-art the-art methods from relevant domains as discussed in Sec. 2. Specifically, we select CubeSeg [64] for primitive-based shape representation, Shape2Prog [55] for shape programs, and BSP-Net [9] for implicit or hybrid shape representations. Note that while LDIF [21] is also an implicit part-base part-based representation, it strives for high fidelity
7

---

GT
Shape2Prog
Seg
BSP-Net
ProURIP
ProGRIP
(program)
Figure 5: We qualitatively compare ProGRIP with state-of-the-art shape decomposition methods (including Shape2Prog [55], CubeSeg [64], and BSP-Net [9]) rendering out their reconstructions. For ProGRIP, we present both plain reconstruction (the last second row) as well as per-part colored rendering (the last row). The ground truth mesh is shown in the first row. ProGRIP reconstructs shapes more accurately and smoothly. Note that the cylindrical parts on the chair back (col. 1 and col. 2) are reconstructed as different copies of the same shape as 4 chair legs for their geometric similarity. Mcanwhile, since there are asymmetric table legs in ShapeNet (col. 4), our method discovers symmetries for left and right legs independently (col. 5).
Chair
Table
Airplane
Figure 6: We visualize the unsupervised part segmentation results produced by our ProGRIP, which generalizes well across different object categories. Besides, ProGRIP is capable of segmenting fine-grained parts, such engines and tails of aircrafts.
reconstruction rather than better structural understanding. We don't directly compare to it due to different encoding schemes, i.e. local feature encoding (I.IIF) vs. global feature encoding (ProGRIP).
Implementation details. We use an auto-encoding architecture for our model. The encoding architecture is identical to Yang and Chen [64] and the programs are predicted by 2 levels of Transformers [58]. We first train our ProGRIP generator (Fig. 2) using the decomposition from Yang and Chen [64] and Lm. Then we train the ProGRIP execution module (Fig. 3) as well fine-tune the program generator by fitting occupancy predictions to ground truth occupancies of samples. Please supplementary materials for more implementation details.

---

recom
recon
match
loss
loss
loss
ProURIP trained by recoal Joss
Ground truth dbape
Abstracted non-repeotable boxes
ProGHEPP truined by matching loss
Our Matching Loss Lm vs. Reconstruction Loss in Yang and Chen [64]. We use either our matching loss f. (bottom) or the reconstruction loss in Yang and Chen [64] (top) to train a ProGRIP and visualize their results on one test example. Note that matching loss uses the abstracted non-repeatable boxes that are unsupervised learned by reconstructing the ground truth shape (vertical arrow). Therefore, both losses are fully unsupervised. However, we observe clearly better reconstruction quality and more plausible part arrangement for the ProGRIP trained by L. on bottom right than trained by reconstruction loss on top right.
Figure 7:
5.1 Shape Reconstruction
As we extend independent non-repeatable primitive modeling to repeatable geometry modeling, we boost the efficiency of geometry learning the same repeatable part can receive gradients from its copies at various local neighborhoods of the object. Additionally, we employ implicit shape representations as our neural implicit primitive to greatly enrich the capacity of our primitive shape space. We validate these two improvements on the point cloud-to-mesh 3D reconstruction task. In this task, each model takes as input point cloud with 4096 points sampled from the object surface and strives for reconstructing the object surface with high fidelity. We compare against baseline methods quantitatively in Tab. 1, Reconstruction and display few qualitative visualizations in Fig. 5. We find that our method consistently outperforms other primitive-based baselines in reconstruction quality. Meanwhile, our predicted repeatable parts are reasonably posed to fit local geometry with similar or identical shapes, such as the four legs of a chair.
5.2 Unsupervised Part Segmentation
We further quantitatively study the primitive assignment our shape programs on the unsupervised part segmentation task. In this task, we aim to learn to segment each object into semantic parts without any segmentation supervision. Specifically, we provide the point cloud of an object to each method and ask for per-por per-point semantic part segmentation labels. Each method is trained on reconstruction tasks only and we use their part decomposition test time evaluate the segmentation quality. In Tab. 1, Unsupervised Part Segmentation, we show the per-label mloU of different primitive-basee methods and find that our ProGRIP achieves competitive performance in comparison to state-of-the art unsupervised methods, such CubeSeg and BSP-Net; it even outperforms Shape2Prog, which uses synthetic semantic structure ground truth for pretraining.
5.3 Analysis: Impact of Matching Loss
We present the matching loss L. in Sec. 4.1 better fit for identifying and learning repeatable parts than the commonly used reconstruction loss. Here we provide an experimental study to validate our design choice. We visualize the results Fig. In this study, we train the same ProGRIP model using either our matching loss L. on abstracted non-repeatable boxes or the reconstruction loss directly from the ground-truth shape as defined in Yang and Chen [64]. We find that ProGRIP learned from the matching loss produces both decent quality of reconstruction and meaningful repeatable parts arrangement. This can be observed in the symmetric arms of the chair that fitted by two copies of the same repeatable part and also in the four repeated legs. In contrast, the part allocation of ProGRIP learned directly from the reconstruction loss appears to be poor quality elucidating the difficulty of learning part decomposition directly from reconstruction. The reconstruction itself is also crippled by the unwise part assignment as each repeatable part receiving conflicting gradients.

---

6 Conclusion
We propose to represent objects as shape programs with repeatable implicit parts, or ProGRIP, a structured high fidelity shape representation that can be learned without structure annotations. While prior works on shape programs suffer from poor fidelity due to the limited representation capacity of their coarse shape primitives, we significantly augment the reconstruction quality by introducing implicit functions to implement the predicted parts. Additionally, we design an unsupervised training objective to facilitate the learning of ProGRIP on any object category, largely enhancing the scalability of shape programs compared to prior works that only work with few categories with ground truth semantic structures. On the shape reconstruction task, our ProGRIP champions on all categories among structured representations. It also achieves top-tier part segmentation accuracy and is shown to be an accurate yet highly compact representation capturing repetitions and symmetries of object parts, enabling efficient shape editing.
Discussion
Broader Impact. This paper studies how capture the regularities (e.g., geomelry and structure priors) of shapes by programs. We believe it will be beneficial to the field of shape representation, and it can potentially be used by practitioners edit existing CAD models build new ones from scratch. This algorithm we build is unsupervised, which may have the beneficial effects of being more cost effective and reducing biased introduced by human annotations. Meanwhile, we are aware that the whole system is not completely bias free: the domain specific language designed may still be biased. At this point, we do not foresee any direct ethical issues and concerns that may rise with our system, but we urge the users to be mindful about the potential bias.
Acknowledgments. We'd d like to thank Pratul Srinivasan for suggesting the compactness application of ProGRIP, Kaizhi Yang for kindly open-soureing the code of CubeSeg [64], the Occupancy Networks [37] team for relcasing their pre-processed occupancy samples, and the Stanford Artificial Intelligence Laboratory (SAIL) staff for maintaining and supporting our computational infrastructures. This work is in part supported by the Stanford Institute for Human-Centenl
Artificial Intelligence (HAI), Center for Integrated Facility Engineering (CIFE), Toyota Research Institute (TRI), NSF RI #2211258, NSF CCRI #2 10095,, ONR MURI N00014-24 1-00---2--------- 1-2440, and Adobe, Amazon, Analog, Autodesk, IBM, JPMC, Meta, Salesforce, and Samsung. BD is funded by a Meta Research PhD Fellowship. SK is in part supported by the Brown Institute for Media Innovation.
References
[1] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In CVPR, 2020.
[2] Irving Biederman. Recogittion-yy-Comporee A Theory of Human Image Understanding. Psychol. Rev, 94(2):115, 1987. 3
[3] Thomas O. Binford. Visual Perception by Computer. Invited talk at IEEE Conf. on Systems and Control, 1971. 3
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 6
[5] Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe, Deep local shapes: Learning local sdi priors for detailed 3d reconstruction. In ECCV, 2020. 3
[6] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao. Li Yi, and Fisher Yu. ShapeNet: An Information-Raa Rich 3D Model Repository. 112.v:152..03012 2015. 2.7
[7] Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning Exploration Policies for Navigation. In ICLR, 2019. 3.5
[8] Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. Bae-net: Branched autoencoder for shape co-segmentattin. In CVPR, 2019. 3
[9] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In CVPR, 2020. 1, 4, 6. 7,8
10

---

[10] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In CVPR, 2020. 3
[11] Julian Chibane, Aymen Mir, and Gerard Pons-Moll Moll. Neural unsigned distance fields for implicit function learning. In NewrIPS, 2020. 3
[12] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.
Cvxnet: Learnable convex decomposition. In CVPR, 2020. 1,6 6
[13] Boyang Deng, John P Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa neural articulated shape approximation. In ECCV, 2020. 3
[14] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Lconidas Guibas. Vector neurons: A general framework for so (3)-eqvivariant networks. In ICCV, 2021. 3
[15] Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielherg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojeiech Matusik. Inversecsg: Automatic conversion of 3d models to esg trees. ACM Transactions on Graphics (TOG), 37(6);1-16, 2018. 3
[16] Kevin Ellis, Daniel Ritchie Armando Solar-Lezama, and Josh Tenenbaum. Learning to Infer Graphics Programs from Hand-Drawn Images. In NeurlPS, 2018. 3
[17] Kevin Ellis, Maxwell Nye. Yewen Pu, Felix Sosa, Joshua B. Tenenbaum, and Armando Solar-Lezama Write, Execute, Assess: Program Synthesis with a REPL. arXiv: 1906,04604, 2019. 3
[18] Yaroslav Gamin, Tejas Kulkamii, Igor Babuschkin, S. M. Eslami, and Oriol Vinyals. Synthesizing Programs for Images Using Reinforced Adversarial Learning. In ICML, 2018.
[19] Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions on Graphics (TOG), (8(1)111 2019. 3
[20] Kyle Genova, Forrester Cole. Daniel Vlasic, Aaron Sarna, William Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In CVPR, 2019. 1,3
[21] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR, 2020. 3,
[22] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In ICML, 2020.
[23] Amir Hertz, Or Perel, Raja Giryes, Olga Sorkine-Horuung, and Daniel Cohen-Or. Spaghetti: Editing implicit shapes through part aware generation. arXiv preprint arXiv:2201. 2022. 3
[24] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. CVPR, 2020.
[25] R Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang. Ellen Jiang, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie Shapeassembly: Learning to generate programs for 3d shape structure synthesis. In ACM TOG, 2020. 3,3
[26] R Kenny Jones, David Charatan, Paul Guerrero, Niloy J Mitra, and Daniel Ritchic. Shapemod: Macro operation discovery for 3d shape programs. In SIGGRAPH, 2021. 3
[27] R Kenny Jones, Aalia Habib, Rana Hanocka, and Daniel Ritchie. The neurally-guided shape parser: Grammar-based laheling of 3d shape regions with approximate inference. In CVPR, 2022. 3
[28] Kacper Kania, Maciej Zięba, and Tomasz Kajdanowicz. Uosg-net-unsupprrvieed discovering of constructive solid geometry tree. In NewrlPS, 2020. 3
[29] Harold W Kuhn. The hungarian method for the assignment prohlem. Naval research logistics quarterly, 2, 1955. 6
[30] Sumith Kulal, Jiayuan Mao, Alex Aiken, and Jiajun Wu. Hierarchical motion understanding via motion programs. In CVPR, 2021. 3
[31] Brie-Tuan Lē, Minhyuk Sung, Duygu Ceylan, Radomir Mech, Tamy Boubekeur, and Niloy J Mitra, Cptin: Cascaded primitive fitting networks for high-resolution point clouds. In CVPR, 2021.
11

---

[32] Lingxiao Li, Minhyuk Sung. Anastasia Dubrovina, Li Yi. and Leonidas J Guibas. Supervised fitting of geometric primitives 3d point clouds. In CVPR, 2019.
[33] Yikai Li, Jiayuan Mao, Xiuming Zhang, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Perspective Plane Program Induction from Single Image. In CVPR, 2020. 3
[34] Yunchao Liu, Zheng Wu, Daniel Ritchie, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to Describe Scenes with Programs. In ICLR, 2019. 3
[35] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig. Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logie programming. In NeurlPS, 2018.
[36] Jiayuan Mao, Xiuming Zhang, Yikai Li, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Program-Guided Image Manipulators. ICCV, 2019,
[37] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In CVPR, 2019. 2, 3,5,
[38] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 3
[39] Kaichun Mo, Paul Guerrero, Li Hao Su, Peter Wonka, Niloy Mitra, and Lconidas Guibas. Structurenet: Hicrarchical graph networks for 3d shape generation. In SIGGRAPH Asia, 2019. 3
[40] Kaichun Mo. Shilin Zhu, Angel Chang, Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In CVPR, 2019. 2
[41] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. In ICCV, 2021. 3
[42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 3,5
[43] Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In CVPR, 2019.
[44] Despoina Paschalidou, Lue Van Gool, and Andreas Geiger. Learning unsupervised hicrarchical part decomposition of 3d objects from single rgb image. In CVPR, 2020. 3
[45] Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In CVPR, 2021. 1,
[46] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In ECCV, 2020.
[47] Daniel Ritchie, Sarah Jobalia, and Anna Thomas. Example-based authoring of procedural modeling programs with structural and continuous variability. In Computer Graphics Forum, volume 37, pages 401-413. Wiley Online Library, 2018. 3
[48] Lawrence G. Roberts. Machine Perception of Phree-Dimensional Solids. PhD thesis, Massachusetts Institute Technology, 1963.
[49] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. Neural shape parsers for constructive solid geometry. JEEE TPAMI, 2020.
[50] Gopal Sharma, Difan L Liu, Subhransu Maji, Evangelos Kalogerakis, Siddhartha Chaudhuri, and Radomír Mēch. Parsenet: parametric surface fitting network for 3d point clouds. In European Conference on Computer Vision, pages 261-276. Springer, 2020. 3
[51] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NeurlPS, 2020. 3
[52] Chun-Yr Yu Sun, Qian-Fang Zou, Xin Tong, and Yang Liu. Learning adaptive hierarchical cuboid abstractions of 3d shape collections. ACM Transactions on Graphics (TOG), 38 2019. 3
[53] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Koan Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
12

---

[54]
Maxim Tatarchenko, Stephan R Richter. René Ranfil, Zhuwen Li. Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In CVPR, 2019. 7
[55] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning Infer and Execute 3D Shape Programs. In ICLR, 2019. 3, 7,
[56] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. Patchnets: Patch-based generalizable deep implicit 3d shape representations. In ECCV, 2020.
[57] Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexci A. Efros, and Jitendra Malik. Learning Shape Abstractions by Assembling Volumetric Primitives. In CVPR, 2017. 1,3, 6
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurlPS, 2017. 5,8
[59] Karl DD Willis, Pradeep Kumar Jayaraman, Joseph G Lambourne, Hang Chu, and Yewen Pu. Engineering sketch generation for compuler-aided design. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2105-2114, 2021.
[60] Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 829-838, 2020.
[61] Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer-aide aided design models. CVPR, 2021.
[62] Xianghao Xu, Wenzhe Peng, Chin-Yi Yi Cheng, Karl DD Willis, and Daniel Ritchie. Inferring cad modcling sequenees using zone graphs. In CVPR, 2021. 3
[63] Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Lin Gao. Dsg-net: Learning disentangled structure and geometry for 3d shape generation. ACM Transactions on Graphics (TOG), 2022.
[64] Kaizhi Yang and Xuejin Chen. Unsupervised learning for cuboid shape abstraction via joint segmentation from point clouds. In SIGGRAPH, 2021. 6, 7,8, 9, 10
[65] Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances Neural Information Processing Systems, 33:6315-6326 2020.
[66] Chuhang Zou, Ersin Yumer. Jimei Yang. Duygu Ceylan, and Derek Hoiem. 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks. In ICCV, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see our discussion in our supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see our discussion in our supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] All the technical details for reproducing all of our experiment results are included in the submission. We plan to release our source code upon publication. We use public data for all our experiments.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] All our models deterministic.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
We don't have any assets.
(d) Did you discuss whether and how consent was obtained from people whose data you' rr using/curating? [Yes] We follow the license of the dataset we use for our experiments. (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "jzWxZAOFlKw", "reviewer": "Reviewer_JZ3w", "summary": "The authors propose a method to reconstruct part-based shape models from point clouds, where each part has a local neural representation of the part's geometry. Additionally, multiple instances of the same geometry can be re-used by different parts, thereby explicitly modelling translational and rotational symmetries. The point cloud is encoded into a global feature vector, that is then used in two transformer-based steps to first generate part geometries and then translated and rotated instances of the part geometries. The method needs a part decomposition of the training shapes as supervision, although this decomposition can be obtained using existing unsupervised methods. A novel loss allows for efficient training of parts without requiring explicit correspondences with the ground truth parts. The authors show that this approach allows for part-aware shape reconstruction, unsupervised segmentation, and shape editing.\n\nThe two-step generation approach with geometries and instances and the loss with shape-based part assignments both seem like interesting contributions to me. Using neural representations of part geometries seems a bit less novel, since it has been used before in prior works (for example, Neural Parts, Local Deep Implicit Functions).", "questions": "Providing a clear description of what the authors consider to be the main practical contributions of their representation with repeatable parts of existing shape generation methods, and how these contributions are demonstrated in the paper would be good.", "limitations": "The authors show some limitations of their method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The separation of part generation into geometry and instances seems like an interesting idea to make use of the compositional nature of many shapes, and seems to be relatively novel in the context of part-based shape generation.\n- Implementation seems technically solid.\n- The paper is generally well-written and easy to read.\n- The evaluation shows some advantages in unsupervised shape segmentation.\n\nWeaknesses:\n- The authors do not clearly demonstrate the advantages of their contributions. What is the advantage of having a representation with repeatable parts over current work with non-repeatable parts or current work without parts? An more thorough evaluation of some applications that are only possible with repeatable parts, or that clearly perform better than the state of the art would be necessary.\n- The evaluation focuses on shape reconstruction, however existing methods like Occupancy Networks perform significantly better on shape reconstruction (Occupancy Networks should be included in the left part of Table 1). There could be other applications where working with repeatable parts is necesary or beneficial (like part editing, part mixing, inductive biases for generation, interpolation, etc.), but these are not demonstrated thoroughly enough. Only one example of shape editing is provided.\n- The segmentation experiments are reasonable, but by themselves not sufficient for acceptance, and Table 1 in the supplementary seems to suggest that using repeatable parts actually hurts segmentation performance compared to using non-repeatable parts.\n\nIn summary, I like the idea of using repeatable parts to represent shapes, but the evaluation does currently not convincingly demonstrate the advantage of such a representation.\n\nDetails:\n\n- Using a part-based representation with part re-use is described as a major advantage of the proposed method in the introduction, and it is a major part of the technical contribution. However, from the evaluation it is unclear what the advantage of such a representation is over the state-of-the-art. For reconstruction accuracy, non-part based representations like OccNet seem to perform better, so that does not seem to be a strong suit of the proposed representation, or possibly of part-based representations in general. The segmentation comparison does show some advantage in Table 1, even if the results are a bit mixed, but Table 1 of the supplementary shows that the segmentation performance is lower with part re-use. There is one edit example in the supplementary that shows some down-stream advantages of part re-use, but additional examples, ideally with more repeated parts (storage furniture?) and possibly mirrored parts (usually the chair arms are mirrored, not related by only translations and rotations) would be more convincing. Also, metrics that specifically measure the success of the part re-use are missing, for example measuring how often symmetric parts in a shape are correctly modeled with the same geometry and how often parts that have different geometry are incorrectly modeled as instances of the same geometry.\n\n- Comparisons to Neural Parts and Local Deep Implicit Functions (LDIF) both seem relevant, since they both perform unsupervised shape decomposition and use neural representations of part geometries. For LDIF, the authors argue that it is not included due to using a local feature encoding compared to the global feature encoding of ProGRIP. What are the specific applications or tasks that LDIF can't do due to its local feature encoding, but that ProGRIP can do? This needs to be discussed and clarified if the goal is to show that an empirical comparisons to LDIF is not needed since there is a clear theoretical advantages. I can imagine, for example, that a single global latent space enables interpolation or generation, which may be more difficult with local latent spaces. Although ideally such an argument is backed up by empirical evidence.\n\n- It might be good to clarify the extent to which the method is unsupervised in the introduction. It is true that the method can be trained on an unannotated dataset, but only if a part annotation is generated for the dataset as a pre-processing step, possibly using existing unsupervised methods. Calling it a fully unsupervised approach in the introduction without further clarification may incorrectly create the expectation that the method is trained end-to-end with only the final occupancy loss as supervision.\n\n- To a lesser extent, calling the representations a shape program may also create the unrealistic expectation of working more complex program. The current representation may better be described as structural representation rather than a program, since only two types of 'operators' are effectively used, always in the same order: first box creation and then instancing. But I would consider fixing this point (for example by changing the title and introduction) optional.\n\n- I agree with the authors that representation compactness is a good measure for a learning-based model. However, it seems like the F-score the authors use measures the reconstruction quality with geometry (the information stored in the z-vectors), but only measures space requirements without geometry (i.e. without counting the information stored in the z-vectors). This seems like an unfair comparison, as the other methods do not make use the z-vectors to store additional information that is useful in improving the F-score.\n\n- The following papers could be added to the related work:\n\t- SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation, Hertz et al., ArXiv 2022 (concurrent work)\n\t- SDM-NET: Deep generative network for structured deformable mesh, Yang et al., Siggraph Asia 2019\n\t- PQ-NET: AGenerative Part Seq2Seq Network for 3D Shapes, Wu et al., CVPR 2020\n\t- DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation, Yang et al., Siggraph 2022 (concurrent work)\n\t- Generative 3D Part Assembly via Dynamic Graph Learning, Huang et al., NeurIPS 2020\n\t- Write, Execute, Assess: Program Synthesis with a REPL, Ellis et al., NeurIPS 2019\n\t- InverseCSG: Automatic Conversion of 3D Models to CSG Trees, Du et al., TOG 2018\n\t- Engineering Sketch Generation for Computer-Aided Design, Willis et al., CVPR 2021Workshop paper\n\t- Learning Adaptive Hierarchical Cuboid Abstractions of 3D Shape Collections, Sun et al., Siggraph Asia 2019\n\t- ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds, Sharma et al., ECCV 2020\n\n- Since the box generation model is pre-trained without knowledge of the geometry, it should create two instances of the same box in cases where two boxes are have the same shape. But same box shape does not necessarily mean same geometry (for example, two boxes may contain mirrored geometry, or just geometry that has slightly different details, etc.). Did you observe the model successfully splitting two instances of the same box into two different boxes in the refinement step if the two boxes are the same but not their geometry? Showing such an example might be good to confirm that this is not a problem.\n\n- Transformers typically work on ordered sequences (step i takes the output of step i-1 as input), some details should be given how the transformer is set up here to be order-invariant. For example, does each step still output a probability distribution that is then sampled probabilistically (the standard setup for transformers), or does each step regress/classify output values deterministically (given the input)? If each step samples a potentially multi-modal probability distribution, how are the samples of different steps coordinated to make sure only compatible modes are selected when sampling?\n \n- Allowing for a reflection transformation in the instances seems like it could enable significantly more part re-use, since parts are often mirrored. Was there a specific reason this was not used? Or can the rotation R also be used to model a reflection? This could be discussed in the future work for example, or a clarification is needed that R can also describe reflections.\n\n- The threshold used to compute the F-score metric should be mentioned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VN3Ex1064qy", "reviewer": "Reviewer_SPGB", "summary": "The paper proposes a new method for geometric and structure reconstruction of composite 3D shapes. The main contribution consist of automatically detecting and encoding the repeated parts. This leads to a relevant contribution in comparison with existing work. Moreover, the single parts are accurately reconstracted using neural implicit function. The proposed approach is fully unsupervised. An exhoustive experimental section shown the improvement of the proposed approach in comparison with other methods for both shape recontruction and shape segmentation. ", "questions": "-A better explanation on the relation between the proposed approach and [55] is required to understand the impact of the proposed method (i.e., what happen if the method [55] fails).\n-What is the computational complexity of the work in comparison with other methods? \n-Authors emphasized that they proposed a shape program approach but it seems that  ‘program’ part is not considered in the methodological section and results. Authors should clarify why they consider their work as ‘program’-like (or clarify better the meaning of shape program).  \n\nMinor points:\nIs it possible that scale s_i change for the same part? (i.e., two reapeted parts with different scales).\nIt is not clear the meaning of NM on line 193. Authors wrote that primitives are estimated using [55], but NM is predicted by ProGRIP, please clarify. \n", "limitations": "Limitations are not discussed. For instance authors should declare their strong dependency from [55].", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n-the paper focuses on a relevant problem that is the automatic encoding of composite objects. The capability of detecting repeatable parts is very important bringing a significative step toward the semantic understanding of objects and scenes. \n-the use of deep implicit function enables the method to reconstruct more accurate details of the observed object improving the geometric representation of the observed parts  (differently than other method that capture only a coarse approximation of shape such as 3D boxes).\n-the proposed composite encoding pipeline is well designed by properly combine existing methods with novel parts. \n-  experiments are well organized and shown promising results in comparison with other methods.  \n\n*Weaknesses*\n-It seems that the proposed work strongly believes on the results of [55]. In this sense the proposed work can be considered as an extension or a refinement of [55]. Authors should better explain this point.\n-The proposed approach seems computational demanding, authors should discuss and evaluate the computational complexity of their work.\n-The learning part is not fully novel since the main parts are based on alredy available neural architectures (for box estimation [55], and implicit function estimation [33, 38, 7]).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Sb_z-g6MqKn", "reviewer": "Reviewer_rmbs", "summary": "The paper introduces a novel unsupervised framework for representing repeatable parts as shape programs with part-based implicit functions. The authors propose to utilize implicit functions to represent parts, which improve the representation capacity of shape programs. Meanwhile, a matching-based unsupervised learning objective is devised to learn the shape program from unannotated shapes. Experimental results show the effectiveness of the proposed method in shape reconstruction fidelity and semantic segmentation for chairs, tables and airplanes of the ShapeNet dataset. ", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ Both the problem addressed in the paper and its proposed solutions are innovative and interesting.\n+ The proposed matching-based unsupervised training objective is sound.\n+ The experiment shows quite marginal performance improvement.\n+ The paper is well-organized and easy to follow.\n\nWeaknesses:\nI don't really have any major concerns; some minor concerns are as follows:  \n1. Since part-based implicit shape representation is used, it would be better to compare the training and testing complexity between the proposed method and the baseline, such as BSP-NET.\n2. Does the proposed method require a canonical orientation of shapes.\n3. the authors should provide qualitative and quantitative results on more categories, such as car and lamp.\n4. It would be good if the authors could conduct more experiments to show the expressiveness of the new shape representation. For example:\ni) It would be interesting to explore the feature space to validate if the method can accomplish smooth interpolation and extrapolation of shapes. \nii) Can the new shape programs be adapted for single-view 3D reconstruction?\n5. Are the trained models category-specific or a single universal model is trained for these 3 categories? How do you make sure the comparison is fair in Tab. 1. All baselines are trained in the same experimental setting.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept