Sequential Brick Assembly with Efficient Constraint Satisfaction
Authors: Seokjun Ahn
Jungtaek Kim
Minsu Cho
Jaesik Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: combinatorial problem
brick assembly

Abstract: We address the problem of generating a sequence of LEGO brick assembly with high-fidelity structures, satisfying physical constraints between bricks. The assembly problem is challenging since the number of possible structures increases exponentially with the number of available bricks, complicating the physical constraints to satisfy across bricks. To tackle this problem, our method performs a brick structure assessment to predict the next brick position and its confidence by employing a U-shaped sparse 3D convolutional network. The convolution filter efficiently validates physical constraints in a parallelizable and scalable manner, allowing to process of different brick types. To generate a novel structure, we devise a sampling strategy to determine the next brick position by considering attachable positions under physical constraints. Instead of using handcrafted brick assembly datasets, our model is trained with a large number of 3D objects that allow to create a new high-fidelity structure. We demonstrate that our method successfully generates diverse brick structures while handling two different brick types and outperforms existing methods based on Bayesian optimization, graph generative model, and reinforcement learning, all of which are limited to a single brick type.

Under review as a conference paper at ICLR 2023
SEQUENTIAL BRICK ASSEMBLY WITH EFFICIENT CONSTRAINT SATISFACTION
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle the problem of assembling LEGO bricks, which can be considered as an instance of combinatorial optimization problems. Such a problem is challenging. since the number of possible structures increases exponentially with the number of available bricks due to complex physical constraints between bricks. To solve this problem, our method assesses brick structure to predict the next brick position and its confidence by employing a U-shaped sparse 3D convolutional network. convolution filter efficiently validates the physical constraints in parallelizable and scalable manner and effectively allows us to process different brick types. To generate novel structure, we devise sampling strategy to determine the next brick position by considering attachable positions under the physical constraints. Instead of using handcrafted brick assembly datasets, our model is trained with large number of 3D objects that enable to create new high-fidelin structure. We demonstrate that our method successfully generates diverse brick structures while handling two different brick types and outperforms existing methods based on Bayesian optimization, deep graph generative model, and reinforcement learning, all of which consider assembling single brick type. To show the validity of our method, elaborate studies on various assembly scenarios are also presented.
INTRODUCTION
Most real-world 3D structures are constructed with smaller primitives. A broad range of studies have tackled an interesting assembly problem such molecule generation (Ertl et al. 2017 Neil
al.
2018 You al. 2018), building construction (Talton et al. 2011 Martinovic & Van Gool]
2013
Ritchie et al. 2015). and part assembly generation (Sung et al. 2017] [ee et al. 2021 Jones
ctt al. 2021p. However, if primitive unit (e.g., LEGO bricks) is used to construct 3D structure, thought of as an instance of combinatorial optimization problems. In particular, the search space
of the primitive assembly problem increases exponentially as a search depth increases: given n primitives with i possible combinations, the search space increases O(n²). In addition, supposing that a constraint between unit primitives should be satisfied, the problem becomes more challenging.
In this work, tackle the problem of assembling LEGO bricks into 3D structure. The generation problem of brick assembly is defined as a sequential dreision-mmking process that sequentially ap- pends new brick by determining the brick type, position, and direction of the new brick. Moreover, a brick to assemble has to satisfy the physical constraints related to the disallowance of overlap, no isolated bricks, and LEGO-like brick connections; see Section 2 for the details of such constraints. Compared to general 3D generation methods such as previous work (Wu et al.) 2016 Gadelha et al., 2018 Achlioptas al.) 2018). the advantages of sequential brick assembly as generation method are that it can generate a structure in open space and provide instructions to build the structure.
Several attempts have been conducted on the generation of sequential brick assembly by utilizing Bayesian optimization (Kim et al. 2020)) deep graph generative model (Thompson et al. 2020)) and reinforcement learning (Chung et al. 2021). Those methods are capable of assembling x 4 LEGO bricks with consideration of the physical constraints. However, the previous literature has several limitations. A Bayesian optimization method proposed by Kim et al. (2020) requires heavy computations due to their iterative optimization process for each brick position. [Tommson et al. (2020) propose to use masks to filter out invalid actions along with their method, but the utilization of masks degrades assembly performance. To predict valid action, Chung al. (2021) propose

---

Under review as a conference paper at ICLR 2023
Table I: Comparisons to the previous approaches and our method in terms of unit primitives, algo- rithms, target conditioning, and constraint satisfaction. SA stands for sequential assembly. Note that our approach does not provide guidance (images or larget shapes) when generating a new shape.
Method
Unit Primitives
Algorithm
Targel Conditioning
Construint Satisfaction
SA with BavesOpt (Kim 2020)
Single type
Bayesian optimization
Target volume
Subsampling
DGMLG (Thomponn of al 20
Single type
Graph generative model
Class label
Masking
Brick-by-Brick (Chung ai
2021
Single type
Reinforeement learning
Single- or multi-view images
Supervised learning
BrBCS (Ours)
Two types
Supervised learning
Incomplete target volume (only for the completion task)
Convolution operations
to employ a neural network, which fails to predict proper moves perfectly. More importantly, these methods share common limitations: they only consider using single brick type (i.e., 2x4 brick) due to their inherent design choices or technical difficulties, and inevitably become slower in validating the constraints as the number of bricks increases due to exponentially increasing search spaces.
To tackle the limitations above, we devise a novel brick assembly method with a U-shaped sparse 3D convolutional neural network utilizing a convolution filter to validate complex brick constraints efficiently. By sampling procedure proposed in this work, our method can create diverse assembly sequences of structures in a training stage, which makes us create training episodes. Also, we use the sampling procedure to generate high-fidelity brick structures. In addition, convolution filter to validate the physical constraints enables our method to be casily parallclizable and scalable to processing different brick types. Henceforth, we refer to our method as sequential Brick assembly with Efficient Constraint Satisfaction (BrECS).
We carry out two scenarios of sequential brick assembly: completion and generation tasks. A com- pletion task is to assemble bricks from a given partial structure, and a generation task is to create a brick structure from scratch; see Section 5 their details. In the completion task, BrECS achieves 30.5% higher loU scores than the best performing baseline on average (as reported in Table while ours is eight times faster than the baseline. Also, in the generation task, BrECS performs the best with 117% higher average classification scores than the next best one (as reported in Table
In summary, the contributions of this work are as follows:
We propose novel generation model for sequential brick assembly, which validates physi- cal constraints with a convolution filter in parallel and generates high-fidelity 3D structures;
We propose a U-shaped 3D sparse convolutional network that can be trained with a voxel dataset, which enables us to assemble high-fidelity structures by exploiting a popular 3D dataset, i.e., ModelNet40 (Wu et al. 2015):
We show that our model successfully assembles two different brick types, i.e., 2x4 and 2x 2 LEGO bricks, in various experiments.
We will release our codes and datasets upon publication.
RELATED WORK
3D Shape Generation. For 3D shape generation, point cloud and voxel representation are widely used. Various methods have been proposed to generate point cloud, including methods using varia- tional auto-encoders (Gadelha al. 2018 generative adversarial networks (Achlioptas et al. 2018), and normalizing flows (Yang et al. 2019). On the other hand, voxel-based 3D shape generation has been studied. [Wu et al. (2016) propose to use a generative adversarial network to generate voxel occupancy. A method, which is suggested by(Choy et al.) (2019), generates 3D shapes on voxel grids with a fully convolutional neural network that includes efficient sparse convolution. Zhhng et al. (021) propose method that generates high-quality voxel-based shapes by iteratively applying 3D convolutional neural networks. A point cloud means that points on the cloud are continuous, border- less, and (possibly) infinite. In contrast, voxels similar to pixels coordinates are discretized and limited. Our method employs a voxel-based generation model to generate brick-wise scores since candidates for brick positions are discrete, and the number of bricks is finite.
2

---

Under review as a conference paper at ICLR 2023
Sequential Part Assembly. Similar to the sequential brick problem, sequential part assembly problem also shares common difficulties to the combinatorial optimization problem, i.e, expo- nentially increasing search spaces. Ghasemipour et al. (2022) aim to assemble multi-part objects and tackle the problem with large-scale reinforcement learning and graph- model architecture. (Sung al.] (2017) assemble incomplete 3D parts with a part retrieval network and position pre- diction network. The retrieval model is trained with an embedding network, so corresponding parts have similar representations in a lowddimesioonal feature space. Huu et al. (2020) propose a model that sequentially moves a piled box into another container with different shapes. Their approach has new container. They tackled the problem by
to predict box removal orders and box positions representing previous boxes into graphs and reinforcement learning with rewards considering phys- ical constraints in the new container. contrast to these methods, we tackle the sequential brick assembly problem, which involves identical bricks to create 3D structures.
Sequential Brick Assembly. By following previous literature (Kim et al. 2020 Thompson et al,) 2020 Chung et al. 2021), we sequentially assemble given LEGO bricks into 3D object by adding on bricks one by one. Unlike generic 3D object generation methods, e.g., previous literature (Wu et al. 2016; Achlioptas al. 2018), these sequential assembly approaches have to consider partic- ular physical constraints that are introduced by available connections between two adjacent LEGO bricks and the disallowance of brick overlap. As discussed in the work (Kim et al. 2020), these constraints encourage us to accentuate the nature of combinatorial optimization since there exist a huge number of possible combinations in the presence of complex physical constraints.
3 SEQUENTIAL BRICK ASSEMBLY PROBLEM
Formulation. Similar to the reference by Chung et al. (002), ) new brick has to be connected to one or (possibly) more bricks of previously assembled bricks. It implies that the position of the next brick can be determined as a relative distance from a certain brick directly connected to the next brick. We denote this indicator brick as a pivot brick. Eventually, after choosing the pivot brick, we determine a relative position from the pivot brick so we can place the new brick on the position determined. To choose pivot brick and relative position effectively, we must consider physical constraints and scores for every possible brick position. As the number of bricks increases, determining the pivot brick and the relative position becomes challenging because of exponentially increasing successive brick positions and complex physical constraints. Consequently, previous methods struggle to generate fine structures, and they tend to use a smaller number bricks. Furthermore, while the recent work (Kim al. 2020: Thompson assembly with 2 x bricks, we tackle 2 x 4 and 2 2 bricks with a more efficient framework. The summary of each approach is shown in Table
al. [020 Chung et al., (021) only focuses sequential
Constraints. We should take into account three physical constraints in the brick assembly problem: () bricks should not overlap each other (NO-OVERLAP); (ii) all bricks of the current structure should be connected to each other, and no isolated bricks exist (NO-ISOLATION) (iii) a new brick must be directly attached to the upper or lower position of other bricks (VERTICAL ASSEMBLE). However, validating such constraints is not trivial. To overcome these difficulties, as presented in Table [ Kim et al. (2020) sample a subset of available next brick positions. Thompson et al. (2020) mask out invalid positions by validating all possible positions, and Chung et al. (2021) train an auxiliary network that is trained to validate positions.
PROPOSED APPROACH
We explain four steps of our method, which are illustrated in Figure 1 To sum up, we generate a brick structure under physical brick constraints by repeating four steps of the next brick position score computation (Section 1.1.1). invalid position exclusion (Section 4.1.2). pivot brick sampling (Section 4.1.3). and relative brick position determination (Section 4.1.4). The neural network for next brick position score computation (utilized in Section 4.1.1) is the only learnable component of our framework. The details of the training procedure and loss function for the proposed network are
described in Section 4.2 The details of the procedure are presented in Algorithms I and 2
3

---

Under review as a conference paper at ICLR 2023
Next brick position predictor (Sec. 4,1.1)
B Voxel-wise scores
UeN
Next brick position selection (Sec. 4.1.3, Sec. 4.1.4)
Brick-wise scores
B;
Apply brick sized convolution filter
+
Ct-1 Validated brick scores
brick
using
Ve 1 Hisdeld
2x4
Brick-wisc validity
Apply brick sized
convolution filter
Brick position validator (Sec. 4.1.2)
Figure 1: Illustration of the proposed efficient constraint satisfaction method with convolution filters for sequential brick assembly. U-Net and symbol denote a U-shaped sparse 3D convolutional neural network and glement-wise multiplication, respectively.
4.1 EFFICIENT CONSTRAINT SATISFACTION
In this section, we propose novel method to tackle the challenge of satisfying the following physical constraints:
NO-OVERLAP, which validated by using convolution operations; NO-ISOLATION and VERTICAL ASSEMBLE, which are satisfied by following the brick assembly formulation with pivot bricks and relative brick positions, presented in Chung et al. (202T). By borrowing the concept of constraint satisfaction (Tsang 1993), which is the problem of finding solutions that satisfy predefined set of constraints, our method is designed to establish an approach to efficient constraint satisfaction for sequential brick assembly.
4.1.1 PREDICTING NEXT BRICK POSITIONS
Given a voxel representation of structure at step f, which is denoted as B, E {0.1) 1]axaxa where a the size of 3D space, we first feed the voxel representation B, into a U-shaped sparse 3D convolutional neural network, inspired by the reference (Choy et al. 2019h: B' U-Net(B:). in order to capture global and local contexts effectively and retain the same dimensionality. Due to its pyramidal feature extraction structure, the U-Net extracts robust features understanding multi- dimensional contexts. We validate that the U-Net effectively extracts important contexts and thus improves overall performance compared to other neural networks in Table 5 Moreover, expect that our neural network produces a likely complete or potentially next-step 3D structure, which is represented by probability of voxel occupancy. Note that the network parameters in this U-Net are only the learnable component in our framework BrECS. The U-Net gives guidance on natural shapes. In this section, we assume that we are given pretrained U-Net, and the training procedure is described in Section 4.2
After obtaining B' Raxxaa score for next brick positions A4+1 E Raxaxa is computed by xd, sliding a convolution filter K E xd, x1 across B/: A++1 = B' * K, where * is convolution operation. Note that we match the size of A to the size of B, by applying zero padding. In particular, the size of the convolution filter is the same as the brick size we assemble wa X d, where we and di are the width and depth of the brick, respectively so that we can determine the scores over all the possible positions of the next brick by aggregating the corresponding voxels where the convolution filter is applied. For example, we use 2 x 4 bricks, the size the convolution filter is 2 x x 1. Moreover, K is always initialized training stage to aggregate wyds voxels equally with a single convolution operation. In the case of assembling structure with brick types, use 2r convolution filters of the same size with brick types and repeat the above process for each convolution filter. In particular, for each brick type, we employ two convolutional filters (i.e., Wb X d, x and de wh 1) to consider brick rotation.
tensor filled with without updating its values in
4.1.2 FILTERING OUT INVALID BRICK POSITIONS
As shown in Figure I to predict the validity of the next brick positions, brick-sized convolution filter is applied to B, V, = B, * K. The filter K is identical to the filter used computing the score for the next brick positions. After applying the convolution filter across B, all possible brick
4

---

Under review as a conference paper at ICLR 2023
x22
+2x
+2x4
(a) 61 bricks attachable.
(b) 23 bricks attachable.
+2
+-2 4
x2
x
(c) 33 bricks are attachable.
(d) 19 bricks are attachable.
Figure 2: Illustration of brick structures and their possible brick positions. The three grids on the left of the bricks show the validity Ve+1 of each brick position the bottommost red brick for brick type 2 2, 2x 4, and rotated 2 x 4. The pixels in a red rectangle indicate attachable brick positions on the bottommost red brick without considering overlap. The gray pixels indicate that overlap will occur a new brick is attached to the corresponding position.
positions are determined if the value of the position of interest zero, which means that overlap exists within the brick positions. As visually presented in Figure 2 the validity of each position in terms of overlap and the number of attachable positions with respect the brick of interest are readily determined by applying convolution filter of the corresponding brick type.
Using these two branches: computing A,+1 with the U-shaped network and V., we can define validated score for next brick positions C++1- Each entry of C++1 is equal to its score for the next brick positions filtering out invalid brick positions due to overlaps. Formally, we describe C++1 of assembling wa d, brick on previously assembled brick of size wp X dp as follows:
C++1 = 1 1og Vi.jAelul (V)) @ Ar41
1 [B Vijjkkjaa (B, * K) @ (U-Net(B,) * K),
(1)
where liv.lg =0 kkjja an indicator function, K Iw is the same convolution filter as in Section 4.1.1 and # is convolution operation. Then, we assemble structure by iteratively placing a brick on one of the positions with non-zero scores; the details of how the position is determined are described in Section 4.1.3 Since the calculation of the masked score tensor C++1 uses convolution operations, we efficiently compute with modern GPU devices. Specifically, the validity check of (64, 64, 64) voxels without parallelization takes 4.7 seconds. On the other hand, the validity check (64, 64, 64) voxels with our method takes only 0.0021 seconds, which implies that this design serves 2,270 times faster inference the validity check. In the next subsection, describe the detailed sampling procedure for the next brick positions.
4.1.3 SELECTING PIVOT BRICKS BY SAMPLING
To place a single brick, we need to choose one of the previously assembled bricks attach the new brick to make the entire brick assembly connected - as mentioned in Section we refer to it as pivot brick. The motivation of our method select a pivot brick is that pivot brick with higher sum of attachable brick scores should be more preferable than one with lower sum of attachable brick scores, rather than choosing a pivot brick that is connected to a position with the highest score of C2+1. Since neural network tends to memorize training samples and their assembly sequences, choosing a position with the highest score fails to create novel structures. To avoid a deterministic approach based C.+1, alter method to select a pivot brick into probabilistic method by sampling. To compare the number of attachable positions, we define pivot score Tisk of the pivot of (i, k)) aggregate scores of attachable positions:
44(W"111 -+(
Tisk
Σ
Σ
Cimrs
I=i- ([w 1-1) ( (EEL'--)DE(RRIAAA
where w/ =bb de and wp x dp the size of a previously assembled brick. After computing pivot scores, determine the most probable pivot brick using arg max operation.
2
5

---

Under review as a conference paper at ICLR 2023
However, mentioned above, employ sampling strategy determine a pivot brick: (a, b, c) P. where [p]afk
Tijk
Piyk
Σ E)l.m.n) Tima
(3)
Our sequential procedure for pivot brick predictions inspired by Monte-Carlo tree search (MCTS) (Coulom 2006). MCTS evaluates possible actions by expanding search tree with Monte- Carlo simulations and back-ups. Similar to this, our method also evaluates pivot brick candidates by aggregating their attachable brick scores, which can be understood
search tree with depth 1. Our method utilizes the U-shaped sparse 3D convolutional network to predict possible brick positions, which analogous policy network of the recent MCTS method (Silver al. 2016), which employs a neural network to predict prior distributions for search tree expansion.
4.1.4 DETERMINING RELATIVE BRICK POSITIONS
After choosing a pivot brick by following method described in Section 4.1.3 a relative brick position to the pivot brick is determined to complete the brick placement. Possible relative brick positions (I, y. 2) of assembling a W/ d, brick on a previously assembled brick of size wp x dp are inherently integer-valued positions satisfying following conditions:
tbb Wp
Wb + up
db + de
du + d2
-15 x
1,
1 y
1, 2E 1-1,13 (4)
2
2
2
By considering the conditions described in Equation 4 we choose the relative brick position (z, 2) with the highest score C,+1. 1. brick must be attached to the pivot brick we add scores of every attachable bricks in Equation 2 In this step, we do not employ sampling based method due to its poor empirical assembly results.
4.2 TRAINING THE SCORE FUNCTION
Similar to language modeling (Mikolov et al. 2010 Sutskever et al. 2014) and reinforcement learn- ing (Sutton & Barto, 2018), our model also predicts next brick position sequentially, which makes problem challenging. To train such a prediction model, pair of ground-truth state transition is required as a training sample. However, a final voxel occupancy is only available as ground-truth information. It implies that we cannot access intermediate states explicitly. By replacing B', i.e., the output of the U-Net, with ground truth voxel occupancy, we generate an assembly sequence [Bo. Bi, Br-1] following the procedure in this section. Generated sequences are unique and diverse, since the stochasticity injected from the sampling strategy introduced in Section 4.1.3
Since there exist numerous possible sequences to assemble bricks to a certain 3D structure, a single- step look-ahead with a pair of contiguous states, i.e., (B,, 1, is not enough to model practical assembly scenarios. In addition, the training becomes unstable even though the training pairs are slightly changed. To address these issues, we train our sequential model to predict k-step look- ahead state using pairs of states at step /. and step /. k; i.e., (B., B.++). From now, we call this technique sequence skipping.
We use voxel-wise binary cross-entropy to train model. By restricting the voxel prediction using a sigmoid function and minimizing the voxel-wise binary cross-entropy, our model learns to predict valid voxel-wise probabilities of the Bernoulii distribution. To sum up, train our sequential prediction model as follows:
1. We generate sequence of brick assembly [Bo, B B,.... .., By) by running brick assembly method with a ground-truth voxel occupancy in a training dataset;
2. We generate multi-step pairs from [Bp, Bi,..., By] in sliding-window fashion, i.e., {[B,, T=++
3. We train a transition function Po (B B,) with {(B, B and the voxel-wise binary cross-entropy.

---

Under review as a conference paper at ICLR 2023
Table 2: Quantitative results of the completion of brick structures. An asterisk after method name denotes that partial or full ground-truth information is given to the corresponding model.
Methods
IoU (1)
% valid (1)
Inference Time (sec., ()
airplane
table
chair
avg.
airplane
table
chair
avg.
airplane
table
chair
avg.
BayesOpt
0.145
0.206
0,233
0.194
100.0
100.0
100.0
100.0
1.2066
1.11e6
1.05e6
1.12c6
Brick-By-Brickk
0.455
0.440
0.434
0.443
12.0
7.0
16.0
11.7
305.6
1502.4
2785,2
1531.1
DGMLG
0.315
0.269
0.271
0.285
0.0
1.0
0.0
0.3
237.3
340.0
473.0
350.4
BrECS (2xx4)
0.571
0.564
100.0
100.0
100.0
36.3
143.9
110.4
BrECS (2x 4 2x
0.399
0.594
0.541
0,578
100.0
100.0
100.0
100.0
73.8
224.1
279.0
192.3
To decorrelate the time steps of training pairs in batch and shorten training time, employ buffer throughout training to store training pairs similar to Zoang et al. (202I). The detailed training process is described in Algorithm 2 of the appendix.
Inference for Sequential Brick Assembly. In an inference stage, we determine the next brick positions by following aforementioned procedure, given By. We provide different form of By depending on a task, i.e., completion and generation, and sequentially generate B, until a terminal intermediate state, i.e., incomplete brick structure, as
step T. For completion task, we Bg- For generation task, we sample an initial brick position from a discrete uniform distribution, i..., (x, y, U([-2, 2) Then, we assemble bricks on a zero-centered voxel grid of size (64, 64, 64) and use the voxel occupancy of the initial brick position sampled as Bo.
5 EXPERIMENTAL RESULTS
We demonstrate that our model generates diverse structures with high fidelity satisfying physical brick constraints in the experiments the completion of brick structures (see Section and the experiments on the generation of brick structures with distinct brick types, e.g., 2x4 and 2x2 LEGO bricks (see Section 5.2) Also, thoroughly validate each component of BrECS in ablation studies.
To generate ground-truth assembly sequences and the training pairs based on the ground- truth sequences,
Dataset.
the ModelNer40 dataset (Wu et al., 2015). 3D meshes in the dataset converted into (64, 64, 64)-sized voxel grids, and then they are scaled down to 1/4 of the original size reduce the number of required bricks. Other details are described the supplement.
Metric. For the completion task, we use intersection over union (IoU) to evaluate the performance:
B
Jokk
B
lOU( B())
(5)
E EE E B link
B
where ((1), (2) E Raxaxa Along with IoU, we measure the ratio of valid bricks in structures of interest. In addition, we utilize class probability of target class, which is the softmax output of the target class, in experiments on the generation of brick structures. The probability of the target class is measured using a pretrained classifier with the Mode/Net4O dataset reported.
Baseline Methods. We compare the brick assembly performance of our method against Bayesian optimization (Kim et al. 2020)), denoted as BayesOpt, Brick-by-Bricc (Chung al. 2021). denoted as BBB, and deep generative model of LEGO graphs (Thompson et al. 2020), denoted as DGMLG, in Tables 2 and BByesSOpt optimizes brick positions to maximize intersection over union (IoU) between assembled shapes and target shapes. BBB learns to assemble bricks given multi-view images of target structures using reinforcement learning. Following its formulation, we also provide three images (top, left, and front) of target structures in a test dataset. DGMLG generates structure by utilizing the graph representation of brick structures and a deep graph generative model. Note that our approach does not provide any guidance (image target shape) to produce new structure.
5.1 COMPLETION OF BRICK STRUCTURES
We conduct our method on a completion task for a sequential brick assembly where unseen partial structures are given. To measure the completion performance brick assembly problems, first
7

---

Under review as a conference paper at ICLR 2023
Table 3: Quantitative results of the generation of brick structures with distinct brick types. Asterisk denotes that partial or full ground-truth information is given to the corresponding model.
Class probability of targel class (1)
valid (1)
Methods
airplane
table
chair
avg.
airplane
table
chair
avg.
BayesOpt*
0.039
0.043
0.069
0.050
100.0
100.0
100.0
100.0
Brick-By-Brick*
0.430
0.042
0.032
0.168
3.0
2.0
3.7
DGMLG
0.228
0.023
0.027
0.093
0.0
0.0
0.0
0.0
BrECS
0.250
0.356
100.0
100.0
BrECS (2 2
0.447
0.229
0.419
0.365
100.0
100.0
100.0
100.0
BO
BBB
DGMLG
Ours
table
chair
Figure 3: Qualitative results brick assembly generation. Each row corresponds to specific base-
lines. Our method generates detailed brick assemblies without brick overlap. Best viewed in color.
assemble LEGO bricks using voxel occupancy in a test dataset. Then, we remove fraction of bricks assembled without losing connectivity between bricks and provide it as start state Bg. Each model trained with training dataset and then complete brick structures from the start states. We compare the completion performance by measuring IoU between ground-truth voxel occupancy and the complete brick structure. addition, we report valid assembly ratio and inference time.
As shown in Table 2 our method outperforms the other three baseline methods in terms of IoU. The results show that our method assembles high-fidelity brick structures compared to other methods, despite exhaustive constraint satisfaction. Moreover, alongside BayesOpt, but ours is also the best in inference time. additionally test our model with distinct brick types by appending 2x2 bricks after assembling 2x4 brick type first. The performance of our method further improved by using two brick types since different brick types can fill brick positions more densely and express the fine aspect of structure in consequence.
method performs the best in valid ratio
5.2 GENERATION OF BRICK STRUCTURES
As our method is generation model, our brick assembly model can generate brick structure that belongs to a particular category in a training dataset. To compare the quality generated structures semantically, train a classifier over voxel grids with small number of 3D convolution layers using the ModelNet40 dataset; the detailed architecture of the classifier is presented in the appendix. Given pretrained classifier over voxel grids, measure the class probability of generated brick structures for a target class. To feed the voxel grid of generated structure into the classifier, we match the voxel grid size of generated structure with the size of the training dataset for the classifier.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Results of ablation study on completion from intermediate states of chair category in the ModelNet40 dataset. Our model without ablation performs better than ablated variants. The perfor- mance is further improved with the additional 2 2 brick type.
IoU
% valid
oo bricks
Inference time (sec.)
BrECS (2 4; default)
0.534
100.0
96.7
151.0
BrECS (2 4 + 2)
0,541
100.0
125,9
279.0
BeFCS (wl validity check)
0,441
0.0
141.6
335.8
BrECS (w/o sequence skip)
0.437
100.0
59.3
6.0
The quantitative results are presented in Table 3 Our method achieves the highest score in terms of class probabilities, The results indicate that our method generates a semantically meaningful brick structure compared to other methods. We also emphasize that the semantic generation quality can be improved using additional 2 x 2 brick types. Brick structures with distinct brick types are generated by assembling 2 x bricks first and then x 2 LEGO bricks. This performance gain is led by the additional improvement on structure refinement that cannot be filled with single brick type.
5.3 ABLATION STUDY
We verify that each of our components in BrECS improves generation performance in the ablation study. Specifically, we compare the ablated variants of our method in terms of completion perfor- mance. Similar to the previous completion experiments, we complete a brick structure from the intermediate state of the unseen structure in test dataset. The results of the ablation study presented in Table g Following the previous experiments, creating brick structures with two brick types improves the performance. validity check using convolution filter plays critical role in constraint satisfaction. Interestingly, the validity check also improves generation performance, i.e., IoU, since we train the model with valid assembly sequences. Moreover, our model performance degrades significantly without sequence skipping the number of bricks added reduces. It supports our claim in Section 4.2 that the sequence skipping encourages our model BrECSto predict diverse brick positions. More detailed ablation studies can be found in the appendices.
6 CONCLUSION
We proposed brick assembly method that efficiently generates high-fidelity brick structures sat- isfying physical constraints. Our model sequentially assembles bricks by repeating pivot selection and relative position selection. To aggregate brick scores and validate brick positions, we employ a convolution filter. Previous brick assembly methods cannot escape from exponentially increasing search spaces since it is the nature of combinatorial optimization problems. However, our method is efficient where the number of bricks increases as convolution operations are easily parallelizable on show that our method performs better than prior methods, as shown in
GPU devices. In addition, quantitative and qualitative results. To our best knowledge, we are the first to attempt to use multiple brick types, i.e., 2 4 and 2 bricks, to generate novel brick structures sequentially.
Potential Applications. Our model can be thought of as an instance of combinatorial optimization problems, which a challenging problem in computer science. Moreover, our model can be adopted to solve a real-worll problem such as building design and product assembly with an industrial robot. Considering such problems, we would like emphasize that they suffer from coping with invalid actions or physical constraints. Thus, ours can help to solve such problems by utilizing our ideas.
Limitations. We utilize only two types of bricks for the demonstration, but we have designed that our model can use any rectangular bricks without an explicit definition of connection rules between different brick types, unlike the work (Kim et al. 2020). Hence, we can increase the number of brick types for sequential brick assembly. Future work will employ more diverse brick types and even irregular brick types since non-rectangular materials are common real-world assembly scenarios. Another limitation is that our performance degrades when we use two brick types simultaneously. In the case of using 2 xx4 and 2x2 2 brick types, smaller bricks are used more frequently compared to scenario with only 2 x 4 brick type. We presume that smaller 2 x 2 bricks in-between reduce attachable positions, resulting in overall performance degradation. Also, we would like to consider physical stability, which is essential for real-world applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3D point clouds. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-tempore convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE International Conference Com- puter Vision and Pattern Recognition (CVPR), 2019.
Hyunsoo Chung, Jungtaek Kim, Boris Knyazev, Jinhwi Lee, Graham W Taylor, Jaesik Park, and Minsu Cho. Brick-by- brick:: Combinatorial construction with deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Proceedings of the International Conference Computer and Games, 2006.
Peter Ertl, Richard Lewis, Eric Martin, and Valery Polyakov. In silico generation of novel, drug- like chemical matter using the lstm neural network. arXiv preprint arXiv:1712.07449, 2017.
Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3D point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Seyed Kamyar Seyed Ghasemipour, Satoshi Kataoka, Byron David, Daniel Freeman, Shixi- ang Shane Gu, and Igor Mordatch. Blocks assemble! Learning to assemble with large-scale structured reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), 2022.
Ruizhen Hu, Juzhan Xu, Bin Chen, Minglun Gong, Hao Zhang, and Hui Huang. Tap-net: Transport- and-pack using reinforcement learning. ACM Transactions on Graphics (Proceedings of SIG- GRAPH ASIA 2020), 2020.
R. Kenny Jones, David Charatan, Paul Guerrero, Niloy J. Mitra, and Daniel Ritchie. ShapeMOD: Macro operation discovery for 3D shape programs. ACM Transactions on Graphics, 2021.
Jungtaek Kim, Hyunsoo Chung, Jinhwi Lee, Minsu Cho, and Jaesik Park. Combinatorial 3D shape generation via sequential assembly. In Neural Information Processing Systems Workshop on Machine Learning for Engineering Modeling, Simulation, and Design (ML4Eng). 2020..
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Youngwoon Lee, Edward S Hu, and Joseph J Lim. IKEA furniture assembly environment for long- horizon complex manipulation tasks. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2021.
Andelo Martinovic and Lue Van Gool. Bayesian grammar learning for inverse procedural mod- eling. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Proceedings of the Conference of the International Speech Communication Association (Interspeech), 2010.
Daniel Neil, Marwin Segler, Laura Guasch, Mohamed Ahmed, Dean Plumbley, Matthew Sellwood, and Nathan Brown. Exploring deep recurrent models with reinforcement learning for molecule design. In International Conference on Learning Representations Workshop, 2018.
Daniel Ritchie, Ben Mildenhall, Noah D. Goodman, and Pat Hanrahan. Controlling procedural modeling programs with stochastically-ondeeed sequential monte carlo. ACM Transactions on Graphics, 2015.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
10

---

Under review as a conference paper at ICLR 2023
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
Minhyuk Sung, Hao Su, Vladimir G. Kim, Siddhartha Chaudhuri, and Leonidas Guibas. Comple- mentMe: Weakly-supervised component suggestions for 3D modeling. ACM Transactions on Graphics, 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jerry O Talton, Yu Lou, Steve Lesser, Jared Duke, Radomir Mëch, and Vladlen Koltun. Metropolis procedural modeling. ACM Transactions on Graphics, 2011.
Rylee Thompson, Ghalebi Elahe, Terrance DeVries, and Graham W Taylor. Building lego using deep generative models of graphs. Neural Information Processing Systems Workshop on Machine Learning for Engineering Modeling, Simulation, and Design (MIAEnng) 2020.
Edward P. K. Tsang. Foundations of constraint satisfaction. Academic Press, 1993.
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning prob- abilistic latent space of object shapes via 3D senerative-adversariil modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie. and Bharath Hariharan. PointFlow: 3D point cloud generation with continuous normalizing flows. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Jiaxuan You, Bowen Liu, Zhitao Ying. Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. Advances Neural Information Processing Systems (NeurIPS), 2018.
Dongsu Zhang, Changwoon Choi, Jeonghwan Kim, and Young Min Kim. Leanning to generate 3D shapes with generative cellular automata. Proceedings of the International Conference on Learning Representations (ICLR), 2021.

---

Under review as a conference paper at ICLR 2023
APPENDIX
In the appendix, we cover the detailed descriptions missing in the main manuscript.
A IMPLEMENTATION DETAILS OF OUR METHOD AND BASELINE MODELS
A.1 BRECS
To train 3D voxel generation efficiently, we utilize MinkowskiEngiin (Choy et al. (2019) and its 3D sparse convolution operation. We train our model with fixed learning rate of 5e-4, Adam opti- mizer Kingma & Bа 2015), batch size of 32, sequence skipping with step size l = 8, buffer size of 1024, and maximum number of bricks of 150. The input size of our model is (64, 64, 64) and output size is (64, 64, 64). We trained the model until reaching 100k steps.
A.2 BAYESIAN OPTIMIZATION
We employ Bayesian optimization to tackle sequential brick assembly problem. We follow the setup proposed by Kim et al.] (2020). The number bricks are limited to 160 bricks at most. Every brick position is optimized by maximizing IoU between currently assembled and ground-truth structures. We would like to emphasize that we have to provide ground-truth voxel information to this strategy.
A.3 BRICK-BY-BRICK
We use Brick-by Frick-by-Brice as a baseline method and compare brick assembly performances. We follow the model architecture and training setup of the model described in (Chung et al.. (2021). We also would like to emphasize that we have to provide partial ground-truth voxel information to this strat- egy, the method requires images of ground-truth shape assemble target shapes. The number of bricks are limited to 75 bricks at most, which is the same as original setup, due to the excessively increasing memory requirements of the model. In inference, the sequence of generation is halted when the newly placed brick violates brick constraints, following the environment reset condition of the method.
A.4 DEEP GENERATIVE MODEL OF LEGO GRAPHS
We solve brick assembly generation problem using Deep Generative Model of LEGO Graphs and compare the performance against ours. We follow the model architecture and training setup of the method described in Thompson et al. (020). To train the graph generation model, we need graphs of target shapes. To this end, we assemble target shapes with ground truth voxel shapes. Then we convert the brick assemblies into graphs by representing bricks as nodes and direct connections between bricks as edges, following (Thompson et al.) 2020). We train the model for 200 epochs.
B BRICK ASSEMBLY PROCESS
We describe brick assembly process in Algorithm i
C TRAINING
We describe brick assembly model training procedure in Algorithm 2
D DETAILED ARCHITECTURE OF U-NET
We design our score prediction network inspired by Choy et al. (019). Detailed model architecture illustrated in Figure
12

---

Under review as a conference paper at ICLR 2023
Algorithm Brick Assembly Process
Input: Voxel of structure at current time step B, assembled brick position list and brick size
Wh x d,
Output: Pivot brick position (a, b, c), relative next brick position (a;, y.
Initialize convolution filter E IR xd, x1 with 1, pivot score list T = Φ. and possible relative
connections. ''' with values satisfying the conditions of Equation 4 Calculate C using Equation for each pivot (a, b,c) in do Calculate Tabe using Equation T T--UU ]
Di Section 4.1.2
end for Sample pivot (a, b,
Tate Σrer" T Relative next brick position (z, v, arg (ab.ee) Catxbty,etz
Section 4.1.3
Section 4.1.4
Algorithm 2 Brick Assembly Model Training
Input: Dataset D, mini-batch size M,, sequence skipping value
Output: Brick assembly model parameter
Generate ground-truth brick assembly sequences (() and store in D3 using voxel shapes x; E DD
Initialize buffer B with (B 2y, 0) where x, D, B D.
repeat
==
Sample and remove mini-batch {B 0:7 T4 4)) from B for
(Bo,7 Z4, tx)
mini-batch do
C C + log po (B,
(B(a)
t (B())
(0)
if 4 k = 7 then
Push (B
(B 0:r: Ry 0) into B where 2j ~ D. B
0. D,
else
B - U ( 0..44444 1)}
end if end for
+ 0 + 77 89 until convergence
E DETAILED ARCHITECTURE OF VOXEL CLASSIFIER
We train shallow voxel classification network to compare assembly generation quality semanti- cally. We add softmax layer at the end of the network to make output logit to be a distribution. Detailed network structure described in Figure 5
F ABLATION STUDY ON MODEL ARCHITECTURES
We compare different model architectures for score prediction model. Specifically, we compare U-Net and fully convolutional networks. For fair comparison, volutional filters. We present the results of generation and completion on Table S We find that the U-Net performs better than fully convolutional network in both generation and completion. This is due to the robust feature extraction from pyramidal structure U-Net.
the same number of con-
G MODEL. PERFORMANCE WITHOUT STOCHASTICITY
We carry out ablation study on the effects of stochasticity a brick position selection process and compare completion and generation performance against the original model. The quantitative results are presented Table 6 Stochasticity improves generation performance as expected. Interestingly, completion quality also improved by stochasticity. In the completion task, remove half of
13

---

Under review as a conference paper at ICLR 2023
Jc3e5, 128
Cond Con-DownBlock 256
ComcBlock St 256
SparoC Con'l 3olck 236
Com-Blockd)) 3xhds 128
Com ComBlockk 3a3s3 256
Spanee Can// 3xhx2 256
Spunet Can Scche, 64
Con willlek 128
ComDownnflesk 256
ConvDowcn Block 64
Con-dloei 3x303 64
Convi Conv www.DooBBllek 12%
Corv Blocklc heh3. 128
Sponed Can't 3x333 96
54565,
ConvBlocke2) 3x3as, 96
Cora Com New-DovnBlees 96
Spars Comy NA07, 32
Com-DwnBBlcc 96
ConeBlock 3t.52 96
Spunce cevv 363c3
Sjunee
Spers
Figure 4: Detailed model architecture of U-Net.
SpineCom 3520,, 128
SpineCom Johck $12
Spanet cm N6.3
Spanet Com 5538
Spanoe 1ele,,
MaaPal
Madheel
Masholl
MC 312
sollmx
Spana
Spunad
Figure 5: Detailed model architecture of voxel classifier.
the bricks and the model has to complete partial assembly results, which lose the majority of their information. Stochasticity may help to infer its original shape by making multiple candidates, Wan al. [202T) also report that stochasticity improves FID score of image completion task if they masked out large area of original images.
H PREVENTION OF SQUEEZING-IN BRICKS
A brick position be predicted in the middle of bricks. We call this brick squeezing-in brick. A Squcezing-in in brick is casily prevented with simple modification to our pipeline. As illustrated in Figure 6 employed additional x 1x convolutional filters after obtain brick-wise validity. Then we check whether the resultant validity score is 2. The value 2 means the brick position and one of neighbor brick positions (top or bottom) is empty.
ANALYSIS ON DIVERSITY OF GENERATED SHAPES
We measure precision and recall to measure the diversity of generated shapes.
We define precision as how much of real shape distribution can be made using generated shape dis- tribution and recall as how much generated shape distribution can be made using real shape dis- tribution by following the work by Sajjadi al. (2018). Distance between distributions is measured in the feature space encoded by pretrained voxel classifier. The results are depicted in Figure 7 We find that the diversity is different for each class. Airplane class lacks diversity the class have
14

---

Under review as a conference paper at ICLR 2023
Table 5: Results of ablation study model architectures. U-Net architecture consistently outper- forms fully convolutional network with the same number of convolutional filters in both reconstruc- tion and generation scenarios.
IoU
Class probability of farget class (f)
airplane
table
chair
avg.
sirplane
chair
avg.
(U-Net)
0.571
0.586
0.564
0445 0.208
0.356
BiECS (Fully convolutional network)
0.510
0.399
0.450
0.453
0,075
0,186
0.156
Table 6:
Model performance without stochasticity. Stochasticity improves both generation perfor- mance and completion performance.
Toll (1)
Class prohability target class (f)
Methods
airplane
table
chair
avg.
airplanc
table
chair
avg.
BrFCS (U-Net)
0,571
0,586
0,534
0.564
0.415
0.250
0.404
0.356
BrECS stochasticity)
0,515
0.533
0.509
0,519
0.238
0.104
0.186
high precision but low recall. Table class shows balanced results between diversity and similarity. Chair class has diverse shapes as has high recall but low precision.
J BRICK ASSEMBLY CONSIDERING INTERMEDIATE STABILITY
Humans assemble bricks a way that every intermediate structure is stable as depicted in Figure S Thus, conduct brick assembly generation considering intermediate stability. To compare inter- mediate stability, we introduce a new evaluation metric, called a trajectory stability ratio. The metric measures how many intermediate structures are stable by dividing the number of stable structures by the total number of structures. We consider that structure is stable when the center point of the structure does not move within given simulation steps. We utilize PyBullet to simulate stability of generated shapes. The quantitative results are presented in Table
We measure stability and generation performance of our method and its variants. The variants are the models that (i) start assembling from bottom and (ii) consider intermediate stability. To consider intermediate stability during inference, we exclude unstable actions by simulating stability of pos- sible outcomes. We find that the variant considering intermediate stability shows higher trajectory stability ratio against other variants. This scenario is only conducted sequential brick assembly generation and one step generation method is not appropriate for this case.
K ADDITIONAL QUALITATIVE RESULTS OF OUR MODEL
Additional qualitative results of our model is shown in Figure 9
15

---

Under review as a conference paper at ICLR 2023
B
As43
U-Net
Voxel-wist ise scores
Brick-wise scores
Cies Validated brick scores
Place brick using scores
B,
Apply brick-sized convolation filter
B1s
+
V.
2x4
Apply brick-sized convolution filter K
Apply 1xlx3 convolution filter
Figure 6: Illustration of the proposed method with squeezing-in brick prevention. The additional part is emphasized yellow.
1.0
Arrtlae
table
0.8
Precisionn
0.6-
0.4
0.2
0.50
0.75
1.00
Recall
Figure 7: Result of precision and recall for generated shapes. Distance between distributions is measured the feature space encoded by pretrained voxel classifier.
Figure 8: An assembly sequence where every intermediate structure is stable.
Table 7: Quantitative result of shape generation and trajectory stability. Changed initial brick posi- tion improves stability and stability check further improves stability.
Class probability of target class (t)
trajectory stability ratio
Methods
cup
lamp
avg.
cup
lamp
avg.
(Start from center)
0.543
BrBCS (Start from bottom)
0.046
0.088
0.074
0.069
0.892
0.772
0.846
0.837
BIECS (w/ intermediate stability)
0.024
0.022
0.056
0.034
0.925
0.921
0.936
0.927
16

---

Under review as a conference paper at ICLR 2023
(a) bed
(b) vase
(c) toilet
(d) lamp
Figure 9: Additional qualitative results our model.
17


Reviews:
[{"review_id": "eCnsN2RgpI", "reviewer": "Reviewer_3PGp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a method for sequentially assembling Lego bricks into 3D structures resembling objects from training classes. Efficient GPU-based methods are developed to filter out invalid attachment points and to learn to identify high-value placements. Unlike prior work, the method handles a heterogeneous mix of brick types (a mix of 2x4 and 2x2 bricks is demonstrated).", "summary_of_the_review": "I think this paper does a good job in developing an improved algorithm for sequential Lego brick assembly. That said, there are some weaknesses which I have identified above. Hence, I think the paper, while interesting, does not represent a very significant step forward. If the authors could argue for greater impact of the algorithm, that assessment might change.\n", "strength_and_weaknesses": "The paper is well-written and presents a plausible and efficient solution to the problem. Long assembly sequences with a wide branching factor can be difficult to control, and the presented results look quite good considering this challenge. The method guarantees that the assemblies are valid by construction. The training is also self-supervised and does not appear to use explicit GT construction sequences.\n\nThat said, the (not great in absolute terms) quality of the results does leave one wondering if combining a voxel-based generator with a subsequent (non-ML) step that fits an arrangement of bricks to the voxel pattern would be a better way to generate (still structurally valid) assemblies of Lego bricks, rather than to impose distributional constraints on very long Markov sequences. The presented method is an interesting technical demo of controlling long assembly sequences with tight constraints, but its practical utility in the context of Lego construction seems limited. Autoregressive methods have their place, but in this case it seems like a particularly challenging paradigm to employ, and there don't seem to be obvious takeaways transferable to other sequential assembly problems. The \"Potential Applications\" section mentions building construction and robotic assembly, and in each of these tasks there seem to be easier ways to achieve the end goal.\n\nFinally, even though the paper claims that the assembly sequences are ones humans can actually follow (\"can... provide instructions to build the structure\"), the sequences don't actually take things like phsyical stability and connection strength of intermediate structures into account, so this may not be true in all cases.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WDV1KvbnFS", "reviewer": "Reviewer_sD14", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for the sequential brick assembly problem (SBA). To the best of my understanding SBA is a special case of sequential assembly problem with a finite number of actions (possible brick placements). The proposed approach is based on a combination of a convolutional NN, which is traned to predict the score of possible placements, and convolutions for quickly filtering out infeasible solutions with respect to given physical constraints.", "summary_of_the_review": "While the proposed approach addresses some computational challenges of the problem considered, its presentation should be improved before publication. Little insights are provided on why the technique is effective and what are its limitations. The experimental section disregards important  aspects in this sequential generation, such as the exploration vs. explotation tradeoff.\nAll these aspects require a major revision of the paper. I cannot recommend the current version for publication.", "strength_and_weaknesses": "Strengths:\n- The proposed approach leverages tensorial computations with a significant performance gain\n- The code will be released upon publication\n\nWeaknesses (for details, see Clarity section below):\n- The presentation could be significantly improved \n- Some crucial aspects of the problem are not empirically validated", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IJLi_t8rgT7", "reviewer": "Reviewer_LQmJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a method for sequential generation of Lego models. This addresses two tasks – (i) generating new class-conditional Lego models a priori; (ii) completing partial models. The proposed approach consists of a neural network to predict how good candidate locations for the 'next' brick is, and various checks to ensure constraints on fabricability are fulfilled. The method is demonstrated on shapes from the ModelNet dataset, and shown to perform favorably versus the chosen baselines.", "summary_of_the_review": "As noted above, I believe the tasks can be addressed straightforwardly using existing methods (at very least, these should be included as a baseline). Moreover, the pipeline is not particularly interesting technically, nor do there seem to be many generalisable insights that would be beneficial to the community.", "strength_and_weaknesses": "- the first task considered is generation of models of certain object classes, from Lego. However, this is straightforwardly accomplished by using an existing generative model of ModelNet shapes (outputting any representation that can be converted to voxels), then applying Brickr [Testuz, Eurographics 2013] to convert this to a valid Lego model. That approach already supports many more brick types than the proposed method, and supports all the relevant constraints.\n\n- the second task considered is completing partial lego models of ModelNet classes. This can again be addressed easily using existing techniques – convert to a point cloud, run a point-cloud completion model, convert back to voxels, and run Brickr on the 'new' part, giving the uppermost part of the input model as the first Brickr layer.\n\n- very few qualitative generation examples are given (three examples from two classes) – making it impossible to judge whether generated shapes are in fact reasonable. Those which are given, are not particularly impressive, and it's not specified whether they are curated.\n\n- there is no quantification (based on nearest-neighbors) of whether the method is in fact generating novel shapes, or simply memorising its training set\n\n- the constraints in sec. 3 need more explanation. Do they correctly disallow 'squeezing in' blocks? Why must structure be fully-connected from start (submodules assembled separately then combined are common in real Lego)?\n\n- the overall approach is a complex engineered pipeline, very specific to the task, and most stages of this are (in my opinion) not of great interest to the majority of the ICLR community\n\n- stochasticity in the generative process appears to be introduced only by randomly selecting which of a candidate set of points to consider at each step. This is fine a-priori; however, the set of candidates appears to be determined not by the learnt model directly, but by the number of possible locations that bricks could be affixed to them. Why not let the model learn the distribution over 'next' attachment points? Also, why is the selection of 'pivot brick' artificially separated from choosing which particular location on this brick to attach to?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EWkwiyUTUr", "reviewer": "Reviewer_khxJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that generates high-fidelity brick structures satisfying physical constraints for two tasks: partial structure completion and novel structure generation for a given class of shape. The main contributions are:\n- A parallelizable brick position predictor and validator based on convolution filters, making the inference of the proposed brick sequence generation method efficient on GPU for large brick structures.\n- The proposed method is agnostic to the brick type, with experimental demonstrations of assembling structures using both 2x4 and 2x2 bricks.", "summary_of_the_review": "Though the results seem good, I cannot give acceptance based on the current state of the paper, as there is so much confusion. I look forward to the authors' clarification on important aspects.", "strength_and_weaknesses": "Strengths:\n- Technically sound methodology. The technical flow in Figure 1 is intuitive.\n- Good empirical performance in terms of shape completion accuracy, semantic generation quality, validity, and efficiency compared to baselines.\n\nWeaknesses:\n- The paper is poorly written with confusing logic, and the task definition is unclear until the end of the paper. Such poor clarity is unacceptable for more potential readers. See more details below.\n- The wider impact or applications of the research problem are not clear. Though the authors believe this method can be extended to building design or industrial robots, the proposed method seems highly specific to LEGO bricks and hard to extend.\n\nPaper writing and clarity issues:\n- The abstract does not illustrate which problem/goal specifically you are targeting. Obviously, you are working on brick sequence generation, but is the generation conditioned on certain target shapes or shape classes? From the current abstract, it seems to be that you are generating ANY assembly sequence that satisfies the physical constraints, which confused me because I don't see why you need such a complicated approach to achieve that.\n- The detailed definitions of completion and generation tasks are only vaguely described in Sec 5.1-5.2. In other words, most of the time when I was reading the paper, I didn't have a clear clue of which specific problem the authors were addressing. The terms \"completion task\" and \"generation task\" are frequently used in the middle of the paper without explanation. So please bring the task definitions forward - let people know what you are addressing even before describing the technical approach.\n\nQuestions:\n- Sec 4.1.3 details the pivot brick selection procedure, where the authors propose sampling to avoid being deterministic. This is reasonable for the novel structure generation task. But for the shape completion task, does adding stochasticity help?\n- In Sec 4.2, I am not sure if I understand this correctly: \"By replacing $B_t^{'}$ with a ground-truth voxel occupancy, we generate an assembly sequence ...\". Would you mind elaborating more on this procedure?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject