Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning
Authors: Hyunsoo Chung
Jungtaek Kim
Boris Knyazev
Jinhwi Lee
Graham W. Taylor
Jaesik Park
Minsu Cho
Venue: NeurIPS 2021 Poster (2021)
Keywords: Combinatorial construction
Sequential assembly
Deep reinforcement learning

Abstract: Discovering a solution in a combinatorial space is prevalent in many real-world problems but it is also challenging due to diverse complex constraints and the vast number of possible combinations. To address such a problem, we introduce a novel formulation, combinatorial construction, which requires a building agent to assemble unit primitives (i.e., LEGO bricks) sequentially -- every connection between two bricks must follow a fixed rule, while no bricks mutually overlap. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit volumetric information to the agent. This problem requires a comprehensive understanding of partial information and long-term planning to append a brick sequentially, which leads us to employ reinforcement learning. The approach has to consider a variable-sized action space where a large number of invalid actions, which would cause overlap between bricks, exist. To resolve these issues, our model, dubbed Brick-by-Brick, adopts an action validity prediction network that efficiently filters invalid actions for an actor-critic network. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.

Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning
Hyunsoo Chung*
Jungtaek Kim
Boris Knyazey28
Jinhwi Lee
Graham W. Taylor
Jacsik Park
Minsu Cho
POSTECH
University of Guelph ( jtkimm ac.kr
Vector Institute
4POSCO
Abstract
Discovering solution in a combinatorial space is prevalent in many real-world problems but also challenging due diverse complex constraints and the vast number of possible combinations. To address such problem, we introduce a novel formulation, combinatorial construction, which requires building agent assemble unit primitives (i.e., LEGO bricks) sequentially every connection between two bricks must follow fixed rule, while no bricks mutually overlap. To construct target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead exact and explicit volumetric information to the agent. This problem requires comprehensive understanding of partial information and long-term planning to append a brick sequentially, which leads us to employ reinforcement learning. The approach has to consider variable-sized action space where a large number of invalid actions, which would cause overlap between bricks, exist. To resolve these issues, validity prediction network that efficiently filters invalid actions for an actor-critic network. We demonstrate that the proposed method successfully learns construct an unseen object conditioned on single image or multiple views of a target object.
model, dubbed Brick-by-Brick, adopts action
Introduction
A combinatorial space, typically characterized by discrete variables and their combinations, often induces interesting yet challenging problems such traveling salesperson and minimum spanning tree [20] 6). The main challenges lie in the vast number of possible combinations as well as complex constraints imposed on them. In similar spirit, we suggest novel problem formulation. combinatorial construction, that focuses incomplete target information (i.e., 2D images or multiple views of target object) 25 12], an agent sequentially assembles unit primitives (i.e., LEGO bricks). The proposed formulation is combinatorial since engages repetitive placement of primitives, which leads to a large number of available solutions. Distinct property of our proposed formulation, however, is that the agent must build the solution incrementally by adding on to the partial solution. Specifically, brick, which unit primitive of the object of interest, is placed on discrete space by connecting to one of the previously assembled bricks. In addition, every connection between two bricks must follow a fixed rule while no bricks mutually overlap. Each assembly (i.c., action) executed by the agent is, thus, modeled as selecting one of the feasible connections to place a new brick.
the real-world construction procedure. Given only
The problem we introduce closely depicts how humans understand object and adapt the acquired knowledge to a downstream task. Humans naturally analyze 3D object by picturing its part-by byrt-by-part part
"Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: Analysis of recent studies in terms of state representation, supervision, conditioning, target objects, and action validation. CE and IoU stand for cross-entropy and intersection over union with respect to volumetric comparisons. Direct forwarding (denoted as direct), sampling & checking (denoted as sampling), and our pretrained action validity prediction network (denoted as pretrained) indicate respective strategies that filter invalid actions; see the corresponding section for their details.
Action
Method
State
Supervision
Conditioning
Target
Validation
Hamrick et al.
11
Image
Task desseddenddnnt
N/A
2D
Direct
Bapst et al.
Object/Image
Task- desk-dependent
Object and/or image
2D
Direct
Kim et al.
19
Set
Overlap
Exact target volume
3D
Sampling
Thompson al.
38
Graph
Step-wise CE
One-hot class info.
3D
Direct
Brick-by-Brick (B* ours)
Graph/Image
IoU
Image set of images
3D
Pretrained
decomposition and consequently grasp rich semantic understanding 15 22 In various fields, they utilize an inherent ability to decompose objects to effectively solve challenging tasks such as object classification (17), robot grasp planning 22. and part segmentation [31 28 Likewise, humans exploit this ability to solve the inverse problem combinatorial construction. Given a desired object to be constructed and no strong supervision (i.e., ordered step-by-step instructions), humans can often still manage to build valid target object by carefully planning sometimes improvising, the sequence of actions. Our environment, which corresponds the proposed problem, designed to learn and test such behavior with only partial information of the desired target available to the agent.
Successfully constructing an object in our setup requires comprehensive understanding of incomplete target information with the current structured state of assembled bricks and long-term planning to append each brick efficiently. These requirements, along with the absence of sequence-level supervision, incentivize us to devise a reinforcement learning (RL) approach [3 37]. In this domain, however, we must carefully handle both an indefinite action space and the existence of many invalid actions when applying RL [43. In particular, both defining an action space that varies by the number of assembled bricks and distinguishing an invalid action that would cause overlap with other existing bricks quickly become intractable as more bricks are placed. To resolve the aforementioned issues, our model, dubbed Brick-by-Brice -Brick (B3), adopts an action validity prediction network that filters invalid actions to actor-critic network. In addition to the novel RI. formulation, we use graph- structured representation of the brick combination to interpret the assembling process as a sequential graph generation process.
Overall, we summarize our contributions as follows:
(i) We propose a novel problem formulation, combinatorial construction, that closely resembles a real-world object construction process that engages repetitive placement of components;
iii We design RL agent for combinatorial construction, dubbed Brick-by-Brin by-icick-Birrr- (B²), to effec- tively address both a growing action space and a vast set of invalid actions;
(iii) We implement the corresponding environment based on OpenAl Gym and introduce new novel evaluation scenarios that vary by their incomplete partial target information.
2 Combinatorial Construction
To formulate the combinatorial construction problem, we start by defining a unit primitive that is used to construct a 3D object and an action space that determines where assemble the next primitive.
As a unit primitive, we utilize 2 x 4 brick, which has eight studs and their fit cavities. This design choice yields consistently varying action space, implying that if we add one brick to the current state of brick combination, we can efficiently define the next action space. We want to emphasize that with only six x 4 bricks, primitive does not make our problem a trivial task; instead, every decision of where we place the next primitive can deteriorate the quality of the final result because there exists a plethora of wrong paths.
create 915, 155,703.75 combinations [9]. Accordingly,
choice of the
With our specific choice of unit primitives, we can define an action space for determining the next action and evaluating the future states. However, since every assembly step gradually expands the

---

action space, naive approaches to defining the growing action space are not appropriate for our problem; an action space with redundant actions [43] is not applicable due varying action space. and an action sampling approach [33 is also not suitable due to nominal or invalid actions. Thus, we define a successive action space composed of a two-step decision: (i) choosing a pivol brick and (ii) choosing an offiset from the pivot brick.
Before explaining a pivot brick, we first assume a simplified assembly scenario that follows an Eulerian patt new brick is always placed by connecting to the last assembled brick. This enables us to define finite action space though most actions are infeasible to perform with the Eulerian path. To broaden the search space by generalizing the Eulerian path, one of previously assembled bricks chosen as a pivot brick. Then, B decides an offset from the pivot brick, which describes how the next brick is placed relative to the pivot. Thanks to the homogeneous brick type, the number of available offsets is finite and consistent where we do not consider the validity of such offsets in a certain state for 2 x 4 bricks, there exist a maximum of 92 available offsets.
Due to the disallowance of overlap between bricks, our agent must consider invalid actions among available actions during assembly. Identifying the validity of actions from the current brick combi- nation becomes intractable as more bricks are placed; the complexity of this process is O[ (f²), where Aur is an action space for offsets and / ís the cardinality of assembled bricks at a given step; see the supplementary material for qualitative results on this complexity. Such expensive overhead for validation naturally leads us to adopt an action validity prediction network that learns to identify invalid actions with a single forward pass.
As described in Section D and this section, our problem has interesting but challenging characteristics derived from the assumptions on discrete placement, connectivity rule, disallowance of overlap, and ultimately invalid actions. We, therefore, present a comparison to other existing studies in terms of state, supervision (or reward function), conditioning, target objects, and action validation, as shown in Table I Compared to the previous work [1 3 19 (38], our method B constructs an object in 3D with incomplete target information and pretrained validity prediction component; see Sections 4 and S for a detailed description.
3 Brick-by-Brick
In this section, we briefly introduce the definition of RL and the corresponding framework for combinatorial construction, where an agent places a brick sequentially. We then explain the details of our model that learns to select appropriate actions given only partial information of the desired target so that the assembled 3D object resembles the target. Moreover, to cope with a vast number of invalid actions in the process of assembly, we propose an action validity prediction network. See p Figure for the overall pipeline of our method 3
Definition. In a standard RL framework, there exists an agent that interacts with an environment by iteratively making decisions given an observation of the environment. This follows general decision making procedure of a Markov decision process (MDP), where a transition function satisfies the Markov property, i.e., p(5++ |80; 81 .1,..., 8t; 0t) = p(*++1 1 (8t, at), where 8, and a are a state and an action at timestep I,, respectively.
In our problem setting, we only consider finite horizon MDP formally defined a tuple of (S, P.R. 7). where S = { is a set of states, A = { a set of actions, R : S x A - R reward function, P: : S A + S a transition function, and 7 E [0, 1) a discount factor. The goal of the agent is to learn a policy π (a, (st) that maximizes the expected future cumulative reward. We introduce our detailed MDP formulation for combinatorial construction in the following sections.
3.1 Problem Formulation
Given target information T, the agent aims construct a 3D target object T. by assembling bricks sequentially, one brick for each f-th step. Each f-th brick is represented by its pose (x,,d)). where x, E Z is the center coordinate of the brick in a 3D space and de E {0, 1} denotes one of two possible directions, meaning that its longer axis is aligned along either x axis or axis in the 3D space.
It is path that visits all the vertices without revisiting the edges visited before.
3

---

MLP.
GNN,
1/7
MLP
MI.P;
E
GNN
V
Action Validity
E
Prediction Network
CNNNr
MLP.c
Figure 1: Overview of our proposed method B A state input 8t =G,, T) is embedded and passed through CNNs, GNNs, and MLPs to predict an action a, that consists of a pivot brick indicator a! and offset from the pivot brick at Moreover, action validity prediction network helps filter invalid actions from the action space where we determine the next brick to assemble. The red brick in both actions of at and off indicates the chosen brick. See Section for the details.
Target Information. We are given as target information T a set of binary images of a target object, which may correspond incomplete and partial information about the target. In practice, this setup with partial information is more realistic than accessing full information of 3D target shape. Our task thus is to create a sequence of unit bricks by inferring a target object from abstract information in combinatorial manner.
State. Each 1-th state 8t of the MDP is represented by a tuple of a directed graph G. composed
of t bricks and target information T, i.e., 8t (G₁, T). The graph is defined G, (V,, Ex). where V = {vi) is set of & bricks, i.e., Vi = (x;, di) E 24, and E, = {ee 18 Jif=1 is set of the offset vectors between two connected nodes, i.e., Gij (x; xj d; @ d,) E Z4. Note that nodes are connected by edges according to sequential actions and relative offsets in pose are used for edge features in order to induce translational and orientational equivariance. Since all the edges bi-directional, we omit the arrows when displaying graphs Figure
Action. We define a successive action space of choosing the pivot brick first and the corresponding
offset next. Formally, with bricks assembled, we define an action a = (a," piv at where aB is to select pivot brick and of is to select an offset with respect to the pivot brick. The pose of the next brick is then (x"iv + Ax, (driv + Ad) mod 2). where (xriv driv) and (Ax, Ad) are determined by at and af respectively. In choosing actions a,"" and ae we exclude invalid actions: (i) choosing pivot brick near which no additional brick can be placed and (ii) choosing an offset for the next brick that overlaps with existing bricks. To select a valid action in the action space with vast number of such invalid ones, we predict invalid actions in advance using the action validity prediction network and exclude them from action candidates preemptively; we mask out all the probabilities of invalid actions, re-normalize the distribution over actions or, and sample an action from the distribution.
Transition Function. Given state 8 and action a,, our transition function p(st+1|ss, az) designed to determine the next state 81+1 by deterministically updating based on dr, The node of the new brick V 1 created so that V VUU(v++1). The edges between the new brick and existing bricks in physical contact via studs are created so that E,+1 E, U (eonc)))iENN U (((++1(()) HeN, where N6+1 denotes the set of bricks in direct contact with the new brick Ve+1- As the result, the graph in the state st+1 is updated to G (V++1, E(+1).
Reward Function. In contrast to the tasks where a direct reward evaluation is readily available, not trivial to quantify the object assembled by combinatorial construction, especially, in the context of graph generative model (26). To mitigate such an issue, exploit the property of a voxel

---

representation. Given a desired object, we first create voxels in closed space and determine the occupancy of voxels with a target object, after normalizing the bottom center of voxels. We then transform the combination of currently assembled bricks into the occupancy of the voxels and measure the overlap with the target object:
Al, T vol C T
Col(C
vol C, TT) vol UT)
where C,, Ct-1, and T are the occupied voxels at timestep t, timestep respectively. In addition, vol(-) is function that measures a volume. The step-wise reward function is then AloU if new brick overlaps at least 50% with the occupied voxels of target object and 0 other- wise. Consequently, our agent will learn sequential placement of bricks to construct the object, without explicit supervision by maximizing Equation (I). as will be described in the subsequent section.
and desired target,
3.2 Sequential Construction
In this section, we describe how we process G2 and T, which comprise a state, with different types of neural networks such convolutional neural networks and graph neural networks. The overview of this construction procedure is illustrated in Figure
Node and Target Embeddings. Given state 8t (Gr, where G, Et), we first use convolutional neural network (CNN) to extract features z from the target:
z = (NN
(2)
the partial information is given as a set of images, the feature is obtained by first applying CNN to each image separately and then concatenating the outputs to single vector.
For node and edge features, an MLP embeds them with the target feature z:
(0) (0) MLP..(V;,.2 z]), and %j MLP: z)).
for all i,j E {I...., f), where [.] and (0) denote concatenation and the first layer, respectively.
Equations 2 and can be viewed as pre-processing inputs to feed in a graph neural network (GNN). Inspired by [4] we apply a variant of graph networks (GNs) in which a global graph feature
omitted. At the f-th layer of GNNs, we update edge features, aggregate the messages for each node, and update node features:
(f+1) 1j
MLP!!) (Iv! v (2) .eeee
(4)
m (0)
Σ
aggregate
(o(tt1).
(5)
JeN:
((+1) = MLP(() (Iv! , m²))
(6)
where M; is the neighborhood nodes of v,, and aggregate(-) :) is the aggregation function that computes a message for each node by aggregating the features of its neighboring nodes. Note that MLP,.. MLP, MLP(!) and MLP(() have their own learnable parameters.
Action Selection. In order to enrich representations for predicting ae, we employ two separate GNNs: GNN piv and GNNor. with L layers in total, to produce sets of node embeddings, V. and Voolt for pivots and offsets:
Vie = GNN (V: EE( (0)), and V. = GNN (V. (.EE
where () [v. (0)) li=i and F (ee (0)
Yiz= are the sets of node and edge features obtained by Equation 3 Each layer of both GNNs updates node and edge features using Equations (4, (5).
and (6) Finally, the set of node and edge features, V. and V. along with a target feature z are
used to decide the next action of and at
p(a."t) o (MLP (IV? z1)). and p(a,") = =o (MLL (lv..21))
(8)
where a is softmax function and vgff is the node feature selected by the index i" of o,
5

---

DSS0 Tran
XLE Trars
90
GNN Trsee
MEEF Trams
G208 Trems
MLP Town
BD
GSN Tox
XIEP Toe
CNX Toxt
SEEP Toee
DO
CCN Text
MLP Text
2000 1800s
2000
20/00
200
1000 1abo 20000 2000o
X008
S10a
2000e
Thamipp
Tiurstop
Thundop
(a) MNIST Class
(c) ModelNet Airplane
Figure Episode return vs. timestep different setups. The return values training and test episodes are reported by repeating 3 times with different seeds.
Action Validation. To tackle the issue of a vast number of invalid actions, we propose to learn an action validity prediction network. Previous work adopts direct forwarding B 38 or sampling checking [9]. In the direct forwarding approach B 38), the agent directly selects an action without
any prior processing. This typically suffers from early termination of episodes since the sequence of actions terminates with deadlock soon the agent selects invalid action. In the sampling & checking approach [19], valid actions are collected by sampling random set of actions and checking the validity for each of them directly. This requires high cost of iterative checking and the actions are limited to a small collection of checked actions.
In contrast to these approaches, we train separate module to predict a large set of valid actions, enabling the agent to sufficiently explore the action space. We train a GNN, of which the node- wise output predicts validity confidences for its candidate actions, i.e., candidate actions for the corresponding brick. The structures of both pivot and offset validity prediction networks are identical to the networks described in Equation (8). but the last activation is sigmoid function and no target feature z is used. Importantly, these networks can be pretrained by the ground-truth action validity, which is obtained from andomly-assemble objects, and such a pretrained network can be used in training an actor-critic network, without re-training.
Note that, if our action validity prediction network fails to filter invalid actions and one of such actions is selected by the agent, the corresponding episode terminated. Unlike the direct forwarding approach, training the agent with the action validity prediction network does not suffer from early termination the validity prediction network masks out the majority of invalid actions.
Training. We adopt the proximal policy optimization (PPO) algorithm (35] which is one of the state-of-the-ars on-policy algorithms. In particular, we optimize the clipped surrogate objective over parameters 0:
(())= = E (0) -1 c).,,)A..)
1-00),,1-,,,1
where ri (0) probability ratio between the previous and updated policy, clip a clipping function between the second and the third arguments, and A, an advantage function (34). To calculate the advantage of a state 8;, our model employs a value network, MLPpar([[vV)) (V.of))) z)), where (u-)) a global average function over instances in a given set.
Experimental Results
We evaluate our image-conditioned 3D object assembly of B in three scenarios: (i) MNIST con- struction, (ii)
object construction, and (iii) ModelNet construction.
For evaluation metric, measure the episode return loU between the constructed object and the desired target at the end of each episode:
vol CN TT) vol(CN UT)
loU T)
(10)
where N is the total number of bricks and the voxel representation of the target object. The maximum number of bricks to be placed depends on T and is pre-defined. After exhausting the brick

---

Tuxt Pulle Bat
Thp Pwsteeree
61.7
Boar Jorr)
Aiady (Pecacal
12. Clid)) Chrraraa
Car (H60) iPotaa
Eoine Utstttro Bate
Bood
Talos Poettor
(a) Pivot ROC
(h) Pivot PR
(c) Offset ROC
(d) Offset PR
Figure 3: ROC and PR curves for the action validity prediction network. All the results are measured using the test dataset of randomly-assembel objects.
Figure 4: Qualitative results MNIST construction. Our model is separately trained each class, and target images are unseen during training.
budget, we terminate the episode and compute the final IoU. Unless otherwise specified, we report the average performance over 3 random seeds, each of which is trained for fixed timestep budget.
To show the effectiveness of method, first analyze our action validity prediction network and test other baseline methods and B different scenarios. In all construction tasks, we compare B to the MI.P-based model where all GNNs are replaced with MLPs, and to the Bayesian optimization- based approach (BO) that sequentially optimizes the step-wise reward in terms of IoU to search for an optimal construction sequence. As presented in Table BO uses both training and test target objects because it cannot assemble an object with only partial information. For each scenario, the episode returns of the BO model averaged over both training and test datasets. In addition, we compare B to the supervised learning method trained with the cross-entropy loss between predicted and ground construction. Since sequence-level supervision is used, the performance of the supervised learning method is only measured on the test dataset. Details can be found in the supplementary material.
volumetric information for
sequences, specifically, in the aaddmml-asseebbl
object
Action Validity Prediction Network. We test our action validity prediction network by creating training and test datasets. The training dataset is composed of 200,000 brick combinations and their ground-truth action validity, and the test dataset is composed of 30,000 brick combinations and their ground-truth action validity. Importantly, the range of the size of brick combination in the training dataset is [1, 20). and the range in the test dataset [1, 30.. While the test dataset contains larger brick combinations than the training dataset, the performance of the action validity prediction network in
7

---

1
f
T
1
(a) Target images
(b) Constructed objects from three viewpoints
Figure 5: Qualitative results on randomly-assembled object construction. Targets obtained from the test dataset, and each row represents pair of target information and constructed object.
terms of precision and recall is satisfactory, predicting reliable validity confidences even for actions in unseen ranges, as presented in Figure Our GNN outperforms MLP as well as GNN baselines, which do not have either node features edge features. In addition, the pretrained network, which is reusable in different scenarios, is slightly better than the action validity prediction network that is jointly trained with training episodes. See the supplementary material for more details of the action validity prediction network.
MNIST Construction. each episode, an agent provided with an image from the MNIST dataset and is provided to create a 3D object resembling the digit target. Similar to [3), binarize the MNIST dataset to convert a real-valued number to either 0 or 1, for brevity of the calculation of IoU. To create 3D target object with 2D MNIST image, original size and then expand an image along the channel dimension, order to assemble with 2 bricks, i.e., image of size 28 28 is transformed to 3D object of size 14 14 4. Furthermore, we limit possible offset candidates to 6 different types of which the values according to the channel of
first rescale image to half of the
dimension fixed to the same value. Training and test datasets the ten classes the binarized MNIST and splitting images from that class. In particular, 500 images from one of available classes are chosen, further divided into 400 samples for training dataset and 100 samples for test dataset.
established by choosing
Due space limit, only report the average reward performance class 0 in Figure 2(a) Results for other classes are available in the supplementary material. The gap between the training and test sets episode returns is marginal, which implies that model generalizes to unseen targets well. In addition, both our training and test results are better compared to both the BO and MLP-based models. We visualize constructed objects for the test dataset of classes 0 through 9 in Figure 4 More qualitative results are also provided the supplementary material. In general, our agent successfully agent catches distinctive details
constructs objects of unseen instances. This be understood in the target information and reflects in a construction process.
Gandomly-Assemin Object Construction. Contrary to the experiments of MNIST construction, this task focuses on building objects that require more than one image fully understand the structure. Accordingly, the agent must construct object with three x images from different viewpoints, which are initially given as the target information. Objects in this experiment artificially generated

---

Airplane 1
(b) Airplane
Airplane
(d) Monitor I
(e) Monitor
Table
Figure 6: Results ModelNet construction. Targets obtained from the test dataset. The first three panels and the last panel of each figure show the target images and the constructed example.
by connecting bricks at random. The total number bricks is also chosen randomly between 10 to 15. For available offset types, only utilize connection types that occupy four more studs and only allow a new brick to be placed on top of the pivot brick so that the resulting target becomes more distinguishable;
the supplementary material for the details. In this scheme, the total numher of offsets 16. Finally, we sample images from 800 target objects for a training dataset while 200 target objects used for test dataset.
As shown in Figure 2(b] our model achieves a return comparable to the MLP-based model while slightly lower return compared to BO. We conjecture that this is due to the relatively small number of used bricks compared to other test suites. Nevertheless, our agent is still capable of associating the target object in 3D space from multiple images illustrated in Figure [S] Our model learns to assemble bricks a way that the resulting object successfully matches the mitiallyy given images, whereas the model trained with the supervised learning method does not generalize unseen images of the test dataset. This clearly demonstrates the effectiveness of applying RI. compared to learning with sequence-level supervision.
ModelNet Construction. Similar randomly-assembler object construction, the agent is given 3 images of realistic target from the ModelNet dataset [40]. To adjust the difficulty of this task, we find an object that is able limit the maximum budget of bricks to less than 60. As result, we choose airplane, monitor, and table categories from the ModelNet dataset. Moreover, we use offset types that connect with four or more studs and allow a new brick be placed above and below the pivot brick. This task is the most challenging due to the excessive search space compared to MNIST construction and randomly-assembler assembled object construction, and assesses the agent's 's ability generate a real-world target object.
We provide training and testing curves for the airplane class in Figure 2 material for more results for monitor and table categories. Despite the difficulty raised from the large search space and long sequence. the result demonstrates that B is capable of learning the construction process of real-world objects. BO with a limited budget achieves lower return compared to B since the search space is too big explore with limited computation. By comparing the constructed object can be observed that B generally captures overall shape
the supplementary
to images of the desired target in Figure of the target. Though, our model tends struggle to catch fine- finegrrined details such as wings of the airplane or legs of the table. However, for example, a table with only three legs (i.e., one leg missing) or two legs
diagonal direction would perfectly match with the same three input views. implies that if we provide more complete target information than three different views of target object, our agent can construct the target object more precisely; see the supplementary material for more detailed discussion on this limitation.

---

5 Related Work
In this section, we briefly cover related work the task solved in this work.
3D Object Generation. Following the studies on 2D object generation, e.g., the work by Doso- vitskiy et al. 8 3D object generation is often achieved in holistic manner [39 [ [4] 41). They generate a 3D object in a single feed-forward operation which limits exploitation of intermediate structures. Compared to these holistic methods, Kim et al. [9 propose an approach to tackle combinatorial assembly problem by using Bayesian optimization (5) not a learning- bbandd method. Unlike [1)] Thompson et al. (38) apply graph-structure generative model in the combinatorial 3D
object generation task, by training to match a ground- -trnthhhruthhhh sequence of LEGO bricks.
Graph-based Reinforcement Learning. A common learning-based technique for creating a graph use one of various models such as recurrent neural networks 42 adversarial networks (7], variational autoencoders [18 33 [6], and Transformers [30]. Unlike these directions, Simm et al.
[3 solve this idea of generating molecules with RL such that generated molecules are placed in the Cartesian coordinate. The key difference to our work is that we sequentially generate 3D shapes which have a much larger search space. Furthermore, Bapst et al. [3] show that an RI. agent can learn physical construction in 2D space, and utilize rich visual information well as graph-structured representation, order define a state and search space.
Inage-Conditioned Reinforcement Learning. Ganin et al. 100 propose an approach to synthe- sizing a program for 2D images when either an unconditional or conditional scenario is assumed. Their method generates an image by sequentially conducting an action in the MuJoCo environment. Nair al. [29] propose goal-conditioned RL approach the goal of which is provided by visual information. Huang et al. (16) suggest method to paint palette with stokes where target image is conditionally given, by utilizing an RL algorithm.
Brick Assembly Optimization. The brick assembly problem satisfying pre-defined constraints is longstanding topic in computer graphics. Lee et al. [23] tackle LEGO brick layout optimization by a genetic algorithm. Similarly, Luo et al. [27] solve building sculptures safely with LEGO brick by stability aware refinement. Zhang et al. 44 propose the method for generating component peppondet--bdddd
building instructions that is safe based on segmentation models. Kozaki et al. (21) tackle a similar problem of brick assembly from images with the octree voxel-based model. This line of research tends to focus on directly utilizing the voxel representation of target object, instead of extracting the representation of target object from incomplete partial information.
6 Conclusion
We have proposed novel problem formulation, combinatorial construction, which asks an agent to construct an object sequentially. We adopt by defining state as graph-structured representation to express assembled bricks and their connections, where incomplete target information is given. In ad- dition, we develop algorithm with a successive action space that does not depend on the number of bricks already constructed and a reward function that measures overlap between a target object and the current state. Through extensive experiments, we demonstrate that our method can construct objects in various construction scenarios, and provide the analysis our action validity prediction network.
Acknowledgments and Disclosure of Funding
This work was supported by the IITP grants N0.2009-0001196: AI Graduate School Program POSTECH, No. N0.22021-002063 AI Innovation Hub) funded by Ministry of Science and ICT, Korea and Samsung Electronics Co., Ltd (I0201200 108000208-011 0008007822-01) JK carried out this research during research internship at the Vector Institute, and JK and HC equally contributed this work. BK was funded by NSERC and the Ontario Graduate Scholarship. GWT and BK also acknowledge support from CIFAR and the Canada Foundation for Innovation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute: http:://www www.orenstititteee wtt.tuuee.eiiieeee. ai/#parrteess We also thank Hyeonwoo Noh for helpful discussions.
10

---

References
[1] Achlioptas, O. Diamanti, I. Mitliagkas, and L., Guibas. Learning representations and genera- tive models for 3D point clouds. In Proceedings the International Conference Machine Learning (ICML), pages 40-49, Stockholm, Sweden, 2018.
[2] Alcotti and S. Caselli. Part-based robot grasp planning from human demonstration. In Proceedings of the International Conference on Robotics and Automation (ICRA), pages 4554- 4560, Shanghai, China, 2011.
[3] V. Bapst, A. Sanchez-Gonzales C. Doersch, K. Stachenfeld, Kohli, P. Battaglia, and J. Ham- rick. Structured agents for physical construction. In Proceedings of the International Conference on Machine Learning (ICML). pages 474, Long Beach, California, USA, 2019.
[4] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, C. Gulcehre, F. Song, A. Ballard, J. Gilmer, G. Dahl, A. Vaswani, K. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y. Li, and R. Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806v:1806.012611 2018.
[5] E. Brochu, V. M. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:
2010.
[6] Q Cappart, D. Chételat, E. B. Khalil, A. Lodi, C. Morris, and P. Veličković. Combinatorial optimization and reasoning with graph neural networks. arXiv preprint 2tl::2002.054444 2021.
[7] N. De Cao and T. Kipf. MolGAN: An implicit generative model for small molecular graphs. arXiv preprint arXiv: 1805. 1973, 2018
[8] A. Dosovitskiy, J. T. Springenberg, M. Tatarchenko, and T. Brox. Learning to generate chairs, tables and cars with convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4)6692-705, 2016.
[9] S. Filers. The LEGO counting problem. The American Mathematical Monthly, 12365)415--266 2016.
[10] Y. Ganin, T. Kulkarni, Babuschkin, S. M. A. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. In Proceedings of the International Conference on Machine Learning (ICML). pages 1675, Stockholm, Sweden, 2018
[11] Hamrick, K. Allen, Bapst, Zhu, R. McKee, J. B. Tenenbaum, and P. W. Battaglia. Relational inductive bias for physical construction in humans and machines. In Proceedings of the Annual Conference of the Cognitive Science Society (CogSci), pages 1773-1778, Madison, Wisconsin, USA, 2018.
[12] W. Han, S. Xiang, C. Liu, R. Wang, and C. Feng. SPARE3D: A dataset for SPAtial REasoning on viee line drawings. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 14690-14699, Virtual, 2020.
[13] J. He, M. Ostendorf, X. He, J. Chen, J. Gao, L. Li, and L. Deng. Deep reinforcement learning with combinatorial action space for predicting popular Reddit threads. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1838-1848, Austin, Texas, USA, 2016.
[14] Henzler, N. Mitra, and T. Ritschel. Escaping Plato's Cave using adversarial training: 3D shape from unstructured 2D image collections. In Proceedings of the International Conference on Computer Vision (ICCV), pages 9984-9993, Seoul, Republic of Korea, 2019.
[15] D. D. Hoffman and W. A. Richards. Parts of recognition. Cognition, 18(1-3):65-99) 96, 1984.
[16] Z. Huang, W. Heng, and S. Zhou. Learning to paint with model-based deep reinforcement learning. Proceedings of the International Conference on Computer Vision (ICCV), pages 8709-8718 18, Scoul, Republic of Korea, 2019.

---

[17] D. Huber, A. Kapuria, R. Donamuukala, and M. Hebert. Parts-based 3D object classification. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 82 89, Washington, District of Columbia, USA, 2004.
[18] Jin, R. Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Proceedings of the International Conference on Machine Learning (ICML), pages 22322,33,, Stockholm, Sweden, 2018.
[19] J. Kim, H. Chung. J. Lee, M. Cho, and J. Park. Combinatorial 3D) shape generation via sequential assembly. In Neural Information Processing Systems Workshop on Machine Learning for Engineering Modeling, Simulation, and Design (MLAEng), Virtual, 2020.
[20] B. Korte and Vygen. Combinatorial Optimization: Theory and Algorithms. Springer, edition, 2018
[21] T. Kozaki, H. Tedenuma, and T. Maekawa. Automatic generation of LEGO building instructions from multiple photographic images of real objects. Computer-Aided Design, 70:13-22, 2016.
[22] B. Lake, R. Salakhutdinov, J. Gross, and J. Tenenbaum. One shot learning of simple visual concepts, In Proceedings of the Annual Conference of the Cognitive Science Society (CogSci), pages 2568-2573. Boston, Massachusetts, USA, 2011.
[23] S. Lee, J. Kim, J. W. Kim, and B-R. Moon. Finding optimal LEGO@ brick layout of voxelized 3D object using a genetic algorithm. In Proceedings of the Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 1215-1222, Madrid, Spain, 2015.
[24] Li, O. Vinyals, C. Dyer, R. Pascanu, and P. W. Battaglia. Learning deep generative models of graphs. arXiv preprint
2018.
[25] Y. Li. K. Mo. L. Shao, M. Sung, and L. J. Guibas. Learning 3D part assembly from a single image. In Proceedings of the European Conference on Computer Vision (ECCV), pages 664 682, Virtual, 2020.
[26] R. Liao, Y. Li, Y. Song, S. Wang, C. Nash, W. L. Hamilton, D. Duvenaud, R. Urtasun, and R. Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, Vancouver, British Columbia, Canada, 2019.
[27] S-J. Luo, Y. Yue, C-K. Huang, Y-H. Chung, S. Imai, T. Nishita, and B-Y. Chen, Legolization: Optimizing LEGO designs. ACM Transactions on Graphics, 34(6):222:1-2222
2015. [28] K. Mo, S. Zhu, A. X. Chang. L. Yi, S. Tripathi, L. J. Guibas, and H. Su. PartNet: A large-scale benchmark for fine-grained and hicrarchical part-level 3D object understanding. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 909-918, Long Beach, California, USA, 2019.
[29] A. Nair, V. Pong, M. Dalal, S. Bahl, Lin, and S. Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pages 9191-9200, Montreal, Quebec, Canada, 2018
[30] C. Nash, Ganin, S. M. A. Eslami, and P. W. Battaglia. PolyGen: An autoregressive generative model of 3D meshes. In Proceedings of the International Conference on Machine Learning (ICML), pages 7220-7229, Virtual, 2020.
[31] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 77-85, Honolulu, Hawaii, USA, 2017.
[32] R. R. Salakhutdinov and I. Murray, On the quantitative analysis of deep belief networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 872-879, Helsinki, Finland, 2008
[33] B. Samanta, A. De, G. Jana, V. Gómez, P. Chattaraj, N. Ganguly, and M. Gomez-Rodrigues NeVAE: A deep generative model for molecular graphs. Journal of Machine Learning Research, 21(114):1-33, 2020.
12

---

[34] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint (tlt::5506.24338 2015.
[35] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1rxXiv:107..000 1707.06347 2017.
[36] G. Simm and J. M. Herááneez-Lobato. A generative model for molecular distance geometry. In Proceedings of the International Conference on Machine Learning (ICML), pages 9949-89556 Virtual, 2020.
[37] G. Simm, R. Pinsler, and M. Hemándezz-Loato Reinforcement learning for molecular design guided by quantum mechanics. In Proceedings of the International Conference on Machine Learning (ICML), pages 8959-8969, Virtual, 2020.
[38] R. Thompson, G. Flahe, DeVries, and G. W. Taylor. Building LEEOO using deep generative models of graphs. In Neural Information Processing Systems Workshop on Machine Learning for Engineering Modeling, Simulation, and Design (MLAEng), Virtual, 2020.
[39] Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D Processing Systems (NeurIPS), volume 29, pages 82-90, Barcelona, Spain, 2016.
modeling. In Advances in Neural Information
[40] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition (CVPR), pages 1912-1920, Boston, Massachusetts, USA, 2015.
[41] G. Yang. X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan. PointFlow: 3D point cloud generation with continuous normalizing flows. In Proceedings of the International Conference on Computer Vision (ICCV), pages 4541 -4500, Seoul, Republic Korea, 2019.
[42] J. You, R. Ying, X. Ren, W. Hamilton, and Leskovec. GraphRNN: Generating realistic graphs with deep auto-regressive models. In Proceedings of the International Conference on Machine Learning (ICML). pages 7708-5717, Stockholm, Sweden, 2018.
[43] T. Zahavy, M. Haroush, N. Merlis, D. J. Mankowitz, and S. Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pages 3562-3573, Montreal, Quebec, Canada, 2018.
[44] M. Zhang, Y. Igarashi, Y. Kanamori, and J. Mitani. Component- based building instructions for block assembly. Computer- Design and Applications, 14(3):293-38) -000, 2017.
13


Reviews:
[{"review_id": "hh9ZWMzN15L", "reviewer": "Reviewer_cnra", "summary": "This paper proposes a GNN+RL pipeline to use 2x4 LEGO bricks for reconstructing 3D shapes from observation of a single image or a set of images. The authors claim three main contributions: 1) the novel problem formulation of combinatorial construction; 2) the RL agent Brick-by-Brick (B^3) to solve the problem of growing action space and invalid actions in the sequential part assembly process; 3) implement an OpenAI gym environment, plus benchmark and evaluations.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**\n\n(positive) The paper has the technical contribution of proposing the novel problem of combinatorial construction and formulating a GNN+RL pipeline for addressing the task. Also, the authors make contributions at 1) designing the action space to make it not growing with the increasing number of bricks during assembly, by defining the action space as a tuple of <pivot brick, offset>; 2) training a neural network to predict if an action is valid or not, to reduce the growing checking time if not using the network.\n\n(negative) However, I have some major concerns regarding the claimed contributions. Firstly, it is weird to claim \"combinatorial construction\" as the novel formulation proposed in this paper, as this is a general term and many previous works [19, 38] are also essentially solving the combinatorial part assembly problem. Secondly, I don't understand the necessity of training a network to predict invalid actions. If checking for O(|A_off|t^2) for an accurate answer is too computationally heavy, can you simply maintain a voxel-grid logging the brick occupancy and adding a new brick is simply a O(1)-time checking?\n\n**Quality:**\n\n(positive) The method design is reasonable, easy to read, and well-stated. The paper conducts a lot of experiments over different datasets. \n\n(negative) I have many concerns regarding the experiments: 1) it seems weird to do the MNIST experiment as MNIST images are 2D shapes. Why using 3D bricks for reconstruction is an interesting task? Also, it seems that you will remove one dimension and only use a 1x2 or 1x4 2D brick to doing a 2D reconstruction, right? 2) The qualitative result for the Table shape in Fig.  6 is pretty bad. Considering that this is the only result shown for Table in the main paper, is this representing your best result? Can you achieve better results than this? Actually, this may slightly exaggerate my concern why using 2x4 bricks as the only primitive is a good choice, given such bad reconstruction quality for real ModelNet shapes? 3) Can you compare to [19, 38] as these two published papers are so related to the task in this work? In my opinion, it should be not impossible to convert them to work with image conditions. Also since your method also uses 3D ground-truth to provide rewards in Eq. 1,  I think the task settings (input/output/supervision/goal) in this paper are quite similar to [19, 38]. I thus think that a fair comparison is necessary to prove the improvement of the proposed method. 4) In table 1, it's weird to put \"AVN\" as a term for comparison. Do previous works not consider invalid overlapping at all? I don't think this is a big point to claim difference. \n\n**Clarity:**\n\nthe paper is overall well-written and easy-to-read. However, the comparison to [19, 38] is not clearly described, in Sec 1/2/5. \n\n**Significance:**\n\noverall, I think the claimed contributions are not strong to meet NeurIPS bar, or not well-proven to be necessary. Besides, necessary detailed discussions and experimental comparisons to significantly similar works [19, 38] are missing.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes, the authors adequately addressed these issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "gYbahc0JlDW", "reviewer": "Reviewer_n1WS", "summary": "The proposed combinatorial construction is a new formulation for combinatorial problems\nwhich have a vast number of possible combinations. This requires the agent to assemble LEGO\nunits sequentially to build the desired structure based on the 2D target image. The authors use a\nreinforcement learning method with Graph-Neural-Network to solve this formulated problem\nwhich requires a comprehensive understanding of the incomplete input information and long-\nterm planning. The authors also introduce an action validation network to help eliminate\ninvalid actions when the action space increases as the assembly processes. The\nproposed method is evaluated on three types of tasks with different inputs and is compared\nwith three types of baselines: Bayesian optimization-based method, fully-MLP architecture, and supervised learning method.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*:   \nFirst of all, the paper is well organized and written, and the reader can easily follow the logic of the\npaper. The proposed method has a good generalization ability among different types of 2D\ninputs: single view and multiple views. The experiment results show the proposed RL based\nmethod outperforms the fully-MLP based method which indicated the effectiveness of using\nGNN architecture for embedding the input data. From the qualitative results on both MINST\nand ModelNet, it is shows that the proposed promise a plausible 3D construction with the\npartial information.\n\n*Weakness*:\n1. The proposed formulation for this combinatorial problem is interesting, but I don’t see it\nwould be easily transfer to real construction problems, especially when the authors claims that they\n\"focus on the real-world construction procedure\". Because it is hard to use the\naction validation network to eliminate the invalid actions when the action space will be\ncontinuous in real world problem.\n2. The action validation network is pre-trained when used in the experiments. It is\nnecessary to show how the performance would be when the action validation network is\ntrained jointly with the reinforce pipeline.\n3. It is also interesting to show some ablation study on this action validation network.\n4. Would the performance be improved when using a more complete 3D model as input for\nthe proposed method? Is it reasonable to assume only images as input, given that 3D models are available?\n5. There are also some concerns about the baseline selection and justification (more on this below).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed propose an interesting formulation of one combinatorial problem and\nimplement this in the OpenAI Gym environment. Although, the proposed method works well\non this combinatorial construction problem and outperforms some baselines, the paper lacks\nsome necessary experiments to fully support the usage the action validation network.\n\nAlso, the motivation of the problem setup does not seem to be well justified, such as the use \nof images instead of 3D models, and why only DRL is picked as baselines? What about other \ntype of baseline solutions such as search-based method, or even supervised learning?", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "GhWQAbJuhf", "reviewer": "Reviewer_2kke", "summary": "The authors propose a combinatorial construction task consisting of assembling unit primitives (i.e. LEGOs) sequentially. The desired target is specified using one or more 2D images. The main challenges in this setting are dealing with incomplete information (of the target) and long-term planning with a variable-sized action space containing invalid actions in a combinatorially large space. To address this, the authors adopt an action validation network that filters invalid actions to an actor-critic network. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\nConstructing structures from primitives is an important task for AI and RL. The use of a 2x4 brick as the unit primitive greatly simplifies the problem, but still allows for enough complexity to be interesting and relevant. \n\n## Weaknesses\nThe main technical novelty of the approach is the introduction of the action validation network, but lacks any ablation studies that demonstrate its effectiveness. \n\nThe experimental results are also a bit underwhelming. In particular, it seems like the authors are training on single classes individually for each of the datasets (e.g. a single digit for MNIST). It is not clear why this is done, and would be very limiting if the training could not be done more generally. Additionally, besides the ablations, only one baseline is tested, a Bayesian optimization-based approach (BO). However, since BO requires exact volumetric information it would be helpful to include others that are more comparable. \n\n## Correctness\nThe methodology seems correct to the best of my knowledge.\n\n## Clarity\nWriting is clear.\n\n## Relation to Prior Work\nPrior work on assembly is discussed, but it is unclear why none were used as baselines in this work (see weaknesses).\n\n## Additional Feedback\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "D84-Dtg4eh", "reviewer": "Reviewer_ataC", "summary": "This paper proposes a new task of constructing a 3D object by assembly of Lego-like primitive bricks, given only image/s as the target. They propose an RL agent that by incorporating Graph neural networks and action validation module is able to tackle the proposed task. They demonstrate the agent’s ability on three different sets of data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed problem is very interesting and hard due to its combinatorial properties, but I think it would be useful if the authors could elaborate more on the justification of why this is an interesting problem, how it could have a broader impact, and potential applications.\n\nSome of the methods introduced in the paper such as the issue of dealing with variable and large action space are a great contribution beyond the use case in the current work.\n\nI do not think there is much evidence that humans learn about objects by decomposing objects into very primitive elements. Rather the segmentation is done semantically. Would be interested to be pointed to literature that demonstrates otherwise.\n\nSection 2. An illustration of brick in this section could help the reader better understand the text. For example in L66: an illustration of some of the combinations of two bricks could be very informative for the reader to visualize the combinatorial space. Same for the offset locations on a brick.\n\nL115: Is x_t in a quantized space due to the nature of the bricks? \n\nWhat happens if the action validation network is wrong and the proposed action is invalid?\n\nMost of the experimental comparisons are more like ablation of the B3. Would it be possible to adopt some of the prior work such as those listed in Table 1 for this task and show comparison results with them?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses a problem a construction a 3d object with lego, in the action space of locations of placement of a new block, purely from visual observation during test time and trained with reinforcement learning to match the volume of the target object.

The main positives of this paper are that this is an interesting setup/problem that has not been addressed before as far as we can tell (the lego construction yes, but from purely visual, trained with RL), they propose an interesting solution of graph neural network with action validation network and succeed at building basic shapes. 

The main drawback is that while they do succeed in making 3d shape objects (like an airplane), the quality of the constructed objects is not very high and the objects are not so complex (e.g. don’t have that many blocks). However, as this has not been done before, it’s hard to know what to expect - how difficult this problem really is. 

Because of this and because the setup is novel and this topic is quite uncommon and yet interesting, important and should be studied more, I think the paper should be accepted. 

For the authors I primarily recommend pushing the performance - getting larger, more diverse 3d shapes constructed of at higher level of quality.

There were also few things unclear in the discussion, I also recommend for the authors to explain them with better writing. 
Decision: Accept (Poster)