Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning

1 Introduction . A combinatorial space , typically characterized by discrete variables and their combinations , often induces interesting yet challenging problems such as traveling salesperson and minimum spanning tree [ 20 , 6 ] . The main challenges lie in the vast number of possible combinations as well as complex constraints imposed on them . In a similar spirit , we suggest a novel problem formulation , combinatorial construction , that focuses on the real-world construction procedure . Given only incomplete target information ( i.e. , 2D images or multiple views of a target object ) [ 25 , 12 ] , an agent sequentially assembles unit primitives ( i.e. , LEGO bricks ) . The proposed formulation is combinatorial since it engages repetitive placement of primitives , which leads to a large number of available solutions . Distinct property of our proposed formulation , however , is that the agent must build the solution incrementally by adding on to the partial solution . Specifically , a brick , which is a unit primitive of the object of interest , is placed on a discrete space by connecting to one of the previously assembled bricks . In addition , every connection between two bricks must follow a fixed rule while no bricks mutually overlap . Each assembly ( i.e. , action ) executed by the agent is , thus , modeled as selecting one of the feasible connections to place a new brick . The problem we introduce closely depicts how humans understand an object and adapt the acquired knowledge to a downstream task . Humans naturally analyze a 3D object by picturing its part-by-part ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . decomposition and consequently grasp a rich semantic understanding [ 15 , 22 ] . In various fields , they utilize an inherent ability to decompose objects to effectively solve challenging tasks such as object classification [ 17 ] , robot grasp planning [ 2 ] , and part segmentation [ 31 , 28 ] . Likewise , humans exploit this ability to solve the inverse problem – combinatorial construction . Given a desired object to be constructed and no strong supervision ( i.e. , ordered step-by-step instructions ) , humans can often still manage to build a valid target object by carefully planning or , sometimes improvising , the sequence of actions . Our environment , which corresponds to the proposed problem , is designed to learn and test such behavior with only partial information of the desired target available to the agent . Successfully constructing an object in our setup requires a comprehensive understanding of incomplete target information with the current structured state of assembled bricks and long-term planning to append each brick efficiently . These requirements , along with the absence of sequence-level supervision , incentivize us to devise a reinforcement learning ( RL ) approach [ 3 , 37 ] . In this domain , however , we must carefully handle both an indefinite action space and the existence of many invalid actions when applying RL [ 43 ] . In particular , both defining an action space that varies by the number of assembled bricks and distinguishing an invalid action that would cause overlap with other existing bricks quickly become intractable as more bricks are placed . To resolve the aforementioned issues , our model , dubbed Brick-by-Brick ( B3 ) , adopts an action validity prediction network that filters invalid actions to an actor-critic network . In addition to the novel RL formulation , we use graphstructured representation of the brick combination to interpret the assembling process as a sequential graph generation process . Overall , we summarize our contributions as follows : ( i ) We propose a novel problem formulation , combinatorial construction , that closely resembles a real-world object construction process that engages repetitive placement of components ; ( ii ) We design an RL agent for combinatorial construction , dubbed Brick-by-Brick ( B3 ) , to effectively address both a growing action space and a vast set of invalid actions ; ( iii ) We implement the corresponding environment based on OpenAI Gym and introduce new novel evaluation scenarios that vary by their incomplete partial target information . 2 Combinatorial Construction . To formulate the combinatorial construction problem , we start by defining a unit primitive that is used to construct a 3D object and an action space that determines where to assemble the next primitive . As a unit primitive , we utilize a 2×4 brick , which has eight studs and their fit cavities . This design choice yields a consistently varying action space , implying that if we add one brick to the current state of brick combination , we can efficiently define the next action space . We want to emphasize that with only six 2×4 bricks , we can create 915,103,765 combinations [ 9 ] . Accordingly , our choice of the primitive does not make our problem a trivial task ; instead , every decision of where we place the next primitive can deteriorate the quality of the final result because there exists a plethora of wrong paths . With our specific choice of unit primitives , we can define an action space for determining the next action and evaluating the future states . However , since every assembly step gradually expands the action space , naïve approaches to defining the growing action space are not appropriate for our problem ; an action space with redundant actions [ 43 ] is not applicable due to a varying action space , and an action sampling approach [ 13 ] is also not suitable due to nominal or invalid actions . Thus , we define a successive action space composed of a two-step decision : ( i ) choosing a pivot brick and ( ii ) choosing an offset from the pivot brick . Before explaining a pivot brick , we first assume a simplified assembly scenario that follows an Eulerian path2 – a new brick is always placed by connecting to the last assembled brick . This enables us to define a finite action space though most actions are infeasible to perform with the Eulerian path . To broaden the search space by generalizing the Eulerian path , one of previously assembled bricks is chosen as a pivot brick . Then , B3 decides an offset from the pivot brick , which describes how the next brick is placed relative to the pivot . Thanks to the homogeneous brick type , the number of available offsets is finite and consistent where we do not consider the validity of such offsets in a certain state – for 2×4 bricks , there exist a maximum of 92 available offsets . Due to the disallowance of overlap between bricks , our agent must consider invalid actions among available actions during assembly . Identifying the validity of actions from the current brick combination becomes intractable as more bricks are placed ; the complexity of this process is O ( |Aoff|t2 ) , where Aoff is an action space for offsets and t is the cardinality of assembled bricks at a given step ; see the supplementary material for qualitative results on this complexity . Such expensive overhead for validation naturally leads us to adopt an action validity prediction network that learns to identify invalid actions with a single forward pass . As described in Section 1 and this section , our problem has interesting but challenging characteristics derived from the assumptions on discrete placement , a connectivity rule , disallowance of overlap , and ultimately invalid actions . We , therefore , present a comparison to other existing studies in terms of state , supervision ( or a reward function ) , conditioning , target objects , and action validation , as shown in Table 1 . Compared to the previous work [ 11 , 3 , 19 , 38 ] , our method B3 constructs an object in 3D with incomplete target information and pretrained validity prediction component ; see Sections 4 and 5 for a detailed description . 3 Brick-by-Brick . In this section , we briefly introduce the definition of RL and the corresponding framework for combinatorial construction , where an agent places a brick sequentially . We then explain the details of our model that learns to select appropriate actions given only partial information of the desired target so that the assembled 3D object resembles the target . Moreover , to cope with a vast number of invalid actions in the process of assembly , we propose an action validity prediction network . See Figure 1 for the overall pipeline of our method B3 . Definition . In a standard RL framework , there exists an agent that interacts with an environment by iteratively making decisions given an observation of the environment . This follows general decision making procedure of a Markov decision process ( MDP ) , where a transition function satisfies the Markov property , i.e. , p ( st+1|s0 , s1 , . . . , st , at ) = p ( st+1|st , at ) , where st and at are a state and an action at timestep t , respectively . In our problem setting , we only consider a finite horizon MDP formally defined as a tuple of ( S , A , P , R , γ ) , where S = { st } is a set of states , A = { at } a set of actions , R : S × A → R a reward function , P : S × A → S a transition function , and γ ∈ [ 0 , 1 ) a discount factor . The goal of the agent is to learn a policy π ( at|st ) that maximizes the expected future cumulative reward . We introduce our detailed MDP formulation for combinatorial construction in the following sections . 3.1 Problem Formulation . Given target information T , the agent aims to construct a 3D target object T by assembling bricks sequentially , one brick for each t-th step . Each t-th brick is represented by its pose ( xt , dt ) , where xt ∈ Z3 is the center coordinate of the brick in a 3D space and dt ∈ { 0 , 1 } denotes one of two possible directions , meaning that its longer axis is aligned along either x axis or y axis in the 3D space . 2It is a path that visits all the vertices without revisiting the edges visited before . Target Information . We are given as target information T a set of binary images of a target object , which may correspond to incomplete and partial information about the target . In practice , this setup with partial information is more realistic than accessing full information of a 3D target shape . Our task thus is to create a sequence of unit bricks by inferring a target object from abstract information in a combinatorial manner . State . Each t-th state st of the MDP is represented by a tuple of a directed graph Gt composed of t bricks and target information T , i.e. , st = ( Gt , T ) . The graph is defined as Gt = ( Vt , Et ) , where Vt = { vi } ti=1 is a set of t bricks , i.e. , vi = ( xi , di ) ∈ Z4 , and Et = { eij } ti , j=1 is a set of the offset vectors between two connected nodes , i.e. , eij = ( xi − xj , di ⊕ dj ) ∈ Z4 . Note that nodes are connected by edges according to sequential actions and relative offsets in pose are used for edge features in order to induce translational and orientational equivariance . Since all the edges are bi-directional , we omit the arrows when displaying graphs in Figure 1 . Action . We define a successive action space of choosing the pivot brick first and the corresponding offset next . Formally , with t bricks assembled , we define an action at = ( a piv t , a off t ) where a piv t is to select a pivot brick and aofft is to select an offset with respect to the pivot brick . The pose of the next brick is then ( xpiv + ∆x , ( dpiv + ∆d ) mod 2 ) , where ( xpiv , dpiv ) and ( ∆x , ∆d ) are determined by apivt and a off t , respectively . In choosing actions a piv t and a off t , we exclude invalid actions : ( i ) choosing a pivot brick near which no additional brick can be placed and ( ii ) choosing an offset for the next brick that overlaps with existing bricks . To select a valid action in the action space with a vast number of such invalid ones , we predict invalid actions in advance using the action validity prediction network and exclude them from action candidates preemptively ; we mask out all the probabilities of invalid actions , re-normalize the distribution over actions at , and sample an action from the distribution . Transition Function . Given state st and action at , our transition function p ( st+1|st , at ) is designed to determine the next state st+1 by deterministically updating st based on at . The node of the new brick vt+1 is created so that Vt+1 = Vt∪ { vt+1 } . The edges between the new brick and existing bricks in physical contact via studs are created so that Et+1 = Et ∪ { e ( i ) ( t+1 ) } i∈Nt+1 ∪ { e ( t+1 ) ( i ) } i∈Nt+1 , where Nt+1 denotes the set of bricks in direct contact with the new brick vt+1 . As the result , the graph in the state st+1 is updated to Gt+1 = ( Vt+1 , Et+1 ) . Reward Function . In contrast to the tasks where a direct reward evaluation is readily available , it is not trivial to quantify the object assembled by combinatorial construction , especially , in the context of graph generative model [ 26 ] . To mitigate such an issue , we exploit the property of a voxel representation . Given a desired object , we first create voxels in a closed space and determine the occupancy of voxels with a target object , after normalizing it to the bottom center of voxels . We then transform the combination of currently assembled bricks into the occupancy of the voxels and measure the overlap with the target object : ∆IoU ( Ct , T ) = vol ( Ct ∩T ) vol ( Ct ∪T ) − vol ( Ct−1 ∩T ) vol ( Ct−1 ∪T ) , ( 1 ) where Ct , Ct−1 , and T are the occupied voxels at timestep t , timestep t− 1 , and a desired target , respectively . In addition , vol ( · ) is a function that measures a volume . The step-wise reward function is then ∆IoU if the new brick overlaps at least 50 % with the occupied voxels of target object and 0 otherwise . Consequently , our agent will learn sequential placement of bricks to construct the object , without explicit supervision by maximizing Equation ( 1 ) , as will be described in the subsequent section .

Summary: The authors propose a combinatorial construction task consisting of assembling unit primitives (i.e. LEGOs) sequentially. The desired target is specified using one or more 2D images. The main challenges in this setting are dealing with incomplete information (of the target) and long-term planning with a variable-sized action space containing invalid actions in a combinatorially large space. To address this, the authors adopt an action validation network that filters invalid actions to an actor-critic network.