Learn2Assemble with Structured Representations and Search for Robotic Architectural Construction
Authors: Niklas Funk
Georgia Chalvatzaki
Boris Belousov
Jan Peters
Venue: CoRL2021 Poster (2021)
Keywords: Structured representations
Autonomous assembly
Manipulation

Abstract: Autonomous robotic assembly requires a well-orchestrated sequence of high-level actions and smooth manipulation executions. Learning to assemble complex 3D structures remains a challenging problem that requires drawing connections between target designs and building blocks, and creating valid assembly sequences considering structural stability and feasibility. To address the combinatorial complexity of the assembly tasks, we propose a multi-head attention graph representation that can be trained with reinforcement learning (RL) to encode the spatial relations and provide meaningful assembly actions. Combining structured representations with model-free RL and Monte-Carlo planning allows agents to operate with various target shapes and building block types. We design a hierarchical control framework that learns to sequence the building blocks to construct arbitrary 3D designs and ensures their feasibility, as we plan the geometric execution with the robot-in-the-loop. We demonstrate the flexibility of the proposed structured representation and our algorithmic solution in a series of simulated 3D assembly tasks with robotic evaluation, which showcases our method's ability to learn to construct stable structures with a large number of building blocks. Code and videos are available at: https://sites.google.com/view/learn2assemble

Learn2Assemble with Structured Representations and Search for Robotic Architectural Construction
Niklas Funk, Georgia Chalvatzaki, Boris Belousov, and Jan Peters Department Computer Science, Technical University of Darmstadt, Germany [niklas.eeorgia, georgia, boris grorosss@roottleeeee de, mail@j aateejjaapp-eeeee aaa--eeers... net
Abstract: Autonomous robotic assembly requires a well-orchestrated sequence of high-level actions and smooth manipulation executions. Learning assem- ble complex 3D structures remains a challenging problem that requires draw- ing connections between target designs and building blocks, and creating valid assembly sequences considering structural stability and feasibility. To address the combinatorial complexity of the assembly tasks, we propose a multi-head attention graph representation that can be trained with reinforcement learning (RL) to encode the spatial relations and provide meaningful assembly actions. Combining structured representations with model-free RL and Monte-Carlo plan- ning allows agents to operate with various target shapes and building block types. We design hierarchical control framework that learns sequence the building blocks to construct arbitrary 3D designs and ensures their feasibility, plan the geometric execution with the strate the flexibility of the proposed structured representation and our algorith- mic solution in series of simulated 3D assembly tasks with robotic evalua- tion, which showcases our method's ability to learn to construct stable struc- tures with large number of building blocks. Code and videos are available at: https: s/sstes.oosssssseeee.... com//ries/learnaaseenbie
We demon-
Keywords: Structured representations, Autonomous assembly, Manipulation
Introduction
Construction and manufacturing becoming increasingly automated in the last decades. However, there is an essential need for sustainable autonomous architectural assembly [1], where game- changer would come with intelligent robot assembly abilities that optimally decide over plans, ac- tions, execution, and efficiency [2]. In this work, our main focus the combinatorial optimization problem of autonomously assembling complex structures with robotic manipulators without priori defined task plan and goal poses for the sequential trcking-placing actions. For constructing abstract target designs, we must consider the combinatorics of the growing action space w.r.t. the number of available modules and the size of the structure. Therefore, an effective representation of the assembly problem is essential. Moreover, the geometric execution of the picking and placing actions by the robot imposes constraints to the assembly sequence, as those actions are subject to the kinematic feasibility the robot's workspace. Eventually, the problem of assembling structures lies in the area of long-horizon manipulation tasks, where most methods in the literature consider a known task plan, and focus on fine manipulability and structural stability, or learn action sequences from demonstrations [3, 4, 5, 6, 7, 8).
In this paper, we propose a novel algorithmic solution for robotic assembly that combines high- level decision-making on the construction sequence with the geometric execution by the robot that should ensure feasibility and stability. We propose a graph-based representation that captures the relations between target shapes and available building blocks. Notably, we design multi-head atten- tion-based graph neural network (GNN) architecture with a purposefully induced inductive bias for encoding the structural representation of the assembly task. The GNN is trained through reinforce- ment learning (RL) to explore feasible actions, resulting in an expressive and flexible representation. When combined as prior to Monte Carlo Tree Search (MCTS), it extrapolates to out-of-distribution (OOD) assembly tasks, i.e., tasks with a higher number and different types of blocks and vari-
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

able target shapes. Ultimately, we provide solution for the motion generation of the sequential picking-placing actions when having the oobot-n-the-loop to ensure reachable and feasible actions, which do not disrupt the assembly process. The proposed learn2assemble algorithm provides a flexible, autonomous robotic assembly agent for constructing 3D shapes using more than ten blocks. Our main contributions are threefold.
(1) We propose multi- head attention- based graph representation for the 3D as- sembly problem that is flexible enough for representing arbi- trary, stable structures and their relations to different building blocks. (2) We design an integrated long-horizon manipula- tion algorithm that learns through exploration, which com- bined with model-based search leads to generalizable skills. Our method considers the robot-in-the-loop integrating high- level action planning with low-level motion generation for learning policies that ensure the kinematic feasibility and sta- bility of the constructed structure. Finally, (3) developed a novel benchmarking environment for 3D robot assembly, which is modular for testing with and without the robot, for ar- bitrary target designs, and with adjustable types and number of objects. Our empirical results on series of representative ex- periments showcase the generalization power of the proposed algorithmic solution, drawing interesting insights on the com- bination of integrated learning and planning for long-horizon manipulation that can apply to a range of robotic applications. Related works. Autonomous robotic assembly is essential for manufacturing and construction, and therefore, many works tried to tackle the problem of automatizing task execution and fine manipulation. Autonomous assembly is a long-horizon manipulation task with multiple stages of decisions and sub- task executions to be made alongside controlling the dynamic exccution of the assembly process. Thus, researchers proposed methods for Task and Motion Planning (TAMP) [9]. and RL [10] to address the challenges of com- binatorial optimization over high-level action sequencing and low-level motion generation.
Figure I: Simulated assembly environ- ment with a 7-DoF manipulator and two sets of blocks: the unplaced white ones on the left and the "base" red block with some already placed blocks by the robot in the centre. The yel- low silhouette denotes the target shape. The goal is to place the available blocks to fill the overall 3-dimensional target shape.
In [11, 12] a TAMP method is proposed for robotic architectural construction and extrusion, requir- ing though exact domain specifications to find an assembly sequence; TAMP with logic-geometric programming is proposed by [13] where the focus lies in optimizing the structure's stability. In [14], the authors propose hierarchical planner using hybrid dynamics models for the "toy-airplane" as- sembly task. The authors of [15] optimize the assembly sequence of complex interlocking blocks through RL assuming single fixed design. A method based on neuro-symbolic planning is pro- posed by [16] for learning to predict sequences of actions for stacking. The authors of [17] propose an end-to-end approach for sequential pick- pick-and-place tasks using shape correspondences and learn- ing to assemble by collecting demonstrations from human operator disassembling. Conversely, we learn2sseemble arbitrary designs from scratch, learning both the sequence of actions and discover- ing goal positions per building block.
Autonomous assembly appeals to the machine learning community due its combinatorial com- plexity, which, depending on the structure's s size and availability of building blocks and their possible combinations [18], can by far surpass the state-action combinatorics of problems like chess and Go [19, 20]. The relational representation [21, 22] and generalization power of GNNs was thoroughly explored solving combinatorial optimization tasks [23], with successful applications in 2D assem- bly [24] when combined with RL. In robot learning, earlier works use deep RL for short-horize challenges like peg-in-a-hole [10, 25, 26]. In [27] the use of structured representations in model-free RL is proposed to induce inductive biases in diffferent stages of curriculum for exccuting assembly tasks of building towers of variable heights. While the general motivation of this work is close to ours, [27] does not learn the sequence of the building blocks for constructing arbitrary structures, but it is essentially a goal-based method for learning pick-and-placing
manipulation with known goal object positions. In similar direction, the authors of [28] use learning from demonstrations train two GNNs, one that selects objects in the scene and another one that selects a suitable goal state from set of possible goal positions.
2

---

2
Proposed Method
The main research question we pose in this work is "how can a robot perform combinato- rial assembly tasks of abstract architectural designs?". Fig. I depicts a typical setup con- taining 7DoF manipulator robot, a varying number of building blocks, and one possible ar- chitectural target design. We formalize the task of autonomous construction of abstract target shapes given some available building blocks as a Markov decision process (MDP) [29]. As il- lustrated in Figs. 1 & 4 we can build one- up to four-sided structures. The overall desired structure is given by each side's target topology, which is turn defined by the coordinates of the target points spanning the area to be filled (see illustrative example in Fig. 2). The de- sired shape is thus parameterized by the set of target points ST = {xr, l E Ni (yellow).
The state also contains set of blocks that are already placed in the scene Sp = [xp, li E N (red) and a set of unplaced blocks Sy = {xu, Ik E N (gray) that need to be added. The vector x E IR includes the 3D position and two booleans indicating the element's properties (placed/unplaced, target point/building block). The state 8, at any time step is thus given by the three sets 8 objective is to use the available blocks to construct stable structure that fills the desired target design. We use a discrete action space and define five relative placement actions, i.c., any unplaced block can be put on top, on the left, on the right, behind, in front w.r.t. any already placed block. In the robot experiments, however, the action space is augmented to also allow the specification of the manipulator's 's orientation during grasping and placing. For most experiments, we choose over two possible grasping and placing orientations, resulting in four grasp-place combinations. Together with the placement actions, this yields N₂ = 4 x 5 = 20 actions when placing one block w.r.t. another. Since the possible actions a depend on the number of placed and unplaced blocks, we end up with varying action space of size N₂ x N₂ No- We assign positive reward r(se; 0e) on all actions that increase the filling of the target structure while preserving its stability. Given the problem's combinatorial complexity of discovering the optimal high-level action plan that will allow the assembly of complex 3D structures which are stable and kinematically feasible, we decompose into three main problems: (i) finding an expressive representation for our state space, (ii) decoding states into meaningful actions, and (iii) ensuring stability and kinematic feasibility. Our research provides a thorough study of how we can tackle these problems, and we provide novel 3D assembly algorithm with the robot-in -obot-in-the-loop tho-loop through combining a learned multi-head attention (MHA) representation with MCTS.
Sr, and has N = N₂ + N₂ + Ng elements. The
Figure 2: Setup with one-sided triangular target shape defined by three points (yel- low), three unplaced blocks (gray) and two placed blocks (red).
2.1 Multi-head attention graph representation
Graph-based representations [30] are an effective tool when dealing with combinatorial problems [31, 32, 33]. Compared to classical satisfiability solvers, their main advantage lies in their real-time capabilities, while their architectural properties allow generalization in contrast to most standard neural network architectures, which operate on fixed-size inputs and outputs. As those are essential properties for learning to assemble structures of combinatorial com- plexity, we will introduce below our proposed GNN model that is inspired by the combination of graphs and attention [34, 35]. GNNs receive input graph G = (N.E), and return high-level encoding over nodes and edges to be further exploited for deciding which action take. In our case, the set of nodes N = (Sr, Sp, Su = {m: J is given by the current state, and the connectivity information E is defined as a matrix of size N x N. If there is an edge connection between nodes n, and n, entry ]( equates to 1, otherwise it is O.
problems of different sizes
Attention mechanisms [34] were introduced in GNNs [35] enable nodes to attend over their neighbours' features and learn different weights for different nodes without requiring costly ma- trix operations. We want to exploit this flexibility as solving the assembly task necessitates drawing connections on multiple levels, i.e., between nodes of the same type to encode the already existing structure or the target shape, as well as between all nodes to come up with meaningful represen- tation for action-decision. In the following, we will introduce the proposed MHA architecture, that naturally reflects the necessary multi-level decision process of the assembly task, and has proven to be effective when combined with policy search for solving combinatorial problems, like the travel-
ling salesman [32]. In the first step of MHA, the initial node embeddings (D) x; are projected into a higher dimensional space by
n (1)
g(n()) ReI.U(FC(()))
(1)
3

---

tepul partallyy conneeeey grapn
I round ol monsgg postrg
graa
Action CRESKRN
Figure 3: GNN architecture illustration, mapping from the input graph Q-values. The coloring follows Fig. After an initial projection into a higher dimensional space follow rounds of message passing using MHA. This results in an encoded version of the graph, which is then exploited for action selection.
using a fully-connected (FC) layer followed by rectified linear unit (ReLU) activation function. Note that the function g will be used repeatedly we progress with the graph update, yet, suming different weights on each further appearance. Next follow I. rounds of message passing, ic..applying (2) L times to obtain the final embedding for each node i according to
0 Mgg(((MHHA(N(00)).))) ()))).
(2)
with a skip connection layer h(f(z)) x + f(x), the current round of message passing 1, all node embeddings from the previous round A/l(-1), and MHA mechanism introduced below. As illustrated in Fig. 3, for a single out of the M attention heads with index o, we first compute three values - the key k, query q, and value using three different weight matrices WK.0. W We.,,
respectively (ki = W (1-1), k.o = W W,,om("-1) 14.0 W (t-1). Multiplying the key and query of all nodes results in a compatibility score C22 for the 1th to--hh node connection
k ir otherwise,
C1,.
(3)
-00,
with d normalizing constant. From this score, we can then compute the attention weights using $isl a softmax as.jo Eta
to aggregate the values for each node, resulting in the output message
mi,o = Ey Uo.jo A weighted sum of all messages yields the final result of the MHA module
MHA(M
W Wm.ooiii
(4)
with weights W controlling the influence of each one of the single attention heads. Fig. 3 depicts a simplified example of the encoding performed by the MHA architecture.
2.2 Learning assembly policies
MHA- GNNNs can only unfold their potential when combined with algorithms that refine their weights to form expressive representations exploited herein for action selection (Fig. 3). The GNN should thus be shaped based on the reward signal, resulting in a RL setup with the goal of obtaining per- formant policies. This powerful combination results in agents that (i) can be applied to different problem instances due to the representation's 's fiexibility, (ii) are reactive, despite the problem's 's com- binatorial complexity, and (iii) can be trained directly in simulation environments which include the nonlinearities of the robot and the contacts. Due to the problem's 's combinatorial complexity and its discrete, time-varying action space, we use model-free Q-learning, [36] which has been successfully applied to complex tasks such as playing Atari games from images. Moreover, we investigate its in- tegration with model- based planning, as the addition of search can counteract the overoptimism of the Q-function approximation and result in more robust behaviour [19, 37].
Action Decoding. The encoded graph representation from Sec. 2. needs further processing to de- cide on the next action to take. As actions are defined relatively between unplaced and placed blocks (i.e., nodes) can directly assign value to all available actions
Q(n,, n,, No) (1 (4) (1) (.FC (N Exngg:
Vn, E Su. n2 E Sp
(5)
with the total number of N nodes in the graph. Note that the Q-value does not only depend on the two nodes' embedding. but also on a global feature based on averaging over all embeddings. As all the operations are defined over the set of nodes, this incoding-decoding architecture can seamlessly generalize to different problem sizes, differefe number of blocks or target shapes. DQN: For our setting, we define the loss function as the smooth L.1 loss between the current action- value estimate of the GNN, noted as Q(n,, n2, a.) and the value obtained from the rollouts using
4

---

target network, noted as Qr, i.e., (nn a.) = r(se,, ae) + S0 Q (mu Ri, a.) with discount factor y and the selected action 0, E Na.
MCTS: MCTS [38] has proven effective for solving tasks with discrete action spaces [39], hence, suitable for our problem. Contrary to model-free RI., MCTS relies on a model of the environment to perform tree search for action selection and does not require any form of function approximation. However, in scenarios with significant branching factors or expensive model evaluations, the time complexity quickly increases, and the use of MCTS becomes intractable. Consequently, [19, 20] exploit Q-learning for estimating the value of the nodes in MCTS and show impressive results in solving tasks of great complexity, like the game of Go. Also, in the context of accelerating and generalizing skill-learning, the combination of learning and planning has received increasing atten- tion [20, 37, 40, 41]. In this work, inspired by these advantages, we explore different algorithmic variations of the interplay between Q-learning and planning, as described below. Pseudocode for all variants is given in Appendix B.
1) DQN+MCTS: Inspired by [19], we propose a variant of AlphaCio, that uses a pretrained Q- network prior for evaluating the leaf nodes of MCTS, leading to increased efficiency. As we seek to effectively combine learning and planning for autonomous assembly with the robot-in-the- loop, we use small search budgets for which the prominent solution for MCTS expansion the exploration strategy of UCT [19, 42] - is unsuitable because it becomes over-optimistic [43]. There- fore, we use an e-greedy expansion strategy during search, that allows better exploration than UCT. 2) Q-MCTS: We follow a generalization of Q-learning using samples based on planning. As in [20, 37], we combine DQN with MCTS during training to perform an informed exploration and collect good samples through search. However, as motivated previously, due to small search bud- gets, our approach uses an c-greedy expansion strategy over unexplored states, instead of UCT. Let (s( 0z) = Qs(8t, ax) + (s) + y maxa; ( af) be the updated value of an expanded node, where sy is the new state arriving during search. The resulting Q-MCTS methods augments the DQN learning objective with a cross-entropy loss defined on the values explored during search, i....oss = ---softmmx(((( (Qs az) )² log(softmaxx(4 (Q(se. (st, (t))) which is intended to regularize and improve the Q-function estimate based on the experience collected while searching.
3) 6-MCTS: This implementation follows [37] more closely. The main differences w.r.t. the Q- MCTS policy are that first, there is an e-greedy decision on whether to do search or uniformly sam- ple a random action, whereas Q-MCTS always conducts search. Secondly, during search, e-MCTS follows the UCT expansion strategy. The Q-learning objective remains the same as with Q-MCTS, however, the cross-entropy loss is computed only the samples where search has been conducted. In essence, the method's 's difference is in the way of collecting the model- based samples.
2.3 Integrated learning and planning for robotic assembly
All previous components can now be combined to obtain an algorithm capable of training agents to build desired target shapes, i.e., a robot capable of abstracting the sequence of actions for build- ing arbitrary stable target structures from individual elements while executing feasible actions in its workspace. Note that the only control component we assume given is point-to-point control strat- egy based on the robot's 's inverse kinematics. To enable this combined docision-making strategy that touches on the ground of TAMP literature, we propose the combination of the previously described graph representations and learning algorithms provide a novel learn2assembl method. Briefly, single step of a learning episode starts with selecting actions (with or without tree-search depending on the learning algorithm), i.e., the next object be placed, the grasping pose, the goal position, and orientation. A path from the picking to the placing position is computed, and the robot executes the placement, for which it receives reward. Consequently, we store the current graph's state, action, reward, and new state in the replay memory to be later used for learning.
3 Experimental Results
For evaluating the different components of the proposed method and their respective contribution, we designed specific experimental scenarios. We start with an investigation over graph architectures, continuing with the different learning methods in environments for 3D assembly without including a robot yet. Selecting the best settings from the previous tests, we experiment with the robot-in-the- loop for our final empirical evaluations of the proposed algorithm.
5

---

(a)
(b)
(c)
(d)
(e)
(I)
Figure 4: (a) Illustration of the single-sided 3D assembly environment, (b) the two-sided 3D assembly environ- ment, (c) the four-sided 3D assembly environment with the robot-in tet-tthelooppp (d) the two-sided 3D environment with the unplaced blocks placed at random. (e) the too-sided environment with different building blocks, and (f) the result of transferring a policy trained in simulation (a) to the real world and even a different manipulator.
Table I: Comparison of different architectures on the single-sided environment without the robot and only one type of block. R is the cumulative discounted return, ff the ratio of runs that ended with failure, i.c., the structure colliding, and b the ratio of rums that ended without success and no more blocks remaining. The star(*) " marks the environment where the agents were trained in.
3-by-3 grid, 20 blocks
3-by-3 grid, 30.34 blocks f
4-by-4 grid, 24 blocks
Method
MHA (FC)
3.22 (0.04)
0.01 0.15
(0.04)
0.00
0.21
3.66 (0.05)
0.18
0.41
S2V (PC)
2.36 (0.08)
(0.10)
0.08
0.87
2.75 (0.20)
0.55
Simulation environments. The assembly environments are depicted in Fig. 4. We investigate, one-, two- and four-sided setups with each side of the target shape defined by the position of 3 (see Fig. 2). 4, or 5 target points. The number of target points and their locations are sampled randomly from grids of different sizes, ranging from 3-by-3 (i.e., the sampled target points can maximum span an area of height and width of 3 times the cube's 's edge length) to 6-by-6. Due to the relative action space, every scene is always initialized with one initially placed element, marked in red, serving as the building base. For evaluating the assembly progression, we use depth cameras, placed on each side of the specified target structure. By projecting the target points into the images after each action, we obtain the change in the target shape's 's filling. The reward functions endorse actions that lead to improvement in the filling (cf. Appx. C.2). The construction process is finished, once the total coverage exceeds a threshold, whenever there are no more unplaced blocks available, or upon executing an invalid action, i.e., an action resulting in an unstable configuration. in destructing the current structure, or if it is kinematically infeasible. In all environments without the robot, the plac- ing is done by directly specifying one of the five placement actions, resulting in reduced action space of No = 5. If not stated differently we use partially connected graphs (see Fig. 3), inducing a stronger inductive bias on the structured representation compared to fully connected (FC) graphs (ef. Appx. A.3 & D.3). In the following tables, for evaluating the agent's performance, we report the cumulative discounted return R. the ratio of runs that ended with failure, i.e., upon an invalid action f. the ratio of runs that ended without success and no more blocks remaining 6, as well as the mean number of actions conducted per run a. The star(*) marks the agents' evaluation in the same setting as in training, while the rest are OOD experiments, i.c., exclusively evaluating the agents in settings with previously unseen target shapes or number of blocks. For more details, see Appx. C.
Graph architectures. We evaluate the proposed MHA representation against the commonly used Structure2 Vector (S2V) architecture [24, 31] (cf. Appx. A. in a simple environment (see Fig. 4a) only considering one type of object but omitting the robot. The learning is conducted with DQN. Results, As shown in the first column of Table 1, already in the original training environment, the MHA approach outperforms S2V significantly. The high rates of failure and exceeding the number of available blocks indicate that S2V cannot draw the connection between the target shape and the current structure. This might be due S2V's different message passing, which cannot weigh the importance of different nodes as with the attention mechanism. When increasing the number of available blocks and the size of the structure to be built (columns & 4). we see an evident advan- tage of MHA in handling OOD tasks. In Appx. 1.2.1, we provide additional results when using only a single attention head, which confirm that using attention is advantageous, and MHA yields the best performance. We, thus, continue our experimentation using the MHA architecture.
Learning algorithms. To investigate the performance of the learning algorithms (Sec. 2.2), we will use the two-sided environment shown in Fig. 4b without the robot, thus using the reduced action space.
Results. Table 2 summarizes the results, starting without any search budget (i.e., no tree search) to evaluate the learned Q-functions. The DQN and c-MCTS agents perform similarly, with DQN slightly outperforming c-MCTS through lower failure rates across tasks. Moreover, DQN can solve

---

Table 2: Combining Q Q-learning and MCTS in the two-sided environment without the robot.
Seurch Budget
3.by.3, grid 20-24
blocks
3- grid, 30.3 34 blocks
4-by-4 grid. 30-34 blocks
5- grid. 40-44 blocks
Method
Rt
R
π
R
0
DXJN
3.21 (0.05)
7.33
0.16
3.44 (0.16)
8.65
0.17
3,61 (0.06)
12.02
0.51
3,63 (0.08)
13.66
0.93
c-MCTS
(0.03)
8.57
(0.10)
9.30
(0.09)
12.53
3.50
12.75
Q-MCTS
(0.15)
7.33
0.51
2.89 (0.09)
7.50
0.64
2.66 (0.08)
7.47
0.92
2.34 (0.17)
6.45
0.99
10
DON+MCTS
3.47 (0.02)
8.16
0.05
3,66 (0.05)
8.96
0.06
3.96 (0.04)
13.92
0.31
4,08 (0.09)
17.33
0.87
c-MCTS
(0.03)
8.48
0.20
(0.11)
9.10
(0.09)
12.65
3.63 (0.16)
13.16
Q-MCTS
(0,44)
8.56
0.27
3.39 (0.08)
9.24
0.41
3.42 (0.07)
10.99
0.76
3:41 (0.13)
11.27
0.96
1000
UCT
0.54
4.00
1,00
the task with fewer actions. Contrarily, the Q-MCTS agents perform significantly worse. We believe that this difference in performance is due to the constant cross-entropy regularization in Q-MCTS from the beginning of the training, especially when search samples might be bad, while for the e- MCTS agent the regularization is only slowly added as training proceeds and less random actions are taken (search is better), which seems to be beneficial. Note that without the addition of search, in- creasing the number of blocks, as well the target size, results in quick increase of the failure rate for all methods. Adding a search budget of 10 already counteracts this trend, especially considering the DQN agent, where the rates of failure can be reduced for all the tasks by using DQN+MCTS at test time. For the Q-MCTS agent, the same trend is noticed, while the gains of the c-MCTS agent marginal. This suggests that the e-MCTS agent is overoptimistic and seems to choose similar ac- tions as to when not using search at all. Overall, combining MCTS with a pretrained DQN results in the best performance in our experiments. The last row of the table underlines the problem's 's combi- natorial complexity, illustrating that performing pure UCT without any prior and search budget of 1000 performs significantly worse for the simplest experimental setting (more details in Appx. D.4). Learn2assemble with the robot-in-the-lopp tree-search in our target environments, including the robot manipulator. The task's difficulty in- creases, as also the grasping and placing poses have to be specified while ensuring action feasibility by the robot and the structure's stability. We start with constructing single-sided sided designs (Fig, 4a) to illustrate the necessity of training with the robot-in-the--opp thop before evaluating the proposed method with multi-sided designs (Fig. 4b & 4c).
Next, we evaluate the DQN agent combined with
Results. We first compare two policies, one trained with, and the other without the robot us- ing plain DQN without any search in a single-sided environment (Fig. 4a). The agent trained with the robot-in-the-loop outperforms the other one resulting in significantly reduced failure rate of 15 % and consistently higher rewards (cf. Appx. D.5). This shows the necessity of in- cluding geometric planning during training for obtaining high-level decisions that are compati- ble with the low-level execution. It is thus not sufficient to only figure out where to place the parts; the kinematic constraints and robot motion have to be considered. In the subsequent exper- iments, we build 3D structures with the robot as shown in Figs. 4b & 4c. Our results in Tables and 4 show that MCTS (search budget of 10) consistently improves performance in terms of higher returns and lower failure rate. In particular, the experiments demonstrate that our proposed pipeline can execute multiple sequential pick-and-placing actions, with a maximum of 17 correctly placed blocks for the 5-by-5 grid in the two-sided environment and up to 22 correct placements in the four-sided environment, using DQN+MCTS with the robot-in-the-loop. the failure rate is higher, in-
Table 3:
Comparing policies with and without tree search on the four-sided robotic environment.
by 3 grid 10-18 blocks
3 3 grid 16-24 blocks
Method
DON DON-MCTS
(0.00))))
6.91
0.30
2.55 (0.06)
7.25
0.35
3.08 (0.06)
7.59
0.16
2.90 (0.07)
8.00
0.20
Compared to the previous experiments, without the dicating the task's 's increased difficulty. and the need of adding soft-placing controller.
Generalization w.r.t. randomized scenes. To evaluate our algorithm's robustness w.r.t. changes in the scene, we transfer the previously trained policies and evaluate them in scenarios where the unplaced blocks are placed randomly around the structure to be built, as shown in Fig. 4d.
Results. Rows 3 & 4 of Table 4 reveal that the policies indeed generalize to these novel scenarios, as the percentage of unsuccessful experiments only increases at maximum by 11 % for the most complex scenario, compared to their performance in the original environment (rows I & 2). This confirms that our proposed method does not overfit the exact layout or geometry, but rather builds meaningful features that allow to successfully transfer the behaviour to scenes that are substantially different from those encountered during training.
Table 4: Comparing policies on the two-sided robotic environment, Rows 1&2 correspond to evaluating in the original environments (Fig. 4b), while rows 3&4 are the evaluation in randomly initialized scenes (Fig. 4d).
Favironment
3 3 grid 10. 14 blocks* hy 3 grid 18 blocks
5 5 S grid 18 blocks
Method
inttiaizzation
f
/
DON
fixed
(0.00)
4,80
0.20
2.10 (0.05)
5277
0.24
2,4400.5
7.3
0.56
MCTS
fixed
2.41 (0.05)
5.32
0.09
2.32 (0.03)
5.64
0.15
3.19 (0.11)
9.38
0.36
DON
random
2.06 (0.10)
4.93
0.24
1.88 (0.13)
5.02
0.32
2,28 (0.11)
7.52
0.64
DON-MCTS
random
2.33 (0.06)
5.38
0.10
2.14 (0.12)
5.54
0.20
2.85 (0.14)
9.31
0.47
7

---

Table 5: Evaluating the trained policies in the environments with multiple different objects available (Fig. 4e).
The results in the first row correspond to using modified environment without including the robot.
Envirenment
5 by S grid 20-24 blocks
5 by 5 grid 30-34 blocks by 6 grid 30-34 blocks
Method
w/wo rohot
R
DON+MCTS
wo robat
2.20 (0.02)
5.05
1889 (0.03)
5.96
1.48 (0.09)
7.27
DONFMCTS
w rubol
(0.05)
3.53
0.21
0.77 (0.09)
3.27
0.61 (0.08)
4.38
Generalization w.r.t. different building blocks. We finally investigate our method's performance when using more complex objects (Fig. 4e). This makes the task significantly more difficult, as the agent has to not only learn each part's admissible grasps but also differentiate the building blocks to select and place the correct object type. All novel objects are a combination of primitive boxes which allows keeping the relative action space, i.c., placing an unplaced primitive box belonging to larger object w.r.t. a placed primitive box results in moving the entire object. For the experiments without the robot, we adjust the action space to enable changing the object's orientation (ef. Appx. D.9). Results. The results in Table 5 illustrate the representation's 's flexibility and ability to successfully deal with the different building blocks. Despite the increased complexity, we achieve similar perfor- mance in the scenes without the robot as in previous experiments (Table 2). When training with the robot, we can still handle the task's complexity with a 80% success rate in the simplest setting, but observe a drop in performance with a larger number of objects and bigger target shapes. While one cause for the performance drop is the setting's 's increased complexity, there ment actions that would require 3D gripper orientation control allowing for a smooth insertion of the complex blocks. As our current top-down placing controller only offers planar orientation control, some placement actions cannot be executed appropriately resulting in increased failures.
also several place-
Remarks. We have conducted extensive experiments to show the superiority of our proposed MHA- GNN approach for solving combinatorial assembly tasks with the robot-in- tobot-in-the-lop... The results demonstrate how strong inductive biases combined with attention can shape meaningful relational representations. When combined with deep Q-learning, this representation allows us to take deci- sions over long horizons despite an increasing action space. Adding search at test time improves performance across all experiments, demonstrating that the proposed method can generalize w.r.. different target shape sizes, number of building blocks, and different scenes. The policies can also be transferred to the real world and to different manipulators, by initializing the simulation to mirror the real scene and executing the obtained actions both in simulation and reality (see Fig. 4f). While we show very promising results in combining learning high-level action decisions with planning ge- ometric executions, with a more informed search in the graph space. Our experiments on using different objects under- line the representation's flexibility, but also reveal current limitations in the definition of the action space, especially considering the robotic execution, which may be mitigated by enriching the action space through 6D grasping and placing. Moreover, several assemblies failed due to "rough" robot actions, meaning that more sophisticated motion generator might be needed for finer placement.
can see combinatorial barrier in the decision-making making that might be tackled
4 Conclusion
We presented new learn2assemble algorithm for learning autonomous robotic 3D assembly from scratch without prior knowledge of any task plan. For addressing the problem's 's combinatorial com- plexity while maintaining adaptability to different scenarios, we propose a graph-based melti-head attention representation that captures the spatial relationships between target construction designs and unplaced blocks, and is trained through deep Q-learning. The powerful representation forms the basis for our hierarchical controller that jointly conducts high-level learning over action sequences and goal specifications together with low-level path planning, ensuring the execution of long-horizon tasks. Our extensive experiments confirm the representation' 's effectiveness and show extrapolation to environments with previously unseen target shapes, larger numbers of available elements, and different object types. When combining the learned Q-network with MCTS with computationally tractable small search budgets, we manage to improve performance and reliability across all tasks. Notably, we resolve the sequential long-horizon character of the assembly task by including the robot-in-the-loop to decide over feasible grasps and placing actions that ensure the stability of the construction. Our algorithm manages to correctly build structures using up to 22 building blocks with good success rates. In the future, we want to extend the algorithm to allow for a richer set of 6D grasping and placing poses, learn fine-placing low level, and investigate the implementation of an assemblyydisassembly strategy, so that the robot can potentially re-use wrongly placed blocks or reconfigure existing structures.
even in-hand manipulation controllers on the
8

---

Acknowledgments
The authors acknowledge the support from the Artificial Intelligence in Construction (AICO) grant by the Nexplore/Hochtief Collaboration Lab at TU Darmstadt. This project has also received fund- ing from the European Union's 's Horizon 2020 research and innovation programme under grant agree- ment No. 640554, and the Emmy Noether DFG Programme iROSA with Grant No. 448644653, The authors would like to thank the reviewers and metareviewer for their useful comments that helped to substantially improve the paper, and Prof. Dr. Oliver Tessmann from the Digital Design Unit (DDU) at the Department of Architecture at TU Darmstadt, for supporting with the robotic experiments.
References
[1] B. G. de Soto and M. Skibniewski. Future of robotics and automation in construction. In Construction 4.0, pages 289-306. Routledge, 2020.
[2] S. Tibbits. Autonomous assembly: designing for new of collective construction. John Wiley & Sons, 2017.
[3] J. O. de Haro, V. N. Hartmann, O. S. Oguz, and M. Toussaint. Learning efficient constraint graph sampling for robotic sequential manipulation. arXiv preprini arXiv:201 1.04828, 2020.
[4] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks as sequential interaction landscapes. arXiv preprint arXiv:2006.048: 04844, 2020.
[5] T. Ren, G. Chalvatzaki, and J. Peters. Extended task and motion planning of long-horizon robot manipulation. arXiv preprint wrXiv:2103.054450 2021.
[6] Y. Lee, E. S. Hu, Z. Yang, A. Yin, and J. J. Lim. Ikea furniture assembly environment for long-horizon complex manipulation tasks. arXiv preprint
2019.
[7] A. Mandlekar, D. Xu, R. Martin-Martin, S. Savarese, and L. Fei-Fei. Learning to general- ize across long-horizon tasks from human demonstrations. arXiv preprint arX1v:2003.06085, 2020.
[8] A. Simeonov, Y. Du, B. Kim, F. R. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. arXiv preprint arXiv:201 1.08/77, 2020.
[9] C. R. Garrett, R. Chitnis, Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez Integrated task and motion planning. Annual review of control, robolics, and autonomous systems, 4:265-293, 2021.
[10] Inoue, G. De Magistris, A. Munawar, T. Yokoya, and R. Tachibana. Deep reinforcement learning for high precision assembly tasks. In 20/7 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 819-825. IEEE, 2017.
[11] C. R. Garrett, Y. Huang. T. Lozano-Pérre Pérez, and C. T. Mueller. Scalable and probabilistically complete planning for robotic spatial extrusion. arXiv preprint arXiv:2002.022:00 2020.
[12] Y. Huang, C. Garrett, I. Ting, Parascho, and Mueller, Robotic additive construction of bar structures: Unified sequence and motion planning. arXiv preprint arXiv:2105./1138, 2021.
[13] V. N. Hartmann, O. S. Oguz, D. Driess, M. Toussaint, and A. Menges. Robust task and motion planning for long-horizon architectural construction planning. arXiv preprint arXiv:2003.07754, 2020.
[14] A. Jain and S. Niekum. Efficient hierarchical robot motion planning under uncertainty and hybrid dynamics. In Conference on Robot Learning, pages 757-766. PMLR, 2018
[15] B. Wibranek, Y. Liu, N. Funk, B. Belousov, J. Peters, and O. Tessmann. Reinforcement learn- ing for sequential assembly of sl-blocks Self-interlocking combinatorial design based on machine learning. 2021.

---

[16] M. Burke, K. Subr, and S. Ramamoorthy, Action sequencing using visual permutations. IEEE Roboties and Automation Letters, 6(2): 1745-12747 2021.
[17] K. Zakka, A. Zeng, J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEE International Conference Robotics and Automa- loo (ICRA), pages 9404-9410. IEEE, 2020.
[18] J. Huang, G. Zhan, Q. Fan, K. Mo, L. Shao, B. Chen, L. Guibas, and H. Dong. Generative 3d part assembly via dynamic graph learning. arXiv preprint arXiv:2006.07793, 2020.
[19] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484
2016.
[20] D. Silver, J. Schrittwieser, K. Simonyan, Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-33 359, 2017.
[21] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, al. Relational deep reinforcement learning. arXiv preprint arXiv: 1rXiv:1806.01830, 2018.
[22] T. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world models. arXiv preprint arXiv: 19.11191...2222 2019.
[23] Q. Cappart, D. Chételat, E. Khalil, A. Lodi, C. Morris, and P. Veličković. Combinatorial optimization and reasoning with graph neural networks. arXiv preprint arXiv:2102.09544, 2021.
[24] V. Bapst, A. Sanchez-Gonzalez, C. Doersch, Stachenfeld, P. Kohli, P. W. Battaglia, and J. B. Hamrick. Structured agents for physical construction. arXi::1904003177 [cs]. May 2019. URL http: htttp://axxvv org/ abs/ 1904. 03177. ZSCC: 0000017 arXiv: 1904.03177.
[25] G. Thomas, M. Chien, A. Tamar, J. A. Ojea, and P. Abbeel. Learning robotic assembly from cad. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 3524- 3531. IEEE, 2018
[26] J. Luo, E. Solowjow, C. Wen, J. A. Ojea, A. M. Agogino, A. Tamar, and P. Abbeel. Reinforce- ment learning on variable impedance controller for high figh-precision robotic assembly. In 2019 International Conference on Robotics and Automation (ICRA), pages 3080-3087. IEEE, 2019.
[27] R. Li, A. Jabri, T. Darrell, and Agrawal. Towards practical multi-object manipulation using relational reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pages 4051-4058, IEEE, 2020.
[28] Y. Lin, A. S. Wang, and A. Rai. Efficient and interpretable robot manipulation with graph neural networks. arXiv preprint 13X11.2102.33177, 2021.
[29] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[30] Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80 2008
[31] E. B. Khalil, H. Dai, Y. Zhang. B. Dilkina, and L. Song. Learning combinatorial optimization algorithms over graphs. In NIPS, 2017.
[32] W. Kool, H. Hoof, and M. Welling. Attention, leam to solve routing problems! In International Conference on Learning Representations, 2018.
[33] T. Barrett, W. Clements, J. Foerster, and A. Lvovsky. Exploratory combinatorial optimization with reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3243-3250 2020.
10

---

[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000-6010, 2017.
[35] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint arXiv:1710.10033,, 2017.
[36] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning, nature, 518(7540):529-333, 2015.
[37] J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, T. Pfaff, T. Weber, L. Buesing, and W. Battaglia. Combining Q-Learning and Search with Amortized Value Estimates. 1912 02807 [cs, stat], Jan. 2020. http://://aree org/ aa//ab///// http//bs///1....eea
ZSCC:
NoCitationData(SO] arxiv: 1912.02807.
[38] C. Browne, E. Powley, D. Whitehouse, S. Lucas, Tavener, D. Perez, S. Samothrakis, and S. Colton. A Survey of Monte Carlo Tree Search Methods. IEEE TRANSACTIONS ON COM- PUTATIONAL INTELLIGENCE AND Al IN GAMES, 4(1):50, 2012. ZSCC: 0002061.
[39] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[40] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. arXiv preprint arXiv:19066 08253,, 2019.
[41] A. S. Morgan, D. Nandha, G. Chalvatzaki, C. D D'Eramo, M. Dollar, and Peters. Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning. arXiv preprint arXiv:2 103. 13842, 2021.
[42] L. Koesis and Szepesvári. Bandit based monte-carlo planning. European conference on machine learning, pages 282-28 223. Springer, 2006.
[43] P.-.. Coquelin and R. Munos. Bandit algorithms for tree search. arXiv preprint cs/0703062 2007.
[44] C. D' D'Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters. Mushroomrl: Simplifying reinforcement learning research. https://githie www.///isarroolR//sstooooooooo http:://gg.ioo.o...a/aarrroooooa
-r1, 2020.
[45] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/pybuull
org, 2016-2021.
[46] I. M. Sobol'. On the distribution of points in cube and the approximate evaluation of integrals. Zhurnal Vychislitel noi Matematiki i Matematicheskoi Fiziki, ((4):784-022, 1967.
[47] J. Carpentier, G. Saurel, G. Buondonno, J. Mirabel, F. Lamiraux, O. Stasse, and N. Mansard. The pinocchio c++ library- fast and flexible implementation of rigid body dynamics algo- rithms and their analytical derivatives. In IEEE International Symposium on System Integra- lions (SII, 2019.
[48] Muratore. Simurlacra framework for reinforcement learning from randomized simula- 2020.
tions. https://gitee
11


Reviews:
[{"review_id": "uFZLpSBmD2K", "reviewer": "Reviewer_a8Bf", "summary": "This paper proposes a data-driven method for robotic assembly by leveraging structured state representations in conjunction with learning-based algorithms to effectively solve tasks. Specifically, the state representation is based on an graph neural network that incorporates attention mechanisms, and the learning algorithm is a hybrid model-based and model-free reinforcement learning algorithm. Experiments on a simulated assembly environment demonstrate that the proposed method can construct a variety of structures, outperforming variants with alternative state representations and learning algorithms.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The method is explained in a thorough yet concise manner\n* The experiments perform a systematic study of the state representation and learning algorithm \n* The experimental analysis is generally thorough: when there is a positive or negative result, the paper accompanies that result with a discussion of the underlying reasons behind the result\n\nWeaknesses:\n* The environment seems a bit simple and artificial —  the blocks are of the same shape and size and the unplaced blocks are laid out in a regularly spaced manner. Intuitively, it seems that the robot can choose to pick up any of the unplaced blocks, as they all have the same functionality. This makes the problem seem artificially simple. Furthermore, the target configurations for the blocks seem quite arbitrary, how do they translate to real-world structures that are of interest in assembly/construction applications? \n* The learning algorithms adopt elements from MCTS, a model-based algorithm. This raises the question, where does the underlying model come from? If making model predictions is based on having access to a simulator rather than a learned model, then this approach is unlikely to scale up due to the expensive computational load of simulators\n* Experiments can still go deeper, ideally answering the following questions: (1) how much of a difference does multi-head attention make compared to single-head attention? (2) what forms of generalization can the proposed method achieves and what forms can it not achieve? (3) are the representations learned by the proposed framework useful for transfer to semantically related tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is well-written, clear, and the experimental analysis is generally thorough. Ultimately, my primary concern is with this paper’s overall motivation and message. The paper positions itself as an applications-focused work, seeking to advance robotic automation for assembly and construction. Yet the experimental domain does not appear to be a convincing or inspiring medium through which to advance the state-of-the-art on these applications. On the other hand, the paper does not seek to make broader points about how its findings are useful to the general robot learning community. As a result, the impact of this paper (in its current form) appears limited. I recommend either revising the message of the paper to be more universal, or revising the experiments to demonstrate more impressive results in assembly / construction.\n\n**Update after rebuttal** my recommendation stays the same. Please refer to my followup post for a detailed explanation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "See the weaknesses list. In addition:\n* Many of the experiments are conducted without the robot — what is the motivation behind this? Perhaps the reason is that introducing the robot may introduce confounding factors in the analysis, in which case the paper should clearly justify why this may be the case     \n* Numerous citations to GNN-based works are missing, including but not limited to (1) Relational deep reinforcement learning, by Zambaldi et al, (2) Relational inductive bias for physical construction in humans and machines, by Hamrick et al, (3) Contrastive learning of structured world models, by Kipf et al"}}, {"review_id": "l_ZwQ7VpaoT", "reviewer": "Reviewer_2DWg", "summary": "The paper proposes a method to learn structure assembly policies via RL where using Graph Neural Networks. The input to the network is a graph representation of the scene, where each node represents an object (rectangular building block). The output of the network is the Q-value for different actions. The task is specified by stacking blocks to fill up a target volume. The authors evaluate a variety of RL algorithms in simulation and achieve a success rate of 85% in the best case and 49% in the hardest case (harder cases have more blocks and larger structures).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tThe proposed method of incorporating graph-based inductive bias in the construction task is suitable.\n-\tThe experiments and results are thorough.\n-\tThe analyses provide insights on comparing various graph-based representations, like multiheaded-attention and structure-to-vector.\nWeaknesses\n-\tMy main criticism of the paper is that it is poorly written, or at least, difficult to parse. The wording is dense both from a language and visual perspective. There is some key information I was unable to easily find in the paper. I list them as the following questions.\n-\tIs the task (target build volume) encoded in the observations, or do we need to train a new Q function for a new target? \n-\tOne of the main benefits of GNNs is its ability to generalize across different numbers of nodes and edges. In the case of this paper, can the GNNs trained on small number of objects generalize to bigger ones?\n-\tIs it the case that the current action encoding limits newly placed blocks can only be placed next to previously placed blocks? If so, it is unclear how the action encoding can be made more flexible to avoid this limitation when needing to build structures that have hollow volumes.\n-\tThe task volume visualization is also difficult to understand, both in the paper and in the videos. I think it may be clearer to represent target structures with semi-transparent volumes rather than outlines. It is also unclear how a set of rectangular blocks is supposed to construct some of the visualized structures that contain slanted sides.\n-\tThe improvement in performance when training with robot-in-the-loop may come at a cost of overfitting to the robot’s geometry and kinematics. How well would this transfer to new robots or even the same robot with a different end-effector?\n-\tIt would be helpful to see an additional baseline comparison against using MCTS with a simple hardcoded heuristic (value) function. This will illustrate the additional information learned by DQN.\n-\tWhile many experiments were performed, it is difficult to understand their significance and what solid conclusions should be drawn from them. Clearer discussions on this would have been helpful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation of weak reject is primarily due to the lack of clarity in the paper writing, which made it difficult for me to clearly assess the paper’s technical merits and contributions.\n\n**Update** I have updated my recommendation to weak accept after the authors' revisions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the Strengths and Weaknesses section.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "dxUAa_BaM8", "reviewer": "Reviewer_o7Zw", "summary": "This work proposes a pipeline to train a robot to learn to assemble blocks to match the given target shape. A graph neural network is utilized to encode the state information, and a deep reinforcement learning algorithm is applied to maximize the reward. The authors conduct extensive experiments to demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) This work proposes to leverage a graph neural network to extract the state information used to learn an assembly policy in a reinforcement learning fashion.\n2) Extensive evaluation experiments are conducted to demonstrate the effectiveness of the proposed approach.\n\nWeaknesses:\n1) The technical contributions are limited. Using graph neural networks to represent the state and deal with the combinatorial problem in the assembly process has been investigated by several works. For example, \"Generative 3D Part Assembly via Dynamic Graph Learning. Huang et al. Neurips 2022.\"\n2) Some important descriptions are missing. For example, it's not clear why a set of points is sufficient to represent the target shape. How the topology information is encoded in the set of points?  \n3)  The current setting is too simple. It's not clear what's the performance if more complicated objects are used. Whether the proposed pipeline could deal with more complicated objects? How to define the action space since there are more grasping poses? \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work proposes a pipeline to train a robot to learn to assemble blocks to match the given target shape leveraging the graph neural network to encode the state. However, the technical novelty is limited and the current setting is too simple. It's unclear whether the proposed pipeline could work with complicated objects.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1) in Line 102, \"desired shape that needs to be assembled consists of a set of points parameterizing this target shape\". It's unclear how the target shape is parameterized by the set of points. \n2) In Line 111, \"we have to decide over Na = 2 × 2 × 5 = 20 actions per block.\" It's difficult for the reviewer to understand these 20 actions per block. It would strengthen the paper if a visualization image is introduced here.\n3) In Line115-115, what is the function of F in equation (1). \n4) In Line 118-119, \"a threshold ∆ is exceeded, or whenever there are no more unplaced blocks available, or after an invalid action. \" What is an invalid action?\n"}}, {"review_id": "QdMYx3lM2rv", "reviewer": "Reviewer_7iDb", "summary": "This paper addresses the problem of autonomous assembly. Goals are defined by a set of 3D points forming a polygon shape that might be 2D or 3D. The robot's actions then consist of picking and placing blocks to construct the structure defined by the shape formed from the 3D points. The task notably has challenging characteristics like combinatorial explosion from a growing number of action, as well as others. A GNN representation with multi-headed attention mechanisms is proposed to represent the state of the constructed blocks and several RL methods are explored to perform model-based and model-free training. The work compares to the prior work structure2vector in one experiment, showing that the proposed architecture has more representational power essential to abstract robot assembly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-The paper is clear, concise, and well written. \n-The benchmarking environment presents another interesting problems for researchers.\n-The approach presents an interesting formulation of techniques to improve robot assembly from scratch\n\nWeakness:\n-The paper differentiated itself often in the related work that had a predefined goal in the form of a demonstration, but also has a predefined goal in the form of the 3D points. Perhaps some emphasis should be place on the abstraction level of the goal.\n\nSide comments:\n-Interesting future direction I thought would define the goals in natural language.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am unfamiliar with existing works on the robot structure assembly problem although I am very familiar with GNNs that include multi-headed attention mechanisms. As a result, I am assuming the main contributions applying the GNN with multi-headed attention mechanism to the robot assembly problem, robot assembly benchmark, and different combinations of learning algorithms for the problem. If other reviewers had strong opinions about missing baselines or prior work I would be open to those discussions. This is the reason for weak accept instead of strong.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Would it be possible to write Section 2.2 to rely less on existing algorithms from prior work? This might help readers that are unfamiliar with those works.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "Jp4iBZkFLqG", "reviewer": "Reviewer_RVUr", "summary": "The paper studies the problem of 3D assembly, more specifically having a 7 DOF robot arm stack blocks into a target shape. The paper aims to maximize combinatorial generalization for this task, by studying different choices of graph representations (through the use of GNNs and multi head attention) and monte carlo tree search algorithms (combined with Q-learning). The experiments show that a partially connected architecture for graph representation learning is superior than the fully connected version, and this is then tested in conjunction with DQN+MCTS algorithm for the best results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The flow of the paper is great, starting with a clear goal in mind, proposing very reasonable choices for each component of the pipeline with just the right amount of explanation, and testing the different components (representation and algorithm) separately.\n- The experimental section is thorough and clear. I really appreciate the targeted experimentation on each of the components, as it makes the results more reproducible and scientifically valuable for future work. The design of the novel benchmarking environment is also a contribution.\n\nWeaknesses:\n- It may be helpful to include some visualizations of the DQN+MCTS search procedure for a small toy example (e.g. with possible future states/actions and their estimated Q-functions). The toy example can also help better distinguish the different search methods described in section 2.2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem of 3D assembly is obviously important, and the paper provides valuable contributions in terms of empirical evaluation of graph representation learning and Q-learning / MTCS applied to this task. The presentation is clear and the experiments are thorough and provides potentially valuable insights for future work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See above.", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Strengths 
- The novelty of leveraging a graph neural network with multi-headed attention mechanism in manipulation assembly setting was noted by multiple reviewers 
- Extensive evaluation experiments to demonstrate the effectiveness of the proposed approach.

# Weaknesses 
- There are mixed reviews about writing clarity
- Multiple reviewers raised concerns about the simplicity of the task settings and connections to real world problems.
- Multiple reviewers noted that the technical contributions are limited
- Multiple reviewers highlight missing experimental and technical details.


# Recommendations
- Please outline limitations of the work 
- Thoroughly go through the recommendations provided by each reviewers to revise the paper
- This is a borderline paper with mixed reviews, please converse and address reviewers concerns

# Decision summary
- Extensive clarifications and additional experiments were received from the authors during the rebuttal period and were considered by the reviewers before final recommendation was made.
- Authors clarifications were acknowledged and factored in. Some concern around the experiments still remains.

We hope that the feedback received will help with the future directions of the work under submission. Thank you for participating in CoRL

Decision: Accept (Poster)