Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning
Authors: Manav Kulshrestha
Ahmed H Qureshi
Venue: CoRL 2023 Poster (2023)
Keywords: Rearrangement Planning
Robot Manipulation
Graph Attention

Abstract: Robotic manipulation tasks, such as object rearrangement, play a crucial role in enabling robots to interact with complex and arbitrary environments. Existing work focuses primarily on single-level rearrangement planning and, even if multiple levels exist, dependency relations among substructures are geometrically simpler, like tower stacking. We propose Structural Concept Learning (SCL), a deep learning approach that leverages graph attention networks to perform multi-level object rearrangement planning for scenes with structural dependency hierarchies. It is trained on a self-generated simulation data set with intuitive structures, works for unseen scenes with an arbitrary number of objects and higher complexity of structures, infers independent substructures to allow for task parallelization over multiple manipulators, and generalizes to the real world. We compare our method with a range of classical and model-based baselines to show that our method leverages its scene understanding to achieve better performance, flexibility, and efficiency. The dataset, demonstration videos, supplementary details, and code implementation are available at: https://manavkulshrestha.github.io/scl

Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning
Manav Kulshrestha, Ahmed H. Qureshi Department of Computer Science, Purdue University West Lafayette, IN 47907, United States mmkulshre, ahqureshi jtqureshi/@purdue. @purdue. edu
Robotic manipulation tasks, such as object rearrangement, play a crucial role enabling robots to interact with complex and arbitrary environ- ments. Existing work focuses primarily on single-level rearrangement planning and, even if multiple levels exist, dependency relations among substructures are geometrically simpler, like tower stacking. We propose Structural Concept Learning (SCL., a deep learning approach that leverages graph attention net- works to perform multi-level object rearrangement plamning for scenes with struc- tural dependency hierarchies. It is trained on a self-generated simulation data set with intuitive structures, works for unseen scenes with an arbitrary number of objects and higher complexity of structures, infers independent substructures allow for task parallelization over multiple manipulators, and generalizes the real world. We compare method with range of classical and model- based baselines to show that our method leverages its scene understanding to achieve better performance, flexibility, and efficiency. The dataset, demonstra- tion videos, supplementary details, and code implementation are available at: attps:///aanaiixullhresttt github. io/scl.
Abstract:
Keywords: Rearrangement Planning, Robot Manipulation, Graph Attention
Target
6
7
Achieved
10
11
13
14
15
16
Figure 1: Our approach performing progressive pick-and-place (we only show some place steps) actions based on its multi-level rearrangement plan to achieve (middle left) a target arrangement based on given goal (top left). A complete figure is available in the Appendix.
Introduction
Robots operating in the real world will often encounter variety of simple objects in com- plex arrangements and structures. To that end, recent years have rearrangement planning which involves the reorganization of objects in given environment to bring them into goal state emerging as prominent area of research within robotics [1]. Simpler applications of this problem include tasks such as setting the table, rearranging furniture, loading a dishwasher, and many more. While manipulation for many real-world tasks performed by robots often reduces to pick-and-plan and-elace, achieving more structured goals requires compositional interpretation the target and the execution of long-horizon hierarchical plans. Autonomously solving more complex rearrangement problems,
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

such as constructing a house or mechanical assembly, require robots to interpret internal dependence among the different parts of the whole structure and plan accordingly.
Despite the importance of these problems, most solutions addressing rearrangement planning largely consider target configurations with simpler dependencies such as blocking obstacles [2, 4], stack- ing objects in towers very simple structures [5, 6, 7, 8], or formulated as bin placement where supporting object is stationary [6, 9]; all of which are strict subsets of a more general construction task. One of the major hurdles for this task is to decipher the order in which objects need to be placed so as to properly construct given target structure since some objects depend on others in that they geometrically support them.
In this paper, explore toy scenario of the construction problem with known object primitives which are to be arranged to create some target structure. Our approach takes multi-view RGB-D observation of the target structure and constructs dependency graph using graph attention net- works. This graph captures the geometrical dependency among the different objects, identifying independent substructures for parallelization. Our structure planner then takes this graph and serial- izes task executions based on the current observations of the scene. The low-level controller further takes the task sequences and executes the tasks via robot control. Our results show better perfor- mance compared to other classical and model-based planners well generalizability to unseen structures of varying complexity. The main contributions of our approach are as follows:
A generated data set and generation procedure of target scenes containing intuitive structures with diverse complexity due to the possibility of variable number of objects, inclusion of structures with varying levels, and the possibility of independent substructures.
scalable dependency graph generator that generalizes to structures with multiple levels and varying numbers of objects.
A structured planner which uses the dependency graph to create a sequential plan for parallelized multi-level rearrangement planning. which is integrated into a complete pipeline from scene ob- servations to control executions.
2 Related Work
Rearrangement. The most relevant area of research to our work would be that of rearrangement planning, which is a subset of task and motion planning (TAMP). TAMP [10] involves planning for a robotic agent to operate in an environment with several objects by taking actions to move and change the state of said objects. Rearrangement specifically proposed an important challenge for embodied AI narrows this by defining itself as the act of bringing given environment to goal state [1]. Various strategies are utilized by systems aiming to address these problems. Some methods propose the combination of a high-level task planner with a low-level motion planner [11, 12, 13, 14). Others tackle the problem by using sampling-base
techniques in conjunction with search algorithms [15, 16, 17J. However, all of these focus on a yet narrower subset of rearrangement where objects are largely restricted to a 2D workspace and the interplay between objects largely ignored not present beyond dealing with clutter.
Deep Learning. More recently. advances have been made which utilize deep learning- based ap- proaches toward solving these problems. These allow relaxing the assumptions on objects in- volved by proposing more flexible collision detection [18, 9]. better generalization to the real world [19, 20, 21, 4, 6, 2]. and unseen environments [7, 2] through vision-based perception. Other methods utilize semantic information but use it to allow for more general goals like similarity to an inferred target distribution [22, 23, 24] guidance using language [25, 26). Perhaps most similar to our method, some approaches make use of graph neural networks to model object relations in the scene [8, 5]. However, in all the methods mentioned above, the target scenes are either restricted to single-level or are much simpler in terms of their construction, and the spatially weaker relations allow for less closely dependent substructures.
Construction and Assembly, Some works explore the task of assembly or construction [27, 28, 29], which allows objects to become fixed to one another that, unlike our case, does not require the creation of stable non-collapsing structures. Furthermore, in contrast to our approach, Blocks

---

Initial Scene X
Target Scene X
Initial Graph Generation
Node Encoder
Edge Decoder ht
Execution
((Ss for kc).
x
00
x (2.
(8%r &x, ks)H
X (
Segmentation
Pose Alignment
Structural Planner
Model architecture overview. We segment out object point clouds (PointNet+++ [30] based model), perform pose aligmment, and establish object correspondences (TEASER+ [31] based) be- tween X2 and Xy. Object level embeddings (another PointNet++ [30] based model) and positional embeddings (positional encoder [32]) create an initial graph G* which our node encoder ge uses output higher-level node features. Our edge decoder hy uses these to create dependency graph GD from which the planner outputs a valid sequence for robot rearrangement.
Figure 2:
[27] and RoboAssembly [28] also assume perfect information of objects in simulation, making generalization to the real world very difficult and requiring much higher planning steps due to the use of reinforcement learning. Additionally, while the aforementioned approach to construction [29] also performs long-houe nogghhorzzon planning like ours, they restrict their target be one of 4 possible predefined structures, though their focus is multi-robot planning.
Problem Definition
Let X 121,.42..... In. C =x the set of all points that can be occupied by a scene, be the powerset of X, and O {01, 02. .2.....,m. be the set of all object instances any given scene where each E R6 x C with being the set of all classes an object instance can take. Next, define scene
sets of points X, Y E such that X C Y where X represents the observable scene, whereas Y represents the complete scene. Furthermore,
can define subset selection operator as E S as S : + E where S(X) S X for all X E E. The goal is to construct high-level planner TH : x E + P that acts on partially observable sets X,, Xy E E of the initial and target scene to (SMr 8M1)} E P where S; E S is subset selection E indicates the number of steps,
produce a hierarchical plan P = (o))
operator, & A C SE(3) is a valid spatial transformation. and each 7th step P. = (S,, 5,) P contains the selection and transformation action. Hence, our objective is to determine plan P that selects subsets of point clouds, (So(X,), Sa (X1)), in the initial scene, X,, and transform them using ....... ... 3m }. in the minimum number of steps, M, to reach the an achieved state XA whose object specific point clouds are subsets of those given by the complete target state Yr. For application to robot rearrangement, we further define a low-level planner TL Q + A where Q and A refer to the configuration and action spaces for the robot. Every step p: = (S 5&) E P from the plan output by ти will have sequence of achievable configurations q6, associated with it that the low-level planner will take and further produce sequence of actions A E AA to achieve the intermediate state defined by the application of p; the previous state. For convenience and brevity, any arbitrary set where |u| n, be denoted as (n). Also, for any arbitrary set U, we denote its association with a scene by specifying subscript as U, and subset of U associated some object or characteristic w as a superscript U C U. And, for any graphs, superscripts denote different GD.
will denote some
notation for the remainder of this paper. Let
graph instances: GT G2 GTG
Method
This section details our approach for multi-level rearrangement planning to generate multi-step plan, the execution of which results in achieving the unseen structured target scene. Figure 2 shows overview of the model architecture and the basic flow of the approach. Additional model and implementation details are available in Section 7.2 of the Appendix.

---

Point Cloud and Feature Extraction. Partial point clouds are generated from the target scene and initial scene images. First, we obtain the RGB-D images from multiple viewpoint cameras sur- rounding the scene area and calculate their corresponding point clouds in the world frame from the camera's known world frame positions. We do preliminary filtering to remove outlier points. This is done for both the target scene and the initial scene to obtain XT and X,. A trained PoindNet++([3] based segmentation network then takes the scene point cloud and returns the segmented identity values for each point. Next, we extract the object-specific point clouds to obtain X and Xi for each object This makes up the Segmentation Module. As part of the Initial Graph Generation, we use another PointNet++ based network to extract object-level latent features for each object point (N) which will act as part of the node features for the scene graphs. Note that N is the number of objects in each scene.
cloud and concatenate them to get w (N) and Wey
Object Pose Alignment. Now is the task of object alignment and correspondence creation. Be- tween the target and initial scenes, we have multiple objects of the same type so we want to select correspondences such that the orientation change for an object o is minimized from the initial the target scene. To that end, for each object X in the target scene, we use its predicted y/ identity and sample its known mesh to obtain a default surface point cloud X. These complete, X and the partial point clouds, X from the target scene, are then used to predict a spatial transformation Tre using TEASER++ [31], which aligns X3 with Xi. This process is repeated for each object point cloud X in the initial scene matching whose identity y/ matches 0,'s 's identity y7 to get a set of transformations (Tamy T2-n: T J. These are then right multiply with Tr-a to ob- tain (Tre- T T--_.... From these, we choose the one which minimizes the magnitude of rotation from the initial to target scene (to avoid reorienting with multiple pick-and-place actions). This gives the canonical correspondence (and associated transformations T. for o, between the initial and target scene. This makes up the Pose Alignment Module. Finally, as part of the Initial Graph Generation, we obtain centroids for X (Tr-a)" and X2 (Tr-)) which are put into a positional encoder [32] to obtain positional features for each object in both the initial ,
and target scenes: ( _b(") Together, the object level features and positional features give the [
node features n {NN} = [w] [N3 67 ], ni (N}
[w (N) || (667
b for the objects in the scenes for the
initial graph, where || denotes concatenation.
Graph Node Encoder. The graph node encoder, go G - 2 is based on graph attention neural network [33, 34], known as GAT, which takes in initially fully connected scene graph GT -V, F) E 9 of the target scene with the aforementioned nf which contain both object- level features and positional features for each object in the target scene serving as the initial node features. These initial node features are updated using modified convolution that occurs for any node i using its neighbor set (i). The exact node feature update done by the convolutional layer is given by ni oz, Tn; + Ejev()) Ci,y Pn,, where & is the learnable parameter for the update mechanism and the attention coefficients a, which quantifies the importance of neighboring nodes, is given by
Lepky ng))
Oii.j EkEN() exp(a"Leakylll Leaky mx]))
(1)
where T' represents transposition and a is the learnable parameter for the attention mechanism (for derivation and more details, please refer to [33, 34]). The output from the graph encoder is a latent scene graph T² E Z for the target scene containing high-level features for each of the objects. In our problem setting, GAT-based model with multi-headed attentions being averaged outperformed vanilla GCNs.
The MLP-based edge decoder h : 2 - 'D maps the latent scene graph G² to the structural dependency graph G for the target scene containing inter-object specific dependency information. Specifically, hy can be queried with a pair of high-level features 2;, 2j representing objects o,, 0j and will decode them into the structural relationship between them. This relationship of dependence is asymmetric, and the decoder is used to query every ordered pair of nodes in the graph to obtain the respective dependence probabilities p XN) where Ptjj A[[zi]|zy]; V). Given
MLP Edge Decoder.

---

Planning Algorithm. Algorithm defines planning algorithm, which yields multi-step plan for performing efficient multi-level rearrangement. For given scenes, extract the observable in- formation and convert them into scene point clouds X,, Xy. These are taken and segmented into a collection of object point clouds x (x and their respective predictive identities y/ yf() (N), where N is the number of objects in scenes (Lines 1-2). Using these, calculate the corre- spondences for objects in the initial and target scene to produce reordered versions of the object point cloud along with transformations Ty-d for each object from the initial to target scene and pNNxN), we construct an inferred adjacency matrix for the structural dependency graph GD of the target scene with values GP = Pr. > 1" where 1" is some threshold value. The directed edges for this graph are visualized in Figure 2 as green arrows.
Structured Plan Creation. Once we have directed acyelic graph GP representing the inferred structural relationship between each pair of objects different objects in the target scene, perform topological sorting to obtain a valid sequencing with which the objects can be introduced so as to construct the structure in the target scene. Next, we make use of the aforementioned object correspondences between the initial and target scene to pick the object from its current position (formulated as a point cloud selection of S) in the initial scene and place it in the target position given by the associated spatial transformation 8 T obtained from pose alignment. All of this provides us with plan P {So: 60 ko), ........ (SN, 1SN,Ōx SN: ky)} where N is the number of objects. For plan step P: (S., 6,, ke) E P. & N E N denotes the dependence hierarchy identifier in the target structure that the object represented by the selection S, belongs to. Particularly, the plan is agnostic to any inversions involving any plan steps Pt. P, E P such that their associated hierarchy identifiers match: & = ky. That is, objects belonging to the same class of dependence hierarchy can be placed in any order relative to each other. This makes up the Structural Planner Module.
transformations (N) from objects' known default point cloud to their respective counterparts in the target scene (Line 3). Next, we use the target object point clouds x with PointNet++ [30] based model to extract object-level features ui (N) and the default point cloud transformed to the target pose with a positional encoder [32] to get higher level embeddings ( for the target posi- tion (Lines 4-6). Together these make up and construct the initial graph G*, which is provided to the graph encoder network Je to get a graph with higher level node features 2( for each object G (Line 7). Using these, we do ordered pairwise queries to the edge decoder fop to populate the probabilities p p[NxN) of the 7th object depending on the j"h object which are used along with threshold to determine the canonical structural dependency graph GO for the target scene (Lines 8-11). Given GD,, now do preliminary check of the inferred dependence by seeing whether the dependence graph is directed acyclic because if this is not the case, circular dependency ex- ists. Since all target scenes are known to be reachable with one manipulator, a cycle indicates prediction error and we report failure (Lines 12-13). Finally, we do a topological sorting of the dependence graph G and iterate over the result, using the predicted object transformations T((N) from the initial scene to the target to create rearrangement plan P (Lines 14-18). For each element (S, 5,, k;) E P. the execution will pick the object represented by the set of points (x ) and execute low level actions to result in an effective transformation of 6; Te on said points in an order such that & increases monotonically.
Figure 3: Examples of generated structures in dataset. For scale, the red cuboid is 3x3x6 cm3
Data Generation and Training. To train our models, we generate synthetic data containing in- tuitive structures built from a set of 8 possible object primitives, which are captured from 3 fixed viewpoints. We restrict the set of initial orientations for each object to allow only those which the bottom surface to allow for stable placement (c.g.,
provide a non-negligible surface
5

---

Algorithm 1: SCI.Plamning(2))
1 Xr, t PointCloudExtraationnn""
> initial and target scene point-clouds (peds)
2 X(N), x " yT +- Segmentation (X,, Xx) & objecct peds and their identities
X (N}
ITT.d Teh T Pose X (N) x
INN
1 ) align and correspond 91 DT
INd
(N) (bjjetFeetuures(xx
Wy
(N}_
с PositionalFeatures(XX (N) T/add M
b (N)
RT
+ NodeFeatures(wr)
1% }
> node features for initial graph graph with higher level node features
7 G = (2(1), + [ = =nn E))
pp p[NxN] +0
for (i,j) E do
| ( (NxN) hw (1=
10
Plg
> edge decoding for each pairwise edge
GD (N, p 1*)
creation of dependency graph based on threshold
if DGG then
return Circular Dependency Failure 13
> detecting unrecoverable prediction error
P(M] + 0
for i, i, E "opooogicalSorrinng(()) do
16 S 8, < RearrangmentSteppii T,(7) 7th object selection operator and transformation & adding step i to plan
17
P < P ((S; k;,
18 return P
cylinder is not allowed to be placed in a way where it may roll). And, for all but the final layer, only object orientations that provide a stable surface of non-negligible surface area will be selected for placements (e.g., a pyramid will not be placed upright but may be placed sideways). For the 1st layer, objects initially placed randomly within the bounds of the target scene, ensuring collision. Following this, 2 step process is repeated for each 7th layer. First, attempt to place objects supported by 2 objects in the previous layer. For this, we consider each pair of objects in the previous layer with available surface area and attempt to place an object in some valid orientation given the distance between the supporting objects. This is done by sampling some points on the top surface of the supporting objects and fitting plane to them, which the initial valid orientation is projected onto to get the object pose. Note that this allows for random placements that are not con- strained to be axis aligned. Once all such supporting pairs in the previous layer have been exhausted, move on to the second phase. In the second phase, we repeat the process but attempt to place objects on top of just one object from the previous layer. This is repeated until the step budget is exhausted. In simulation, we calculate the ground truth dependency graph using the y-components of the contact force vectors between each pair of objects. The graph encoder Go and edge decoder ho were trained together supervised manner to minimize the binary cross entropy loss between the adjacency matrices of the predicted and ground truth dependency graphs. Some examples of generated structures are given in Figure 3. We use PyByllet [35] for the simulation environment and use Trimesh [36] for collision and geometric checking. More information, including the generation algorithm's 's pseudo-code is provided in Section 7.1 of the Appendix.
5 Results
We perform four sets of experiments. First, we tested our method on unseen structures in the sim- ulated environment and compared it with some model-based and classical baselines. Second, we show how our method generalizes to multi-level object rearrangement tasks with structures contain- ing higher number objects. Third, we show our method's 's generalization to multi-level object rearrangement tasks with unseen structures consisting of higher number of levels than were in the training set. Finally, we demonstrate our method's sim-to-real generalization on multi-level object rearrangement tasks in the real world. The real world experiments also show our method gener- alizing to and operating on structures with objects placed in locations that cross between different

---

Performance Metries
Planner
Success (%) 1
Completion (%) t
Steps 4
Pos Error (m) 4
Om Error (0-1)4
SCL (Ours)
95.1
98.7
1.0
0.002 + 0.003
0.0024 + 0.0028
MLP
76.2
81.7
1.0
0.002 + 0.004
0.0024 1 0.0031
Classical Random
90.1
98.0
1.48 0.22
0.002 + 0.004
0.0029 + 0.0033
Classical Iterative
45.8
81.8
1.56 : 0.25
0.002 + 0.004
0.0021 + 0.0025
Table 1: Comparison between our approach with classical and model-based baselines. Our approach shows better-performing plans with fewer steps. Classic baselines were given planning budget of 2.V and take, on average, around 50% more steps for success than our approach, as indicated by the step factor.
levels something also not present in the training data. Also note that the errors and steps were only calculated for successful cases.
Evaluation Metries. We use the following metries for quantitative comparisons of different tasks. Success Rate: The percentage of successfully solved unseen scenes where success is defined as no object's achieved pose differing from the target position by more than cm or the target orienta- tion by 0.03 (normalized quaternion distance). Completion Rate: The percentage of objects whose achieved pose was within the success threshold (defined above) of their target in the scene, averaged over all scenes. Planning Steps: The ratio of the planning steps required to rearrange the objects from the initial scene the target scene and the number of objects in the scene, averaged over all scenes. Position Error: The mean Euclidean distance between object's 's achieved and target position for all objects in the scene, averaged over all scenes. Orientation Error: The mean quater- nion distance (1 92) = min + 22ll2. llqı 92 |2] [37]. normalized to be between 0 and 1, between the quaternions representing the achieved and the target orientations for all objects in the scene, averaged over scenes.
Baselines. The following baseline planners take over once the object correspondences between the target and initial scenes have been caleulated. Classical Iterative Baseline: This is model-free planner which iteratively selects objects from the initial scene that are not yet in their target pose. Once it has a candidate selected, it checks whether they can be stably placed in their target pose without falling by attempting a set of ray checks on the target location. If the check fails, continues the iteration. Once it finds a valid object for placement, places the object and restarts its iteration. Classical Random Baseline: This is a model-free planner which randomly selects objects from the initial scene that not yet in their target pose. Once it selects a candidate, it does stability check similarly to the other classical planner. If the check fails, it removes said object from its selection pool and continues random selection. If the check succeeds, it moves the object to its target pose and resets its selection pool to the current set of objects not in its target pose. MLP Baseline: This is an MLP-based object selection network that was trained on node features [n. 1 obtained from the scene after the execution of step pt from ground truth plan with the goal of having the
selection network learn and output the correct object selection operation M, E ps for planning. N
was set to a maximum value of 10 and node features were padded with zeros if the scene had fewer objects.
Comparison Analysis. Table shows our comparison results with the aforementioned baselines, evaluated on upwards of 400 scenes of unseen structures. Our approach outperforms all baselines in terms of success and completion rate. The MLP baseline had a tendency to get stuck in a local mini- mum for which object to move next, causing it to sometimes not be able to attempt rearrangement of the remaining objects, resulting in a lower completion rate. Classical baselines were given a budget of 2N steps (where N is the number of objects in a scene) and, on average, take around 50% more steps for success. Our approach generalizes to multiple objects, unlike MIP,, and provides better performance in fewer steps compared to the classical approaches.
Scalability Analysis. Table 2 shows our method's ability to generalize to scenes with variable number of objects, evaluated over more than 400 unseen scenes of structures with at most 3 layers. Our approach performs very well in scenes with less than 15 objects despite having been trained using a set containing an average of 9 objects. While a higher number of objects scene cause
7

---

Performance Metrics
Number of Objects (N)
Success (%) f
Completion (%)
Position Error (m) 4
Orientation Error (0- (0-1))
8 < N 10
95.1
98.7
0.002 0.003
0.0021 + 0.0028
10 & N < 15
93.9
98.2
0.002 0.003
0.0024 +: 0.0036
15 & N < 20
81.1
97.9
0.003 0.004
0.0028 + 0.0043
20 $ N < 25
73.7
96.7
0.003 0.004
0.0031 + 0.0054
Table 2: Performance of our approach as the number of objects in the scene increases. Larger number of objects in the same space results in occlusion, causing lower success rates but with the completion rates remaining high.
occlusion resulting in poorer prediction, our algorithm can still recover and deliver high completion rates even when success rates reduce. Table 3 shows the performance of our approach on over 1000 scenes containing meaningful structures with a variety of levels despite having been trained only on scenes containing at most 3 levels, showing its ability to generalize to novel and unseen structure configurations. On structures with less than levels, has 100% success rate. While occlusion due to denser structures causes lower success rate the number of levels increases, our algorithm still maintains high completion rate.
Performance Metries
Number of Levels
Success (96) 1
Completion (%)
Position Error (m) 4
Orientation Error (0-1),
1
100.0
100.0
0.001 0.000
0.0011 + 0.0004
2
100.0
100.0
0.002 + 0.003
0.0025 + 0.0038
3
94.5
98.6 6
0.002 + 0.003
0.0036 +: 0.0045
4
88.0
98.4
0.002 0.003
0.0036 + 0.0038
5
79.8
97.0
0.002 0.003
0.0038 + 0.0039
Table 3: Generalization of our approach to structures with higher levels. Meaningful structures with higher levels again cause occlusion, and we see similar trend of lower success rates but good completion rates. Note that our method only trained on structures containing at most 3 levels.
Sim2Real Generalization. We performed a set of real-world experiments using UR5e robot with a suction gripper and three Intel RealSense cameras for scene observation. We set up target structures ranging from 8 to 16 blocks with variable levels within the structure and block locations that cross between levels (example shown in Figure 1). Despite being trained in simulation with structures limited to 3 levels and discrete level layers, our method generalizes well to novel problem settings in the real world. The demonstration videos are provided on the project web-page.
6 Conclusions, Limitations, and Future Works
We presented Structured Concept Learning (SCL), a graph attention network-based approach for multi-level rearrangement planning. SCL an end-to-end approach which infers the dependency of objects and substructures to build multi-level structures from point cloud sets of the initial and target scene from RGB-D cameras. Our approach was trained on a novel data gathered by our intuitive multi-level structure generation procedure. In addition demonstrating its sim-to-real generaliza- tion, we evaluate our approach on challenging problems defined by target scenes containing different unseen structures with variety of objects and level hierarchies. As for limitations, very dense struc- tures can cause occlusion which results in very incomplete point clouds leading to faulty inferred dependence or incorrect spatial transformations for movement. The current approach also requires multi-view perception and lacks feedback control to account for execution error due hardware lim- itations such as forward momentum from the suction gripper. For future work, aim to augment SCL with robust segmentation and point-cloud shape completion to reduce incorrect predictions. Another future objective is to extend SCL with multi-robot task allocation to use its existing ability of finding independent substructures to control multiple robots for faster execution. More avenues for exploration would include analysis of various graph networks and their application to structural planning and augmenting the dataset with objects having more diverse dimensions, leading to even more interesting structures. Lastly, we would like to extend SCL to more general multi-robot tasks and incorporate multi-agent specific considerations allowing for improved complex execution ac- complish more demanding target states, including non-monotone cases and those requiring in in-place manipulation.

---

References
[1] D. Batra, A. X. Chang. S. Chernova, A. Davison, J. Deng, Koltun, S. Levine, J. Malik,
I. Mordatch, R. Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint
arXiv:2011.01975, 2020.
[2] A. H. Qureshi. A. Mousavian, C. Paxton, M. C. Yip. and D. Fox. Nerp: Neural rearrangement planning for unknown objects. Robotics: Science and Systems (RSS), 2021.
[3] H. Tian, C. Song, C. Wang, X. Zhang, and J. Pan. Sampling- planning for retrieving near-cylindrical objects in cluttered scenes using hierarchical graphs. IEEE Transactions Robotics, 39(1):165-188) 2023, doi: 100.1000/TO..202..311999
[4] B. Tang and G. S. Sukhatme. Selective object rearrangement in clutter. In Conference on Robot Learning, pages 1001-1010. PMLR, 2023.
[5] Y. Huang, A. Conkey, and T. Hermans. Planning for multi-object manipulation with graph neural network relational classifiers. IEEE Conference on Robotics and Automation (ICRA), 2023.
[6] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRL), 2020.
[7] H. Wu, J. Ye, X. Meng, C. Paxton, and G. Chirikjian. Transporters with visual foresight for solving unseen rearrangement tasks. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[8] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu. Hierarchical planning for long-horizon manip- ulation with geometric and symbolic scene graphs. International Conference on Robotics and Automation (ICRA), 2021.
[9] A. Murali, Mousavian, C. Eppner, A. Fishman, and D. Fox. Cabinet: Scaling neural col- lision detection for object rearrangement with procedural scene generation. arXiv preprint arXiv:2304.0990 2023.
[10] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano- -érez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[11] C. R. Garrett, T. Lozano-Péree and L. P. Kaelbling, Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 440-448, 2020.
[12] H. Cheong, B. Y. Cho, J. Lee, C. Kim, and C. Nam. Where to relocate?: Object rear- rangement inside cluttered and confined environments for robotie manipulation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 7791-7797. IEEE, 2020.
[13] R. Wang, K. Gao, D. Nakhimovich, J. Yu, and K. E. Bekris. Uniform object rearrangement: From complete monotone primitives to efficient non-monotone informed search. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6621-6627. IEEE, 2021.
[14] K. Gao, D. Lau, B. Huang, K. F. Bekris, and J. Yu. Fast high-quality tabletop rearrangement in bounded workspace. International Conference on Robotics and Automation (ICRA), 2021.
[15] H. Song, J. Haustein, W. Yuan, Hang. M. Y. Wang, D. Kragic, and A. Stork. Multi- object rearrangement with monte carlo tree search: A case study on planar nonprehensile sort- ing. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9433-9444 IEEE, 2020.

---

[16] Y. Labbé, S. Zagoruyko, I. Kalevatykh, I. Laptev, J. Carpentier, M. Aubry, and J. Sivic. Monte- carlo tree search for efficient visually guided rearrangement planning. JEEE Robotics and Automation Letters, 5(2):3715-3722, 2020.
[17] Lee, C. Nam, J. Park, and C. Kim. Tree search-based task and motion planning with pre- hensile and non- pon-ereeessile manipulation for obstacle rearrangement in clutter. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 8516-8522, 2021. doi: R0.110901RR.4455.220011.0 004445006.222955000
[18] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned im- plicit collision functions. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 6010-6017. IEEE, 2021.
[19] W. Goodwin, S. Vaze, I. Havoutis, and I. Posner. Semantically grounded object matching for robust robotic scene rearrangement. In 2022 International Conference on Robotics and Automation (ICRA), pages 11138-11144 IRFE, 2022.
[20] L. Weihs, M. Deitke, Kembhavi, and Mottaghi. Visual room rearrangement. Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5922- 5931, 2021.
[21] A. Goyal, A. Mousavian, C. Paxton, Y.-.. Chao, B. Okorn, J. Deng, and D. Fox. Ifor: Iter- ative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14787-14797, 2022.
[22] M. Wu, F. Zhong, Y. Xia, and H. Dong. Targf: Learning target gradient field to rearrange ob- jects without explicit goal specification. Advances in Neural Information Processing Systems, 35:31986-31992 2022.
[23] Q. A. Wei, S. Ding, J. J. Park, Sajnani, Poulenard, Sridhar, and L. Guibas. Lego- net: Learning regular rearrangements of objects in rooms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19037-19047, 2023.
[24] I. Kapelyukh, V. Vosylius, and E. Johns. DALL-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters, 8(7):3956-3963 jul 2023. 1o::10.110% lra.2023.327255 URL https: //t::///dooi/ http://ii.oggg.///////oe
org/ /0. 100.% 2023 202.. 3272516.
[25] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for language-guiin guided semantic rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 6322-6329. IEBE, 2022.
[26) E. Stengel-Eskin, A. Hundt, Z. He, A. Murali, N. Gopalan, M. Gombolay, and G. Hager. Guiding multi-step rearrangement tasks with natural language instructions. A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1486-1501. PMLR, 08-11 Nov 2022. URL https: http://ppoceeee...eeeep/ mlr. wwwpppeeeeeeeeeeee 2et.//////eeeel-ee -aangeleesknn222. htt.iin222...........
[27] S. K. S. Ghasemipour, D. Freeman, B. David, S. S. Gu, S. Kataoka, and I. Mordatch, Blocks as- semble! learning to assemble with scale structured reinforcement learning. International Conference on Machine Learning (ICML), 2022.
[28] M. Yu, I.. Shao, Chen, T. Wu, Q. Fan, K. Mo, and H. Dong. Rohoassembly: Learning gener- alizable furniture assembly policy in a novel multi-robot contact-rich simulation environment. arXiv preprint arXiv:2112.101433 2021.
[29] V.N. Hartmann, A. Orthey, D. Driess, O. Oguz, and M. Toussaint. Long- horizon multi-robor rearrangement planning for construction assembly. IEEE Transactions Robotics, 2023.
10

---

[30] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Conference on Neural Information Processing Systems (NIPS), 2017.
[31] H. Yang. J. Shi, and L. Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics (T-RO), 2020.
[32] Ortiz, A. Clegg, J. Dong, E. Sucar, D. Novotny, M. Zollhoefer, and M. Mukadam. isdf: Real-time neural signed distance fields for robot perception. Robotics: Science and Systems (RSS), 2022.
[33] Veličković, Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. International Conference on Learning Representations (ICLR), 2018.
[34] S. Brody, U. Alon, and E. Yahav. How attentive are graph attention networks? International Conference on Learning Representations (ICLR), 2022.
[35] E. Coumans, Y. Bai, and J. Hsu. Pybullet, 2015. URL https://pybullett org/
[36] Dawson- Howson-Haggerty et al. trimesh, 2019. URL http:::t//trri
[37] D. Q. Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35:155-164, 2009.
[38] M. Fey and E. Lenssen. Fast graph representation learning with pytorch geometric. Interna- tional Conference Learning Representations (ICLR), 2019.
[39] A. Paszke, S. Gross, Chintala, G. Chanan, E. Yang. Z. DeVito, Lin, A. Desmaison,
L. Antiga, and A. Lerer. Automatic differentiation in pytorch. Conference on Neural Informa- tion Processing Systems (NIPS), 2017.

---

This utilized PyTorch Geometric's 's [38] implementation of PointNet++ with an MLP attached the end to do classification our set of 8 object primitives. Our MLP used had 3 layers. The first layer had an input size of 1024 and an output size of 512, the second layer had an input size of 512 and an output size of 256, and the final layer had an input size of 256 and an output size of 8. To obtain the 7 Appendix
7.1 Data Generation
The basics of data generation are 8 possible object primitives to construct structures. All objects may be present in the top layer in any valid orientation, but middle (or supporting) layers may only contain certain objects in certain orientations conducive stable structures. For example, an upright pyramid would only provide top surface of single line and cylinder that is not upright would be likely to roll and collapse the whole structure so these are not allowed. Overall, we initially generated 8000 scenes for training and 2000 scenes for evaluation. Our generator was also used to create upwards of 10000 on-the-fly target structures for the evaluation our pipeline. A rough pseudo code for the structure generation algorithm is specified Algorithm 2, but the generation heuristic will be open-sourced with the final manuseript. Note that there are some implementation details regarding the algorithm not specified here, like how the placement criteria has metric involving the area of the top surface of objects, placement pose of objects is found by fitting plane sampled points from the top surface of the supporting objects in the lower level, validation of orientation for placement in layers is an exhaustive search, etc.
7.2 Model Architecture Details and Training
All graph neural networks were implemented using PyTorch Geometric (PyG) [38], and all conven- tional neural networks were implemented using PyTorch [39].
7.2.1
Positional Encoding
We utilized the positional encoding implementation specified in [32]. Specifically, the positional encodings be. for some object instance in the target scene are given by
by- (sr,) cos)) cos)) s2n (2))22_xx),cooo cos (20AAr))
(2)
where To is the 3D position vector, the position associated with o (which we get by calculating the centroid of the point cloud obtained from sampling the known default position, transformed to the target orientation), and the rows of A the outwards facing unit-norm vertices of twice- tessellated icosahedron. We use no offset, scale of 1, min degree of O, and max degree of 5 to calculate L, which results in an encoding of size 511. For more details, please refer to [32].
7.2.2 PoinNet++ based segmentation
We used PyTorch Geometric's [38] example model for segmentation, as is, without significant changes. The input point clouds from each scene were downsampled to have 1024 points using random sampling, and training was done in a supervised manner using negative log- log-likelihood loss calculated from the output and the ground truth point identities extracted from the simulation. We trained on a set of 8000 scenes and validated performance on 2000 scenes before use.
7.2.3 PointNet++ based feature extraction
object level features wi (N) for each object in the target scene, remove the final layer and take the 256-sized output to use as the object's 's latent features. The network was trained using classification
task for objects in over 800 scenes (each containing an average of 9 objects) and evaluated on objects in over 200 scenes before use.
7.2.4 GAT Graph Encoder
Our graph encoder zo contains 2 graph attention convolution layers, each of which convolves around every node in the graph using its neighbor set (i).. The exact node feature update done by the
12

---

Algorithm 2: Generation- Generation-AlgorihmmZ3 Algorithm(B, (K), O). B are the bounds for the scene, K(K} specifies the maximum number of objects on each level, K specifies the number of levels, and O are all possible object instances that can be placed.
Os ValidSubLeveOOjeeet(((
2 0p f-0 0
D number of objects placed on level objects placed
O
A(K) 0
D Ag contains all objects available for placement level
while co < Lo do
0 - SouudomlyPicc((C))
o E B and Not NotInCollision(.. O) then
PlaceObject(0)
+ 0 {o}
10
Ao + Ao U
Co +1
((N)
Gj the number of objects placed on level
1-
14
while
E K do
for (a, b) E A:- do
16
V + ValddPlacomenOOjjicttaaaaabbbb b, O,
b obj instances that can be supported by
b V + CollisionFree(r, O)
17
b filters to give only collision free placements
for do
19
PlaceOnOjjects(o,
places o to be supported by a, b
A; (o))
21
0 0 Ou
22
G + c +++1
23
c < L, then
24
break
c then
26
break
27
for a E 4,1 do
V ValddPaacmennOOjeetssaa.. O)D object instances that can be supported by a
28
29
V + CollisionFree(r, O) o do PlaceObject(o,
b filters to give only collision free placements
30
31
D places to be supported by
32
A, - (o) 0 Oufo
33
34
6 - G +1
35
if c < L. then
36
break
37
G < L, then
38
break
1-111
40
return O
convolutional layer is given by
n/. = oi, Pn
Σ JEN()
Oii,j Dn
where & is the learnable parameter for the update mechanism and the attention coefficients which quantifies the importance of neighboring nodes, is given by
c n2))
ai ExeN( exp(a" [2)m
13

---

where o is the non-linear activation LeakyReLU with a slope parameter of 0.2. Each of the graph convolution layers had 16 attention heads with averaging used as the aggregation function. The first graph attention layer has an input size of 511 and an output size of 256 whereas the other graph attention layer has an input size of 256 and an output size of 128. Training was done a supervised manner with
7.2.5 MLP Edge Decoder
The edge decoder he can be queried with pair of high-level features 24, 2j, resulting from the graph network encoder, and will decode them into the structural relationship between them. This relation- ship of dependence is asymmetric, and the decoder is used to query every ordered pair of nodes in the graph to obtain the respective dependence probabilitics p (NNXN) where pt.j = /[[=]||z]]: V). he has fully connected layers and uses LeakyReLU as its non-linear activation function after the first layer only. The first layer has an imput size of 2 128 256 and an output size of 128, whereas the second layer has an input size of 128 and output size of 1. We apply SoftMax to get the sociated probabilities for training with binary cross-entropy loss. The reason for the input layer for he being twice the output size for Jo is because finding the existence probability for the edge (i. involves concatenating the high-level node features 2,, 2j before inputting them into the edge de- coder. The graph encoder Jo and edge decoder hy were trained together in a supervised manner to minimize the binary cross entropy loss between the adjacency matrices of the predicted and ground truth dependency graphs, which, in turn, was obtained from simulation information (specifically, the g-components of the contact force vectors on each pair of objects).
Achieved
1
2
3
4
Target
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
A
22
23
24
25
26
27
28
29
30
31
82
Figure 4: Our approach performing progressive pick-and-place (with all steps shown) actions based on its multi-level rearrangement plan to achieve (top left) a target arrangement (middle left).
14


Reviews:
[{"review_id": "xmUKpHl_VUy", "reviewer": "Reviewer_mpK3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach based on a graph attention network for multi-level rearrangement tasks. This approach can reason about the dependency between objects at the same or different levels. With the reasoning of the dependency, this approach can sequentially pick and place objects to build a goal structure. Furthermore, based on the training in simulation, the framework can generalize to real-world scenarios. ", "summary_of_recommendation": "This paper proposes an approach based on a graph attention network to solve multi-level rearrangement tasks. This task is interesting for the community and the real-world demo with long-horizon executions is good. The assumptions are strong but if the authors discuss more about how to solve this in future work, this will be a useful paper for CoRL. Therefore, I recommend \"weak accept\". ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1: The authors should discuss more how to learn the spatial transformations as an end-to-end learning. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths of the paper: \n\n1: Building the stack with a meaningful structure is interesting. \n\n2: The real-world demo with a long-horizon sequence of actions is cool. \n\n3: This approach is trained purely in simulation and can generalize to real-world without any fine-tuning. \n\nWeakness of the paper:\n\n1: The assumptions of the paper are strong. This paper assumes three cameras to get a calibrated point cloud and then leverages this calibrated point cloud to calculate the spatial transformations. \n\n2: The assumption of known spatial transformations makes the task much easier. Thus, even if the SCL performs great, one of the baselines (classical random) also performs more than a 90% success rate and completion rate. \n\n\nMinor points:\n\n1: Figure 2 is hard to follow and understand. \n\n2: multi-head instead of multi-heads. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "v5L1Dk-4rf", "reviewer": "Reviewer_hfsk", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method called Structural Concept Learning (SCL) for multi-level object rearrangement planning in robotic manipulation tasks. SCL leverages graph attention networks to generate a hierarchical plan for scenes with structural dependency hierarchies. It uses a self-generated simulation dataset with intuitive structures and can handle scenes with an arbitrary number of objects and complex structures. The method infers independent substructures to enable task parallelization over multiple manipulators and generalizes well to the real world. The proposed approach is compared against classical and model-based baselines, demonstrating better performance, flexibility, and efficiency.", "summary_of_recommendation": "The paper proposes SCL, a novel multi-level object rearrangement planning method using graph attention networks (GAT). The approach demonstrates promise in handling complex scenes with structural dependencies, making it valuable for robotic manipulation tasks. However, several weaknesses need addressing. Firstly, evaluating the method on real-world data is crucial to assess its robustness and practical applicability. Secondly, discussing the complexity and efficiency of the approach, particularly for larger scenes or multiple manipulators, would enhance its practicality. Additionally, the paper lacks a clear explanation of how GAT's attention mechanism benefits hierarchical planning. Comparing GAT's performance against other standard GNNs, like GCNs or GraphSAGE, would bolster the paper's claims. The authors are encouraged to address these concerns in the revision to strengthen the paper's impact and significance in the field.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I fail to see the benefits of using GAT for multi-level rearrangement planning. The paper claims that a GAT-based model with multi-headed attentions outperformed vanilla GCNs. Can you provide theoretical or empirical evidence to support this claim?\n\nThere are other standard GNN structures like GraphSAGE. Why was GAT chosen over these alternatives? Is there a specific advantage of GAT in the context of this problem?\n\nWhat is the depth of the GAT model used in this work? DeepGCNs have shown good results in point cloud segmentation by going deep. Though here, it is used as an encoder, can you provide insights into the choice of the GAT depth?\n\nThe process of obtaining edge embeddings based on node embeddings using an MLP decoder may lead to loss of topology information. For instance, nodes zi1, zi2 with the same embedding may have different relationships. Can you clarify the benefits of obtaining a new adjacency matrix in this manner? Why not directly use GAT's weight matrix as the new adjacency matrix? It seems like you are using learned nodes to infer the learned adjacency matrix, and I am unsure of the advantages of using an MLP decoder over GAT's adjacency matrix.\n\nOverall, I did not observe the advantages of using GAT in Rearrangement Planning based on the presented results. Can you provide more insights or explanations to highlight the benefits of GAT in this context?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an innovative approach for multi-level object rearrangement planning using graph attention networks. The proposed SCL method demonstrates promising results in handling complex scenes with structural dependencies, and its generalization to the real world is a significant contribution. The paper is generally well-written and provides a clear description of the method's architecture and implementation details. The use of a self-generated simulation dataset and comparison against baselines strengthens the paper's empirical evaluation.\n\nThe main strength of the paper lies in its novel combination of graph attention networks for multi-level planning, which addresses an important problem in robotic manipulation. The method's performance, flexibility, and efficiency make it a valuable contribution to the field.\n\nHowever, some weaknesses should be addressed. First, the paper relies on a synthetic dataset, and it would be beneficial to evaluate the method on real-world data to assess its robustness and practical applicability. Second, the complexity and efficiency of the approach should be discussed further, as the method may be computationally demanding for large scenes or multiple manipulators.Moreover, the paper lacks a clear explanation of how GAT's attention mechanism is relevant to hierarchical planning. A more detailed discussion of the advantages of using GAT over other standard Graph Neural Networks (GNNs) like Graph Convolutional Networks (GCN) or GraphSAGE would strengthen the paper's arguments.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "_YTdoiTlFm", "reviewer": "Reviewer_Ej1j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed Structured Concept Learning, a graph attention network-based approach for multi-level rearrangement planning. Particularly, their approach takes a multi-view RGB-D observation of the target structure and constructs a dependency graph using graph attention networks. First, the model segments out object point clouds, perform pose alignment, and establish object correspondences to different scenes. The object level embeddings and positional embeddings create a graph that would be used by the graph node encoder to give higher-level node features. Then the edge decoder uses this information to create a dependency graph from which the planner outputs a valid sequence for robot rearrangement. The work conducts 4 types of experiments to show their performance, flexibility, and efficiency. They also performed a set of real-world experiments to demonstrate their generalization.", "summary_of_recommendation": "The paper proposed a novel idea in the usage of node encoder and edge decoder to extract extracts the high-level features for each of the objects and recovers the inter-object specific dependency information for planning. Most of the experiments are well conducted, show the strength, and provide insight for the model. However, there seems to be one missing experiment to show the execution time and parallelization over multiple manipulators. There are also some missing information and explanation.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What is the success threshold for the completion rate? What is the exact size of the objects, or their relative size compared to the environments? Without knowing those, the position errors do not provide much information.\n- Why are the classical baselines given a budget of 2.N since it’s stated that 2.N is too little for an iterative approach? Is N the number of nodes or iterations? From the previous section, N is the number of nodes, then how does it affect the planning?\n- In the data generation, the work mentions that after the first layer, there is a 2-step process for each layer in which they attempt to place an object by 2 objects in previous layers and then 1 object in the previous layer. Is it possible to have an object to be supported by more than 2 objects in previous layers? Could a random number (1-4) be chosen for this procedure? Would that generate more diverse training datasets? From the experiments, we know that there could be up to 25 objects in a scene.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The authors clearly define the problem they want to solve.\n- The usages of graph node encoder and MLP edge decoder are very novel. The graph node encoder extracts the high-level features for each of the objects within the scene. The MLP edge decoder recovers the inter-object specific dependency information.\n- The planning algorithm is well-explained and makes sense.\n- The experiments are quite thought-out. They show the model performs well on challenging problems defined by target scenes containing different unseen structures with a variety of objects (up to 25 objects) and level hierarchies (up to 5 levels).\n- The work performs experiments on real world robots. Thus, it shows the feasibility of the approach for practical application.\n\nWeaknesses:\n- The work claims the approach allows for task parallelization over multiple manipulators but fails to show any result to support that.\n- The work mentions 5 sets of experiments, but it only shows the results for 4 of them.\n- Do all the compared methods have the same run time? If not, the authors should include the total run time for each compared method. Since both classical iterative baseline and classical random baseline have an iterative approach for solving the planning problem and the time they run will affect their performance.\n- In the limitation section, the authors mention dense structures can cause occlusion and in future work, the authors plan to augment SCL with robust segmentation, point cloud shape completion, and matching techniques. However, most of these are gotten from\nother works (such as PointNet++ and TEASER). So, these limitations seem to be not from the proposed work but from the other works. The discussion of limitations is a bit lacking.\n-  There are a few typos within the paper such as the figure 2 caption.\n\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "B6bZIwmd8-", "reviewer": "Reviewer_fLRU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on the rearrangement planning task, which involves the reorganization of objects in a given environment to bring them into a goal state. The paper particularly explores a scenario of constructing a structure with known object primitives. The proposed method takes a multi-view RGB-D observation of the target structure and constructs a dependency graph using graph attention networks, which captures the geometrical dependency among different objects, identifying independent substructures for parallelization. Then, a structure planner serializes task executions based on the current observations of the scene. A low-level controller further executes the tasks via robot control. The results demonstrate superior performance and generalizability compared to other classical and model-based planners.", "summary_of_recommendation": "The paper provides valuable insights into rearrangement planning, proposing an innovative method that showcases potential scalability and generalizability. Despite modest improvements over the baseline, its contributions to the dataset and procedure generation make it a noteworthy addition to the field.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Could you justify the proposed architecture given the 5% improvement over the baseline?\n2. Can you clarify the impact of the fixed rotation axes assumption on the method's robustness?\n3. How does your system manage unexpected failures, considering it operates as an open-loop controller?\n\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1. The paper was a pleasure to read, offering a well-articulated and lucid description of the implementation details.\n2. The approach of employing a graph architecture to model and infer the relationships between blocks is both innovative and logical.\n3. The real-world experiment outcomes were both promising and engaging to observe, which points to a good system design.\n\nLimitations:\n\n1. The enhancements demonstrated by the proposed method appear marginal when contrasted with the traditional random baseline in Table 1. It shows only a 5% improvement in task success rate, which raises questions about the necessity and efficacy of the proposed architecture.\n2. The proposed method hinges on an assumption that two rotation axes of an object must remain in their initial state, as the robot cannot perform reorientation along these two axes (primarily x and y axes). This assumption substantially constrains the task's state space. It would be beneficial for the authors to explicate this assumption within the problem formulation.\n3. The overall framework appears to function as an open-loop controller, which may lack the capability to rectify failures in a closed-loop manner. Is there a re-planning mechanism designed to handle unexpected failures?\n4. Why not employ the simulation directly as a model and use Model Predictive Control (MPC) to generate viable task plans? Given that the task is kinematic pick-and-place without dynamic movements, what are the primary challenges for achieving sim-to-real generalization in this context?\n5. How can the proposed method be applied to more significant and impactful manipulation domains?\n6. What is the distribution of objects in the initial state? Does the model have the capacity to dismantle an existing structure and construct a new one?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach to the rearrangement planning task, focusing on the reorganization of objects in an environment to achieve a goal state. The method leverages multi-view RGB-D observations and graph attention networks to capture geometrical dependencies between objects, enabling the identification of independent substructures for parallelization. A structure planner and low-level controller execute tasks, showcasing promising real-world performance and generalizability. The paper introduces an innovative approach using graph attention networks to model object relationships, enhancing the understanding of the dependencies between blocks in the rearrangement task. The real-world experiments demonstrate the efficacy of the proposed approach, showcasing its potential for scalability and generalizability.
Concerns were raised about the modest improvement (5%) over the traditional random baseline.  The authors justified this by highlighting the budget constraints and the ability of their approach to execute tasks within N steps. There were queries regarding the assumption that two rotation axes of an object remain fixed.The approach appears open-loop, lacking a mechanism to rectify failures. Authors acknowledge this and leave handling unexpected failures for future work, suggesting that dependency graph errors can be recovered from. The question of employing Model Predictive Control (MPC) for generating task plans directly from simulation was raised and in the rebuttal stage, the authors highlight the computational cost of MPC and emphasize their approach's scalability.
The paper introduces some contribution to the field of rearrangement planning, leveraging graph attention networks for modeling dependencies and achieving promising real-world results. While some concerns were raised regarding the improvements over baselines and assumptions, the authors' responses addressed most of the queries satisfactorily. The paper's technical quality, clarity, and innovation were generally acknowledged.
Decision: Accept (Poster)