Selective Object Rearrangement in Clutter
Authors: Bingjie Tang
Gaurav S. Sukhatme
Venue: CoRL 2022 Poster (2022)
Keywords: Rearrangement
Robot Manipulation
Task and Motion Planning

Abstract: We propose an image-based, learned method for selective tabletop object rearrangement in clutter using a parallel jaw gripper. Our method consists of three stages: graph-based object sequencing (which object to move), feature-based action selection (whether to push or grasp, and at what position and orientation) and a visual correspondence-based placement policy (where to place a grasped object). Experiments show that this decomposition works well in challenging settings requiring the robot to begin with an initially cluttered scene, selecting only the objects that need to be rearranged while discarding others, and dealing with cases where the goal location for an object is already occupied – making it the first system to address all these concurrently in a purely image-based setting. We also achieve an $\sim$ 8% improvement in task success rate over the previously best reported result that handles both translation and orientation in less restrictive (un-cluttered, non-selective) settings. We demonstrate zero-shot transfer of our system solely trained in simulation to a real robot selectively rearranging up to everyday objects, many unseen during learning, on a crowded tabletop. Videos:https://sites.google.com/view/selective-rearrangement

Selective Object Rearrangement in Clutter
Bingjie Tang Department of Computer Science University of Southern California United States bingjiet@use edu
Gauray S. Sukhatme"
Department of Computer Science University of Southern California United States gauravOusc.o edu
Abstract: We propose an image-based, learned method for selective tabletop ob- ject rearrangement in clutter using a parallel jaw gripper. Our method consists of three stages: graph-based object sequencing (which object to move), feature- based action selection (whether to push or grasp, and at what position and orientation) and a visual correspondence- bbasendeeeeeddddddeee placement policy (where to place a grasped object). Experiments show that this decomposition works well in challenging set- tings requiring the robot to begin with an initially cluttered scene, selecting only the objects that need to be rearranged while discarding others, and dealing with cases where the goal location for an object is already occupied making it the first system to address all these concurrently in purely image-based setting. We also achieve an ~88 improvement in task success rate over the previously best reported result that handles both translation and orientation in less restrictive (un- cluttered, non-selective) settings. We demonstrate zero-shot transfer of our sys- tem solely trained in simulation everyday objects, many unseen during learning, on crowded tabletop. Videos: google......oooooo.o www.loo..o/viee//////oooo///cc// rerrreeeeeeeeeerrrreeeeeennnttt
real robot selectively rearranging up to 15
https://sites: http://iitss.ogle..../..
Keywords: Rearrangement, Robot Manipulation, Task and Motion Planning
Introduction
Goal
Achieved
Selected Objact Moving Direction
Target Position All non-target objects moved to side bin. No target positon specified.
Figure 1: 15-object selective rearrangement from cluttered initial state. Given an initial ar- rangement everyday objects and an image specifying the goal arrangement, the robot learns to remove objects that do not need repositioning (1-11) and repositions all other objects accurately (12-15) as specified by the goal image (top left) resulting in the final arrangement (bottom left).
Repositioning objects to desired configuration is rooted in the activities of daily living [1]. Many skills underlie this capability extracting useful information from raw perceptual data, performing accurate object manipulation, and optimizing long-term sequential action planning making object rearrangement an essential challenge for both roboties and embodied AI [2]. Figure illustrates our
"Gaurav Sukhatme holds concurrent appointments as Professor at USC and an Amazon Scholar. This paper describes work performed USC and not associated with Amazon.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

setting: faced with a tabletop with many everyday objects (clutter) the robot is tasked to rearrange a subset of objects (selectivity) to a goal configuration, while discarding others in bin. Another challenge is in situations where the desired locations for some objects are already occupied (swap). Object rearrangement has been studied in the context of both task and motion planning and learning- However, existing methods do not concurrently address these three challenges. Our system is the first to do so in a purely learned setting where the goal is given by a single RGB-D image.
In contrast to e.g., suction mechanisms, we use a parallel jaw gripper requiring object singulation before grasping. Our method consists of three stages: graph- object sequencing that picks the next object to manipulate by minimizing the Graph Edit Distance (GED) between the current scene graph and the goal scene graph. caature-based action selection that maps the RGB-D image to robot actions (pushing or grasping) through a deep Q- Qllearning framework, and a visual correspondence- based placement policy that uses the cross-correlation visual features between the grasped object and the goal specification image to locate object placement. Experiments show the successful rear- rangement of objects with >90% completion (error <2.99 cm), and 16-20 objects with 882.33% success (error <1.64 cm). We also achieve ~8% improvement in task success rate over the pre- viously best reported result that handles both translation and orientation in a less restrictive setting (uncluttered, non-selective), We demonstrate zero-shot transfer to real robot (Figure 1) selectively rearranging up to 15 everyday objects, many unseen during learning, on a crowded tabletop.
2 Related Work
Task and motion planning-based systems (TAMP) [3] either have high-level task planner and low-level motion planner [4, 6, 7, 8, 9, 10], or use sampling- based algorithms or optimization to solve single unified formulation of the problem [11, 12]. Some TAMP solutions rely on known ob- ject models or a known environment [13, 9, 10], which makes it difficult to deploy them with novel objects or where explicit object pose estimation is difficult to obtain. TAMP approaches that incorpo- rate learning- -bandd vision models, such as [14, 15, 16, 17] can adapt to novel objects/environment while [14] is based on one initial scene image, [15] uses structural constrained predicates for plan- ning, [16] depends on the knowledge of the environment, and [17] assumes round collision radius for all object shapes, making it difficult to scale to adversarial environments (e.g.. highly cluttered). The number of possible action sequences increases exponentially with the number of objects and changes in environment observability increase the difficulty of back-tracking and replanning.
Deep bearning-based systems have relaxed some of these constraints by incorporating yearning-based models in perception, planning and actuation. They have been shown to learn general policies to handle varied rearrangement tasks [27, 28, 29, 30, 31, 32], e.g., highly- cluttered, partially-observable environments or deformable objects. Our work is related to beasinngbbssed methods for grasping in clut- ter [18, 19, 20, 33], target object retrieval [21, 22, 23]. and rearrangement [24, 25, 10, 26] (Ta- ble 1). Most related to our work, Zeng et al. [20] and Tang et al. [33] proposed using deep Q-learning to learn synergies between push and grasp improve grasping accuracy in densely cluttered environment. Inspired by [20] and [33], adapt collaborative PUSH and GRASP to solve highly cluttered environments for object rearrange- ment tasks. Different from previous works, we learn action primitives, distinguish objects to rear- range from those to discard, and plan sequential actions simultaneously making this the first work to concurrently solve image-based selective object rearrangement in cluttered tabletop environment.
Method
Robot Action
Clutter
Selectivity
Grasping
DexNet [18]
GRASP
GPD [19]
GRASP
x
VPG [20]
PUSH&GRASP
x
Targei object retrieval
Mech Searh [21]
GRASP
V
Murali et a). [22]
GRASP
MORE [23]
PUSH&GRASP
V
Rearrangement
NeRP [24]
GRASP
X
IFOR [25]
GRASP
X
TRLB [10]
SUCTION
RevrientBot [26]
SUCTION
Ours
PUSH&GRASP
Table Related Manipulation Tasks
3 Learning a Selective Rearrangement Policy
We decompose the rearrangement problem into three parts: object sequencing (which object to re- locate next), action selection (how to manipulate it), and object placement (where to place a grasped object). We rely on three primitives: pushing objects (PUSH), picking them up (GRASP), and placing

---

Graph-bosed Object Sequencing
Esccute Selected Astion
Correspondente--op Rippsiition
Camera Input
Pesture-based
Current
Upstae
Action Enscution Roward
X Creage 0 Paate
Figure 2: System overview. Our system uses RGB-D images as input and builds a scene graph based on the object segmentation given by UOIS-Net-3D [34]. Graph-based object sequencing (subsection 3.3) selects the optimal object for next rearraneement and we mask the Q-value map for GRASP with its segmentation mask. Then the system picks the highest Q-value action candidate from PUSH and GRASP Q-value maps and executes the action Figure 3a. If GRASP ASP is chosen and successfully executed, the system locates the PLACE of the grasped object (Figure 3b).
them at target locations (PLACE). PUSH and GRASP can be initiated by the robot at any time, how- ever PLACE can only be performed if the robot is already holding an object. This suggests a natural decomposition into our three part strategy. When the robot is not holding an object, it must make a decision on which object to manipulate next (object sequencing). After choosing an object, it must decide whether (and how) to push the selected object or whether (and how) to pick it up (action selection). When holding object, it must decide where to place (object placement). We model object sequencing as a supervised learning problem on graph transformations (subsection 3.3), ac- tion selection as Partially Observable Markov Decision Process (POMDP) (subsection 3.1), and object placement as supervised learning problem (subsection 3.2).
3.1 Feature-based Action Selection: PUSH or GRASP
The choice of whether PUSH or GRASP (and what location and orientation) is Markovian since it is based solely on the current state (object poses). Further, the state is partially observable we do not assume the robot has direct access full state information, needs be inferred from images. Hence, we formulate the problem of selecting whether to push or pick up an object (and at what location and orientation) as goal-conditioned POMDP where S is the state space, G is the set of goals, A is the action space, p(s (st at) is the time- invariant (unknown) dynamics function, R S A + R is the reward function, Ω is a set of observations, O is a set of conditional observation probabilities, y E [0 is the discount factor, Pn the initial state distribution, and Pg is the goal distribution. The objective is to obtain a policy r (ar|St g) to maximize the expected sum of rewards EE R((s,, g)). where the goal is sampled from Pg and the states are sampled according to So ~ po. and $ ~ p((5+ (Se: at).
tuple (S, G, A. p. R, Q2 O, 7, Po. Py)
We define the state as the poses of N objects in the scene. The actions a E A consist of the choice of action tb, end-effector position x and planar orientation 0: a (v,,, 0), (PUSH, GRASP} x E R3, 0 E R. We choose sparse reward for actions 1.0 for successful GRASP and 0.5 for successful PUSH. The higher reward for GRASP incentivizes the robot to prioritize it over PUSH when both are available. We consider a PUSH successful if the pixel-wise change in the depth image after PUSH larger than a pre-defined threshold. The intuition behind designing the PUSH reward this way is that we only use it for singulating objects in clutter where direct GRASP is not available, not for object rearrangement, between the parallel-jaw gripper fingers after GRASP attempt is higher than pre-defined thresh- old. Observation o is defined as the RGB-D image captured by a statically mounted camera. The goal specification 0g is the RGB-D image of the goal arrangement from the same camera viewpoint.
GRASP considered successful the antipodal distance
Given the current observation o we use fully convolutional neural networks (FCNs) model Q- functions that estimate the expected reward for each PUSH and GRASP candidate. The deep Q- leamnigg framework shown in Figure 3a. A 121-layer DenseNet [35] pretrained on ImageNet [36] is used to extract visual features from raw RGB-D images. In each FCN, we have three convolutional layers; we apply batch normalization and ReLU activation before every convolutional layer. After FCN, we upsample with bilinear mode to have pixel-wise Q-value estimate of the same size as input images. Each pixel unit in the Q-value map corresponds to the expected reward for executing an action at this pixel location. For each action we model end-effector orientation
3

---

Current Observation
Grasp FCN DenseNerri
wiee Map
CTS with 20
Adoer Suaecerr
soopa
Push FCN (Dana/li1121)
PRO wid
Map
Ganpo EMEN
(b) Correspondence- based Reposition Figure 3: Subpolicies. (a) A deep Q-learning framework maps the visual observations to actions,
(a) Feature- based Action Sclection
similar to [20]. (b) The grasped object placement is conditioned the cross-correlation between the visual feature of the goal scene and the local features of the grasped object.
by rotating og to 16 difffrrnnt orientations. Thus have 32 pixel-wise Q-value maps (16 each for PUSH and GRASP). Each represents the Q-value estimate of executing the corresponding action at that orientation at all pixel locations. At each timestep t, before the robot chooses the next action, we mask all 16 GRASP Q-value maps with the output from the graph-based object rearrangement sequencing module (Figure 3b) to rule out objects that do not currently need to be repositioned. Following this, the robot picks an action (PUSH or GRASP) with the highest Q-value and executes the corresponding pixel location and end-effector orientation.
Loss is calculated by computing the temporal difference (TD) between the estimated reward and the actual obtained reward after execution. We only compute the loss for the selected pixel/pose (where the robot will take the next action); all other pixels/poses backpropagate with loss We generate the label for PUSH at time f, 2u by calculating the depth image change after the push if it is higher than a predefined threshold we consider the PUSH successful, yf = 0.5. For GRASP, we obtain the label time t, 3/ yr via the feedback signal from the gripper, if the antipodal distance between parallel jaws is larger than a predefined threshold, we consider the gripper is holding the object and hence the GRASP is successful, S. = 1. We use Huber Loss for both PUSH and GRASP. For action executed at time I, let yt denote the label, Qr denote the estimated reward. The TD is given by IQt- yrl, and the primitive learning loss is calculated
C, { 3 (Q: -y) 10: -0l
1Q--a yı 1, otherwise.
3.2 Correspondence-baseee Reposition: Where to PLACE
We model finding PLACE pose at time a template matching problem [37] conditioned on the current observation ot, the goal specification Og. and the successfully executed GRASP T--1 at time t 1. We use a pretrained ResNet [38] to extract visual feature maps for both o, and Og- Let ())) denote the visual feature map for Given the executed GRASP π-1 = (It-1. 6b-1), where _ represents the GRASP location and 6,-1 represents the end-effector rotation, we crop a visual feature segment ((({)))(TE-]] with predefined crop window size centered at 1T-1, and we consider (((t-1)(TE-1] as a template for the grasped object (Figure 3b). The cross- correlation between ((0:-1) [-:]] and Φ(0y) outputs similarity distribution showing the re- semblance between [[ and the local features at every placement in (00,) given by aimilarity = = )
PP:
Different from [37], we also apply cross-correlation between depth images og gepth and o depth, L Φ(0, ((oeeph * @ dept)) which outputs a pixel-wise distribution over the workspace indicating whether a pixel location is occupied by objects in the current scene or in the goal scene. The prediction for
depth By lowering the value for occupied pixels we avoid PLACE is given by: ofLLAEE st similarity yp, placing the grasped object on top of other objects or at goal positions of other objects. y pixel-wise prediction and each pixel represents potential placement for the grasped object; to model the end-effector rotation of PLACE, we rotate the current image o; to 16 different orientations as input and pick the one with the highest prediction value. The non-occupied location in ((0,))
with the highest cross-correlation value considered as the best PLACE T3 place for the grasped object. If match cannot be found in 0g. i.e. the similarity score is below a predefined threshold for all pixel locations, the object placed aside in the bin. The training loss for PLACE policy learning is cross-entropy. The ground-truth goal position and orientation of the grasped object are extracted

---

directly from the simulator. We generate the label y by assigning value to the pixel at the goal location of the grasped object; all other pixels are set O. The learning objective is maximize the visual feature extraction model's prediction accuracy given a goal image and a template. While we rotate the input image in 16 different directions to differentiate placing orientations, only assign value I for the one with correct goal orientation. The rearrangement loss is calculated as:
b
= ---i Vt
PLACE log PLACCE
+ (1 -
PLACK) log(1 - 26 (PLACR)
3.3 Graph-based Object Sequencing: Which Object is Next
Graph Generation We construct an accessibil- ity graph representing reachable traversal paths from the robot end-effector location to every ob- ject. Unlike [39] (which assumes a known geome- try for objects and uses the graph for target object retrieval tasks), we use UOIS-Net-3D [34] to pro- vide set of object segmentation masks from raw RGB-D images. We consider each segmented ob- ject vertex v E V in the scene graph and add Ur as the robot vertex. An edge e E E between a pair of vertices means collision-free end-effector path exists between them. The graph generation algorithm is shown in algorithm 1. Examples of generated scene graphs are in the supplement. The traversal path from the robot vertex shortest path from the robot base that object including objects blocking the straight line path.
Algorithm 1: ACC-GRAPH GENERATION
Input: camera observation O of scene.
Output: accessibilitty graph 9 = (V,E).
1 Ø. V.V. <- 2, V <- @
2 Get segmentation from UOIS-Ner-DD(O) Each segmented object maps to v y
4 Create robot vertex 6,, V [x.]
5 while iu E V and v of V do for every vi E V do for every v; V do linear distance path (v).25) is collision-free then
6
7
E 8 ( m3]]
10
V vu (2,
1 12 return = (V.E)
V' - V (2,)
object vertex in the generated scene graph captures the
Object Sequencing Let og and 0g denote the cur- rent and the goal scenes, and Gz, Gg denote the current and the goal scene graphs. We establish a list of sub-graphs of G. by individually remov- ing each vertex and its related edges. We cal- culate the similarity between each sub-graph and the goal graph through a pretrained SimGNN [40], previously shown to be an excellent approxima- tor (MSE < 1.18 x 10-3). The graph similarity corresponds to the Graph Edit Distance (GED) be-
Algorithm 2: OBJECT REARRANGE-
MENT SEQUENCING
Input: accessibility graphs of the current and the goal scene, G. = &V,,&) and Gy = (V., Eg),
Output: selected object v E G. for next rearrangement.
n t- Vt.size
Initialize an array sim[1, n| 4
for every U2 E V2 do
9 t- G. {er}
tween graphs G1 and G2 1 the number of edit oper- ations in the optimal alignment that transform G1 into G2. where an edit operation on graph an insertion or deletion of a vertex/edge or relabelling
sim[i] <- SiN.G 9,)
selected <- arg max sim[l...., n]
return {/[selected]
of a vertex (isomorphic graphs have GED 0). The removed vertex (object) from the highest similar- ity sub-graph is selected to be rearranged next. The robot thus chooses the object responsible for the largest difference between the current and goal scene graphs, to keep the number of task completion actions as low possible. We show the process of choosing next object to rearrange in algorithm 2.
Loss Calculation We use A calculate the ground-truth GED between graphs [41], since the scene graphs are small. To scale up to complex graphs in the future (where the ground-truth GED might be inaccessible or computationally expensive to obtain) we use SimGNN to approximate GED for all graphs instead of using the ground-truth GED. We transform the ground-truth GED between G2 and G2 to ground- truth similarity labels y in the range (0, 1] [40]:
GED G2) Norm Norm GED G2) (IG + /22))2
y Norm G.,,.
where |G| is the number of vertices in G. Let 8; denote the similarity prediction output between Gi and Gg from SimGNN and Ve, i = 1..., NN denote the ground- ground-truth -trutt similarity label. We use the cross-entropy loss for the graph- based object rearrangement sequencing:
N
L, =
Ly log 8 + (1 - y1) log(1 - 8:)].
5

---

After selecting an object for rearrangement, its placement is located as described in subsection 3.2.
4 Evaluation
4.1 Experimental Results in Simulation
We use a position controlled Franka Panda arm with a parallel-jaw gripper in Pybullet [42]. A simulated RealSense D415 RGB-D camera with resolution 640 x 480 is statically mounted. A side bin placed to hold objects removed from the workspace.
Method
Rotation
Swap
Clutter
Selectivity
Init. #obj.
Goal #obj.
Completion t
Position Error
NeRP [24]
x
x
X
3-8
3-8
94.56 + 0.73
1.90 1.30
IFOR [25]
x
X
1-9
1-9
81.80
2.70 2.30
x
X
X
3-7
3-7
96.67
1.67
1.29 - 00.11
/
X
X
3-7
3-7
90.00 + 3.00
2.99 2.37
x
X
V
3-7
1-5
97.33 + 0.67
1.41 2.70
Ours
V
3-7
1-5
97.00 + 1.00
1.81 2.66
X
16-20
5-10
85.67 + 2.33
1.64 0.44
16-20
5-10
82.33 + 2.67
1.22 0.93
Table 2: Task Completion (mean %) and Position Error (10-2m). Init./Goal #obj. is the number of objects in the initial/goal scene respectively. NeRP [24] and IFOR [25] are state-of-the-an models for image-based tabletop rearrangement. Statisties quoted here from their original paper (code- bases not publicly available). NeRP is restricted to translation. Since we handle both translation and rotation compare with IFOR. We achieve higher task completion (90%) than IFOR (81.8%) in the same setting (reduced clutter, no selectivity, but swaps may be needed). We handle signifi- cantly more complex cases than either previous model (e.g.. last row of the table shows concurrent challenges handled by our system: high clutter, selectivity, with swaps needed).
We conduct 6 sets of simulations (3 random seeds each, 100 episodes) with variations shown in Table 2. In each episode, we randomly pick 3 < N < 20 YCB objects [43] and select subset (or all N) to Setrrange..bjecte to be rearranged are placed at random goal positions and orientations and an RGB-D image is captured. This image is the goal specification. Next, we randomly reposition and rotate all these objects and add the remaining objects (those not designated for rearrangement) at randomly generated positions and orientations. The resulting scene is the initial state of the episode.
We add objects in testing that were not seen in training to show the system's adaptability to novelty. We differentiate task difficulty by measuring the degree of scene clutter, the degree of selectivity (how many of the objects are designated for rearrangement), and how many swap actions are needed. {II1 (111,81)... ..., (In, }n)} denote object positions. We define a clutter coefficient:
Let P
cPP) - log {
(z, - (14-2,124 2 (y: - 3. a, = kNN(Ny) d. = kNN(x),
in which kNN(y), kNN(x) estimates f. Vs through k-nearest neighbor regression every other object's position. We consider arrangements with c(P) 2 1.0 as 'cluttered' (examples in supplement Appendix A). In selective episodes, proper subset of initial objects is in the goal arrangement. Hence, the system must identify which objects need to be manipulated. Some episodes require swaps, where the goal positions of certain objects initially occupied by others. This requires the robot to first move the blocking object and then reposition the original object to be rearranged.
We evaluate our method with three metrics: (1) Task completion is the percentage of completion in all rearrangement episodes. We consider an episode to be complete when all target objects are placed within 5 em from their goal position (consistent with [25]) and all non-target objects are placed aside. (2) Position error is the average Euclidean distance between the desired arrangement and the final arrangement achieved. (3) Planning steps is the average number of actions the robot takes in each completed episode. It is a measure of the planning efficiency of the learned rearrangement policy.
Our system performs rearrangement in variety of settings, generalizing readily from 3-20 objects (Table 2, Figure 4). Task completion is calculated over test episodes; planning steps and posi- tion error only reported successful episodes. The task completion rate decreases as the task

---

agu Plarming y Sleps 3-7 Oljacia (hursClullar
Ave Plarning Staps, Olgecia (Cldller)
[c]
185 seleet enesome
Narber OleeSS
SLMEEE of Ofpects
(a) 3-7 objects
(b) 16-20 objects
Figure 4: Average Planning Steps Tasks with target selection require fewer planning steps, in- troducing swap actions in the task setting increases planning steps. The number of planning steps increases as the number of objects the scene grows.
setting becomes more difficult, the position error is stable across all task settings, which indicates accurate placement prediction from the sub-policy shown in Figure 3b. In selective episodes, system has a higher task completion rate than non-selective episodes when other task settings are the same. We ascribe this the graph-based object sequencing module (subsection 3.3) that prioritizes removing non-target objects over rearranging target objects, thus decreasing the clutter coefficient of the current scene and potentially improving the success
(see supplement Table 2 for details). The task completion rate decreases in situations with high clutter and swap actions. With increased clutter, it is more difficult find 'buffer' locations for objects whose goal positions are occupied by other objects or objects that are occupying others' goal locations.
NeRP [24] and IFOR [25] are state-of-the-art models for image-baset based tabletop rearrangement, Like IFOR, our method includes planar rotation alignment of objects (examples in supplement subsec- tion B.2) while NeRP only considers translations. Compared to IFOR achieve 8.2% higher task completion rate in the same task setting at a comparable rotation error (ours: 3989,, IFOR: 13.70%)
In Figure 4, we observe that when the task setting remains the same, the number of planning steps increases as number of objects increases. When target selection involved, the number of plan- ning steps decreases, as the object sequencing mechanism prioritizes removing non-target objects from the table, leaves a more sparse arrangement objects in the workspace, potentially reducing subsequent task difficulty. The introduction of swap actions, however, significantly increases the number of planning steps in each task completion. The swap action requires the robot to sample 'buffer' locations for objects whose goal position is occupied, place objects at "buffer' locations, remove the 'placeholder' objects at their goal positions and then reposition the objects at their goal locations. This process naturally adds more required actions towards task completion.
Two noteworthy recent rearrangement systems are TRLB [10] and ReorientBot [26]. Both rely on suction mechanisms to manipulate objects in clutter without the need to singulate them. TRLB relies on the initial and goal states being fully specified as object poses with focus on fast planning for rearrangement and ReorientBot relies the goal state being fully specified object poses. Our task sufficiently different (gripper instead of a suction mechanism, goal specified only by a single image) making direct comparison between work and these two systems infeasible.
4.2 Ablation Studies
Target Object Selection: In selective rearrangement, the objects in the goal scene (target objects) might be a subset those in the initial scene. We evaluate the significance using ResNet to obtain accurate visual feature cross-correlation and target object classification by testing 2 differ- ent encoder-decoder structured visual feature extractors, ResNet [38] and U-Net [44]. We measure the match success rate, average position prediction error and target object classification accuracy over 100 different initial and goal arrangements. The choice of visual feature extraction model is crucial because it directly affects the accuracy of target object identification and reposition. Experi- ments show that ResNet achieves match success rate of 93.33%, position error <2.04 cm and target classification accuracy of 98.58% with 1-20 objects (details in supplement subsection B.3).
7

---

Scene Graph
10
20
Graph-based Object Sequencing: To verify the importance of graph-based object sequencing to minimize the number of actions, we test two scene graph generation methods and measure their im- pact on the average number of planning steps. We also consider the situation when no sequenc- ing mechanism is used (no scene graph) and the robot picks the next object only based on PUSII and GRASP Q-value estimates. We generate the scene graph in two ways; a position-based approach which captures the basic spatial relationships among objects and an accessibility approach (subsec- tion 3.3 algorithm 1). We perform object sequenenn sequencing (algorithm 2) given the scene graph Gi and the goal scene graph Gg- Compared with no sequencing, using the accessibility graph decreases planning steps by 55.56% (10 objects) and by 43.71% (20 objects). Compared with the position- based approach, using the accessibility graph decreases planning steps by 21.72% (10 objects) and by 13.11 % (20 objects) thus confirming the efficacy of graph-based object sequencing. A detailed analysis is in the supplement subsection B.4.
N/A
35.13:3.55
45.22:t4.70
Position Accessibility
19.9444..93
29.29++.52
15.61+3.88
25.45++3.88
Table 3: Scene Graph Comparisons. Aver- age planning steps vs. # of objects in the initial scene. All scenarios have 10 target objects.
4.3 Demonstration on a Physical Robot
We test our system on a Panda robot arm with parallel- jaw gripper. and statiaall-moundee overlooking the tabletop (Figure 5). A bin next to the workspace holds the redundant (non-target) objects. Ob- jects included in the demonstration vary across experi- ments, including collection of 20 daily use objects (e.g. peanut butter jar, ketchup bottle). The robot demonstration generalizes to novel objects not available during training. We show zero-shot transfer from simulation to a real robot setting in the video.
RGB-D camera
Figure 5: Robot experiments.
5
Limitations
Our system has several limitations. (1) DOF rotation. We are limited planar object rotations. We do not currently handle 6 DOF object reorientation, and our system is poor at orienting rotation- ally symmetric natural objects (e.g., oranges). (2) Cluttered final state. Even though our method solves difficult rearrangement tasks with cluttered initial object arrangements, it struggles with sce- narios where the desired goal arrangement is cluttered. Unsurprisingly, this is significant challenge for other existing systems too with large number of objects it turns into difficult packing or stacking problem. (3) Segmenting objects. Our system object-centric - we use scene segmenta- tion to build scene graphs and sequence objects. Incorrect object segmentation produces inaccurate object sequencing and performance degradation. (4) Underlying motion planner limitations. In some experiments,
experienced difficulties with joint limits being reached when the initial grasp for an object turns out to not be feasible for object placement in the new location or when the robot carrying object collides with another object. We believe limitations 1,3 and 4 can be addressed respectively by expanding the action space in the action selection module, better/multiple cameras, and improved image segmentation techniques and vrajectory-awar obstacle-avoiding planners.
6 Conclusions
We proposed an effective image-based learned method for selective tabletop object rearrangement in clutter. Our simulated experiments provide evidence that the method works well in challenging settings which require the robot to begin with an intially cluttered scene, select only the objects that need be rearranged while discarding others, deal with cases where the target location for an object is already occupied- making the system the first of its kind to be able to address all these concurently. Ablation studies provide an analysis of system performance. We also demonstrate zero-shot transfer of our system to real robot and generalization to unseen objects.
8

---

References
[1] S. Katz. Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living. Journal of the American Ceriatric Society, 31(12):721-7, 1983.
[2] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik,
I. Mordatch, R. Mottaghi, et al. Rearrangement: A challenge for embodied AI. 2020.
[3] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 2021.
[4] A. Krontiris and K. E. Bekris. Dealing with difficult instances of object rearrangement. In Robotics: Science and Systems, 2015.
[5] E. King, M. Cognetti, and S. Srinivasa. Rearrangement planning using object-centric and robot-centric action spaces. In JEEE Intl. Conf. on Robotics and Automation, 2016.
[6] S. D. Han, N. M. Stiffler, A. Krontiris, K. E. Bekris, and J. Yu. Complexity results and fast methods for optimal tabletop rearrangement with overhand grasps. The International Journal of Robotics Research, 2018
[7] C. Garrett, T. Lozano-Péére Pérez, and L. P. Kaelbling. Pddistream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the Intl. Conf. on Automated Planning and Scheduling, 2020.
[8] S. H. Cheong, B. Y. Cho, J. Lee, C. Kim, and C. Nam. Where to relocate?: Object rearrange- ment inside cluttered and confined environments for robotic manipulation. In IEFE Indl. Conf. on Robotics and Aulomation, 2020.
[9] R. Wang, K. Gao, D. Nakhimovich, J. Yu, and K. E. Bekris. Uniform object rearrangement: From complete monotone primitives to efficient non-monotone informed search. In IEEE Inil. Conf. on Robotics and Automation, 2021.
[10] K. Gao, D. Lau, B. Huang, K. E. Bekris, and J. Yu. Fast high-quality tabletop rearrangement in bounded workspace. In IEEE Intl. Conf. on Robotics and Automation,
[1I] M. Stilman, J.-U. Schamburek, J. Kuffner, and T. Asfour. Manipulation planning among mov- able obstacles. In IEEE Intl. Conf. on Robotics and Automation, 2007.
[12] M. Toussaint. Logic-geometie programming: An optimization- based approach combined task and motion planning. In 24th International Joint Conference on Artificial Intelligence, 2015.
[13] I. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In IEEE Intl. Conf. on Robotics and Automation. IEFE, 2011.
[14] D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. In Robotics: Science and System, 2020.
[15] C. R. Garrett, C. Paxton, T. Lozano-Pérez, L. P. Kaelbling, and D. Fox. Online replanning in belief space for partially observable task and motion problems. In IEEE Intl. Conf. on Robotics and Automation, 2020.
[16] B. Ichter, P. Sermanet, and C. Lynch, Broadly local-policy trees for long-horizon task planning. arXiv preprint arXiv:2010.06491, 2020.
[17] Y. Labbé, Zagoruyko. I. Kalevatykh, I. Laptev, J. Carpentier, M. Aubry, and J. Sivic. Monte- carlo tree search efficient visually guided rearrangement planning. IEEE Robotics and Automation Letters, 2020.
[18] Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg. Learinng ambidextrous robot grasping policies. Science Robotics, 2019.
[19] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 2017.
[20] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In IEEE/RS! Intl. Conf.on Intelligent Robots and Systems, 2018.

---

[21] M. Danielczuk, A. Kurenkov, A. Balakrishna, M. Matl. D. Wang, R. Martín-Martin A. Garg, S. Savarese, and K. Goldberg. Mechanical search: Multi-step retrieval of target object oc- cluded by clutter. In IEEE Inil. Conf. on Robotics and Automation, 2019.
[22] A. Murali, A. Mousavian, C. Eppner. C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. In IEÉE Intl. Conf. on Robotics and Automation, 2020).
[23] B. Huang, S. D. Han, J. Yu, and A. Boularias. Visual foresight trees for object retrieval from clutter with nonprehensile rearrangement. IEEE Robotics and Automation Letters, 2022.
[24] A. H. Qureshi, A. Mousavian, C. Paxton, M. C. Yip. and D. Fox. Nerp: Neural rearrangement planning for unknown objects. In Proceedings of Roboties: Science and Systems, 2021.
[25] A. Goyal, A. Mousavian, C. Paxton, Y.-W Chao, B. Okorn, J. Deng, and D. Fox. Ifor: Iterative 2022.
flow minimization for robotic object rearrangement. In arXiv:2202.0077:
[26] K. Wada, S. James, and A. J. Davison. ReorientBot: Learning object reorientation for specific- posed placement. In IEEE Intl. Conf. on Robotics and Automation, 2022.
[27] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to Rearrange Deformable Cables, Fabrics, and Bags with coal-Conditioned Trans- porter Networks. In IEEE Intl. Conf. on Robotics and Automation, 2021.
[28] H. Ha and Song. Flinghot: cloth unfolding. In Conference on Robot Learning. 2022.
The unreasonable effectiveness of dynamic manipulation for
[29] M. Shridhar, I.. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, 2022.
[30] Ahn, C. Kim, and C. Nam. Coordination of two robotic manipulators for object retrieval in clutter. arXiv preprint arXiv:2 15220, 2021.
[31] H. Wu, J. Ye, X. Meng, C. Paxton, and G. Chirikjian. Transporters with visual foresight for solving unseen rearrangement tasks. arXiv preprint wxXv:2202.11076: 2022.
[32] C. Paxton, C. Xie, T. Hermans, and D. Fox. Predicting stable configurations for semantic placement of novel objects. In Conference on Robot Learning, 2022.
[33] B. Tang, M. Corsaro, G. Konidaris, S. Nikolaidis, and S. Tellex. Learning collaborative pushing and grasping policies in dense clutter. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6177-61 84. IERE, 2021.
[34] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics (T-RO), 2021.
[35] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In IEEE conference on computer vision and pattern recognition, 2017.
[36] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012.
[37] A. Zeng, P. Florence. J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning, 2020.
[38] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[39] C. Nam, J. Lee, S. Hun Cheong, B. Y. Cho, and C. Kim. Fast and resilient manipulation planning for target retrieval in clutter. In IEEE Intl. Conf. Robotics and Automation, 2020.
[40] Y. Bai, H. Ding, S. Bian, T. Chen, Y. Sun, and W. Wang. SimGNN: A neural network approach to fast graph similarity computation. In ACM Intl. Conf. on Web Search and Data Mining, 2019.
[41] K. Riesen, Emmenegger, and H. Bunke. A novel software toolkit for graph edit distance computation. In International Workshop on Graph-Based Representations in Pattern Recogni- tion. Springer. 2013.
[42] E. Coumans and Y. Bai, Pybullet, python module for physics simulation for games, robotics and machine learning, 2016-2021.
[43] B. Calli, A. Singh, Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Vale-cmu-berkelee dataset for robotic manipulation research. IJRR, 2017."
[44] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference on Medical image computing and computer- assisted intervention, 2015.
10


Reviews:
[{"review_id": "kLgD-w8bzdb", "reviewer": "Reviewer_W89x", "summary": "Selective Object Rearrangement in Clutter presented an image-based learned method for table-top object rearrangement in clutter using parallel jaw gripper. The method consists of three stages, including a graph-based object sequencing to select the next object to manipulate (using the current scene graph and goal scene graph's graph edit distance), feature based action selection to map image to robot actions (select between pushing or grasping), and a correspondence-based placement policy. Overall, authors claim that the method is the first to address concurrently object selection from initially cluttered scene, discarding the unrelated objects, and dealing with cases where goal location is occupied. Experiments show that the method is able to outperform SOTA methods in less restrictive settings, while also providing reasonable performance in more restrictive settings. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Real robot experiments: The authors demonstrated zero-shot transfer from simulation to real robot. Cool!\n* Interesting choice of actions: Push & Grasp, where grasp is preferred, push designed to singulate objects in clutter.\n* Experiments are well designed, including definition of a clutter coefficient. Quantitative results of experiments analyzed, and convincing justifications given for observed behavior.\n* Outperforms existing SOTA methods.\n\nWeaknesses\n* Pre-defined assumptions: Pre-defined window crop size for visually defining grasped object, would be better to see some ablations on generalization to out-of-distribution objects, would the assumption still hold?\n* Dependence on pre-trained methods: UOIS-NET-3D is used to provide set of object segmentation masks. How does the system react to errors in the predicted segmentation mask?\n* Missing experiment: It appears the NeRP baseline has adequate performance when only Swap is handled. It would be interesting to see how the presented method works when only swap is handled as well.\n* Can the authors comment on whether this is an online method, and what kind of computational resources is required to run the pipeline?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presented a novel comprehensive method tackling selective object rearrangement in clutter. Overall, the technical contributions are sound and clearly explained, with a full suite of experiments (simulation and real-robot). Through the reported results, it appears that the method outperforms existing baselines by a large margin, and is able to be scaled to real-world experiments zero-shot. There are a few points that I am hoping the authors can comment on / provide experiments for as outlined in the strengths & weaknesses section, including an evaluation of the method's dependence on pre-trained methods robustly performing, impact of pre-defined assumptions (i.e. fixed window crop size for template matching), and more careful comparison with NeRP.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See strengths and weaknesses section.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "a3PKdzlLePl", "reviewer": "Reviewer_Exub", "summary": "This work presents a novel method for tabletop object rearrangement amid clutter for cases where a subset of the objects may need to be removed from the scene. It works through sequencing the objects to manipulate through a measure of the graph edit distance from the current state to the goal state, where each object is manipulated by either a learned grasp or a learned push primitive (both implemented as fully convolutional Q-value maps), and where each grasped object is either moved away from the scene or placed in a location with the greatest feature correspondence to the goal scene. Extensive evaluation is done in simulation, and it is demonstrated that the learned policies can successfully transfer to a real robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Clearly written and largely easy to follow\n* The problem addressed is important to the field of robotics\n* The proposed solution is novel in several respects (in particular the scene-graph edit based object sequencing and the depth-image cross correlation aspect of the feature-based repositioning). \n* The experiments are detailed and convincingly demonstrate the approach works well, and the video clearly demonstrates the approach can be adapted to work on a real robot\n* The discussion of limitations is thorough \n\nWeaknesses:\n* Several of the figures (1, 3, 4) have images and/or text that are so small that even zooming in does not make it easy to make out the details in the PDF, and that would be hard to understand in a printed version.\n* I felt some of the text not covering the main novel ideas or results could have been cut or moved to allow for more results or analysis to be included in the main paper instead of the appendix. In particular, the stanard RL notation preliminaries are covered in section 3.1 and the summary of the Q-value map method (which as stated has a lot of overlap with [20]) to have been moved to the appendix or cut down. I feel including the planning step and ablation results in the main text would have been more informative.\n* While the citation of the relevant prior work is well done, there is very little summary of it. I feel that for readers less familiar with the cited works would hard a hard time understanding which are the key novel contributions of this work and which parts are largely similar to prior work (such as the Q-value map). \n* While the ablation results for the action sequencing is informative, I was left wondering how well a simple hand-coded heuristic (such as moving each non-target object to the bin first, and then repositioning each object by Q-value) might perform. In general, it is not intuitive why the first set of actions would not be just removing all non-target objects from the scene, followed by planning to reposition the remaining objects to the target.\n* The goal specification including an image of the final scene (needed for template matching) seems a bit limiting -- the method would be more powerful and widely applicable were it able to work with just a scene-graph based of the goal scene. \n* In some parts the paper can be a bit dense with extremely long paragraphs, and it seemed that it could have been edited a bit more to make reading easier.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While I feel this paper could still be improved in various ways, I still think the clear presentation of ideas, novel approach to an important problem, and impressive results strongly merit it being accepted.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See my summary of weaknesses - I feel the paper could still be improved with fairly substantial re-writing and iteration of the figures.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "NxWQjzYg84c", "reviewer": "Reviewer_Z5Z1", "summary": "This paper presents a system for image-based object rearrangement in cluttered tabletop environments where given a goal image, the robot plans a sequence of grasp and pushing actions to achieve the configuration corresponding to the goal image. The presented approach has 3 components: (1) an object sequencing method, which constructs a scene graph and encourages the robot to manipulate objects which  result in a graph closer in topology to the scene graph corresponding to the goal image, (2) an action selection method, which selects a grasping or pushing action on sequenced objects based on their likelihood of success, and (3) a placement module which places objects such that they match their pose in the goal image. The action selection method and placement module are very similar to ideas from prior work, but the object sequencing method appears to be relatively novel. The constructed system is also impressive: the videos demonstrate the efficacy of the approach on robot hardware.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The presented method is clearly described and for the most part all components of the system are easy to understand.\n(2) The resulting system is deployed on physical robotic hardware and videos suggest that the proposed method is able to successfully rearrange configurations with up to 15 objects. This is a very challenging task and the effort put into deploying this system in hardware experiments is much appreciated.\n(3) The graph based object sequencing method is very interesting and novel to the best of my knowledge. I appreciate the ablation in Section 4.2, which appears to demonstrate the utility of the approach as opposed to heuristics based only on grasp Q-values (such as in Mech Search by Danielczuk et al.)\n(4) I appreciate the extensive evaluation of the proposed method on a number of different numbers of initial objects and goal objects in simulation.\n\nWeaknesses:\n\n(1) The novelty of the work with respect to prior work needs to be made a bit more clear. The paper claims that this is the first paper that allows the robot to selectively rearrange objects from image observations in clutter. While the authors claim that prior TAMP approaches that incorporate learning-based vision models [14, 15, 16, 17] struggle to scale to adversarial environments, the reason for this should be expanded in more depth, and preferably, the authors should experimentally show that prior methods which seem in principle capable of addressing the same problem setting are intractable for the problem settings considered in this work.\n\n(2) The comparison to prior work in simulation experiments does not appear calibrated. The authors should implement prior methods in the same setup as their algorithm is evaluated in. Otherwise, the numbers in Table 2 do not seem to be a fair comparison, and it is hard to draw any meaningful conclusions about the relative performance of the presented method. It seems important to implement at least 2 of [14, 15, 16, 17, 24, 25] based on which the authors believe will be the strongest baselines in their experimental setup. \n\n(3) The discussion of the physical experiment results is insufficient in the main body of the paper. It is important the paper discusses quantitive results such as the number of trials, success rate, etc... of these experiments for the reader to be able to evaluate the significance of the results. It is also preferable to compare to prior methods in physical experiments, but it is OK to omit this if sufficient comparison is at least provided in simulation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an impressive robotic system for object rearrangement with convincing evaluation on physical hardware. However, the experimental evaluation with respective to prior work and reporting of results in physical experiments need substantial improvement to fully evaluate the contributions of the paper. I am happy to update my recommendation if these issues are sufficiently addressed, but I am currently learning towards rejection for these reasons.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Major issues:\n\n1) Please clearly describe in more detail why prior TAMP methods with learning-based vision models [14, 15, 16, 17] would not scale to the scenarios evaluated in this paper and ideally provide experimental evidence for this argument. Without this, it is difficult to understand why the presented algorithm is needed with respect to prior existing work.\n\n2) It would be good to implement some prior algorithms as discussed in the weaknesses above in simulation to provide a calibrated comparison to prior work. Currently, there does not seem to be concrete experimental evidence that the proposed approach provides significant benefit over prior approaches.\n\n3) Please report full quantitive details of physical experiments as indicated in the weaknesses section above.\n\nMore minor issues, but which would benefit from clarification:\n\n4) Why are pushes considered successful based on pixel change in the image? If the purpose of a push is to increase grasp access, shouldn't pushes be considered successful based on the degree to which the increase grasp Q-values?\n\n5) The grasp success criteria is a bit strange. Just because the antipodal distance between parallel jaws is larger than a predefined threshold doesn't seem to necessarily suggest that the executed grasp will be stable when the object is lifted. More clarification on this point would be much appreciated.\n\n6) The learning objective in Section 3.2 is somewhat confusing. Specifically, I am confused by how $\\varphi_t^{\\text{PLACE}}$ and $\\phi_t^{\\text{PLACE}}$ are related to each other. Given $\\varphi_t^{\\text{PLACE}}$ (which requires no learning to compute since a freezed Resnet backbone is used to featurize images), it seems like all you would need to do is place the object at the location where $\\varphi_t^{\\text{PLACE}}$ is the highest.\n\n7) I am also confused by the learning objective in Section 3.3. The authors initially state that a pre-trained SimGNN is used to determine graph similarity. Given this, I am confused why any learning is necessary at all (and am thus confused by the loss definition $\\mathcal{L}_s$: given this graph similarity measure, you have all the information you need to determine which object to sequence next.\n\n"}}, {"review_id": "0L6dyR97uhf", "reviewer": "Reviewer_C9hA", "summary": "The paper proposes an image-based learning framework for object rearrangement. In particular, three major challenges are addressed: (1) Objects are closely located with each other. (2) The planner can determine which objects are subject to rearrangement. (3) The planner can resolve a situation when other objects occupy the goal locations of target objects. The proposed method is evaluated on simulations and real-world experiments and shows its promising performance in terms of success rate, accuracy, and efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed framework is novel, and it is convincing why this learning approach should work.\n- The results from both simulation and experiments look promising.\n\nWeaknesses (details are included in the Summary Of Recommendation and Issues sections):\n- Contributions seem less significant as the justification of the fundamental limitations of the previous methods is not well addressed.\n- Object sequencing has some critical limitations, which may fail in practice in environments containing many objects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well-written overall, and the presentation is clear. Although the evaluation results are promising, I am concerned that the contributions may not be significant enough compared to existing methods. The authors state that the previous methods cannot handle the challenges considered in this work as those papers do not treat these challenges. However, that does not mean those methods fail completely with clutter and swap. Since the authors did not evaluate those methods themselves, a more substantial justification is needed by providing insights into the fundamental limitations of those methods. A novel extension to the previous methods is selectivity, which I think is the weak component in the framework, but the same principle also seems to work for them. Moreover, the current setting of the problem having a bin that is not a part of the workspace makes selectivity not a very challenging task.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Major comments:\n- Contributions compared to existing methods: As discussed above, please explain in the introduction or related work section why previous methods cannot handle clutter and swap. Unless there are clear reasons, the capabilities of existing methods made in Tables 1 and 2 do not look fair. I believe this would enhance the contributions of this work considerably. \n- Object sequencing 1: An exponential number of subgraphs can be generated, prohibiting working with a large number of objects in practice. How long does it take to solve 20 objects in the evaluation? \n- Object sequencing 2: As pointed out in the limitations section, a straight line path is very naive. When many objects are located closely, motion planning must be used to avoid collisions. In this regard, the high task completion rates in Table 2 are surprising to me. Incorporating motion planning in the learning framework is not straightforward as it requires the notion of configuration space. \n- Object sequencing 3: It is unclear how object sequencing handles swaps in Section 3.3. Please explain this. Also, in Figure 1, it seems to clear non-target objects first. Is it always expected to occur?\n- POMDP in Section 3.1: It is confusing how solving a POMDP is related to the framework. Is it referring to deep Q-learning, or are they two separate things?\n- Section 3.2: The proposed approach does not seem to find a placement right behind an object as that area will be considered as occlusions. If it is incorrect, please explain this.\n- In the evaluation, it is not intuitive what makes the proposed method handle clutter. Is it because of feature-based action selection that makes pixel-wise prediction?\n\nMinor comments:\n- Do NeRP and IFOR consider the same assumptions and inputs (e.g. both initial and goal images)? \n- I was not able to understand what the authors mean by rotating an image in Section 3.1. How do you maintain the resolution of a rotated image before feeding it into FCN?\n- In Section 2, \"changes in environment observability increase the difficulty of back-tracking and replanning\": I don't understand the relationship between changes in environment observability and backtracking and replanning. Please clarify this. \n- In Section 3.3, what is \"relabeling of a vertex\" for?\n- In Section 3.3, it is hard to understand how A* can be used to calculate the GED between graphs.\n- In Figure 3, images are too dark to see. \n- In Figure 1, there are typos. (1-10) -> (1-11) and (11-15) -> (12-15).\n- Typo in Line 223: \"needs identify\" -> \"needs to identify\"."}}]

Author rebuttal:

Decision: Accept (poster)