Predicting Object Interactions with Behavior Primitives: An Application in Stowing Tasks
Authors: Haonan Chen
Yilong Niu
Kaiwen Hong
Shuijing Liu
Yixuan Wang
Yunzhu Li
Katherine Rose Driggs-Campbell
Venue: CoRL 2023 Oral (2023)
Keywords: Robotic Manipulation
Model Learning
Graph-Based Neural Dynamics
Multi-Object Interactions

Abstract: Stowing, the task of placing objects in cluttered shelves or bins, is a common task in warehouse and manufacturing operations. However, this task is still predominantly carried out by human workers as stowing is challenging to automate due to the complex multi-object interactions and long-horizon nature of the task. Previous works typically involve extensive data collection and costly human labeling of semantic priors across diverse object categories. This paper presents a method to learn a generalizable robot stowing policy from predictive model of object interactions and a single demonstration with behavior primitives.  We propose a novel framework that utilizes Graph Neural Networks (GNNs) to predict object interactions within the parameter space of behavioral primitives. We further employ primitive-augmented trajectory optimization to search the parameters of a predefined library of heterogeneous behavioral primitives to instantiate the control action. Our framework enables robots to proficiently execute long-horizon stowing tasks with a few keyframes (3-4) from a single demonstration. Despite being solely trained in a simulation, our framework demonstrates remarkable generalization capabilities. It efficiently adapts to a broad spectrum of real-world conditions, including various shelf widths, fluctuating quantities of objects, and objects with diverse attributes such as sizes and shapes.

Predicting Object Interactions with Behavior Primitives: An Application in Stowing Tasks
Haonan Chen, Yilong Niu, Kaiwen Hong, Shuijing Liu, Yixuan Wang, Yunzhu Li, Katherine Driggs-Cammbel University of Illinois, Urbana-Chin Saana-Chmpaainn
[haonan2, yilongn2, kaiwen2, sliu105, yixuan22, yunzhuli, krde)@illinois.edu
Abstract: Stowing, the task of placing objects in cluttered shelves bins, is a common task in warehouse and manufacturing operations. However, this task is still predominantly carried out by human workers as stowing is challenging to au- tomate due to the complex multi-object interactions and long-horizon nature the task. Previous works typically involve extensive data collection and costly human labeling of semantic priors across diverse object categories. This paper presents method to learn generalizable robot stowing policy from predictive model of object interactions and single demonstration with behavior primitives. We propose novel framework that utilizes Graph Neural Networks to predict ob- ject interactions within the parameter space of behavioral primitives. We further employ rrimitiv-augmented trajectory optimization to search the parameters of a predefined library of heterogeneous behavioral primitives to instantiate the control action. Our framework enables robots proficiently execute long-horizon stow- ing tasks with few keyframes (3-4) from single demonstration. Despite being solely trained in simulation, tion capabilities. It efficiently adapts to broad spectrum of real-world conditions, including various shelf widths, fluctuating quantities of objects, and objects with diverse attributes such as sizes and shapes.
framework demonstrates remarkable generaliza-
Keywords: Robotic Manipulation, Model Learning, Graph-Based Neural Dy- namics, Multi-Object Interactions
Introduction
Stowing, defined relocating an object from table to cluttered shelf, is one of the dominating warehouse activities. In stowing, an agent is required to pick an object from table. The agent must then actively create free space within the shelf before inserting the object from the table. A suc- cessful stow execution is characterized by the placement of all objects with poses some predefined thresholds. While stowing can be performed effortlessly by humans, it remains challenging when automated with robots. The difficulty stems from the long-horizon nature, multi-object interactions. and the variety of objects and configurations involved in stowing tasks.
The challenge of long-horizon stowing task is not only due to the nature and variety objects in- volved but also due to several inherent constraints in existing methods. First, the nature these tasks requires determining the characteristics of contacts and frictions, which is task that presents considerable difficulties. Conventional first-order models fall short in capturing the physical effects, and identifying the specific parameters of contact and friction is challenging [1]. Thus, designing controller for such tasks becomes a tedious and laborious process. Second, the existing methodolo- gies, including manually pre-programmed systems and recent advancements in category-level object manipulation, exhibit notable limitations. Classical pre-programmed systems struggle with adapt- ability, unable to efficiently handle variations introduced by different arrangements of objects on the
"Equal contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

shelf. Meanwhile, recent strategies for category-level object manipulation are curbed by the need for expensive data collection and human labelling, thus failing to provide a scalable solution [2]. Ad- ditionally, pure learning-base drawbacks in terms of extensive training time and poor data efficiency [3, 4], making learning from scratch on real robots impractical for long- monghoorionn tasks.
methods, such as Deep Reinforcement Learning (DRL), also present
To address these challenges, we propose framework that uses Graph Neural Networks (GNNs) to predict object interactions within the parameter space of behavior primitives. When trained with various situations in the simulator, GNNs can learn to model the forward dynamics associated with interactions of rigid objects. Instead of explicitly determining contacts and frictions, our GNN framework designed to learn these underlying interactions during the training process. Thus, we eliminate the need for explicit detection and intricate calculations related to contacts and forces. Our framework also applies primitive-augmente trajectory optimization to search the parameters of predefined library of heterogeneous skills or behavior primitives. The incorporation of behavior primitives enables our policy to handle tasks of significant complexity with improved efficiency.
We make three key contributions: (1) We in- troduce novel model-based imitation learning framework to learn from minimal demonstra- tions, which enables robots to acquire complex skills. (2) We create comprehensive stow- ing benchmark for
Transport
manipulations, which is highly prevalent in various both the industrial and household applications. (3) We demonstrate the effectiveness and generaliza- tion of framework across a wide range of real-world experiments in stowing tasks.
Push
Figure I: The robot places object into cluttered
shelf by exploiting object interactions. It uses the grasped object to manipulate other objects within the through pushing and sliding actions and finally places all the objects in the appropriate location.
2
Related Works
One-shot Imitation Learning in Manipulation: Recent advancements in imitation learning aim to solve unseen tasks with minimal demonstrations [5, 6, 7, 8, 9]. Typical approaches use two-phase pipeline: meta-learning phase to train a general policy from numerous tasks and fine-tuning phase to optimize the policy for specific task using single demonstration [9]. Certain replay- based methods employ strategy of estimating a 'bottleneck pose', splitting the trajectories into two segments, and then replaying the demonstration trajectory from the bottleneck [6]. Other tech- niques emphasize learning object representations and identifying correspondence between specific objects [5, 10] or objects within the same category [8]. However, these methods primarily handle relatively short-horizon tasks and face difficulties in modeling object dynamics.
Model Learning in Robotic Manipulation: Dynamic models have emerged as crucial com- ponents in robotic systems, with the capability to predict future world states based on current states and actions [11, 12, 13, 14, 15, 16). These models can handle a variety of input repre- sentations, ranging from images and latent vectors to point sets and graph representations. No- tably, graph representations have shown an exceptional ability to capture interactions among ob- jects [17, 18, 19, 20, 21, 22, 23]. The ability model interactions between various objects and the robot's end effector is pivotal for our research, leading us to use a graph to model interactions between objects in our system. Tekden et al. [24, 25] introduce a graph-based modeling approach that places emphasis on object-level representation, leveraging GNNs to capture interactions among multiple objects. Similarly, RD-GNN [26] uses an MLP for action encoding and treats each object as a unique node in the graph, concentrating on inter-object relations. While both approaches provide broad perspectives on object interactions, our framework diverges by representing robot movements as point clouds and objects with multiple particles. This approach offers more granular under- standing of actions and interactions, enhancing the accuracy of object movement predictions.

---

Skilp-1
Skilk
Actio
Subgral
GNN
#po1
CNN
arg ((era)
MSE
Samplei
Statt Fz
Stor 96-3
(9) Dyunnics Training
(b) Policy Exceution
Figure 2: Overview of the proposed framework. (a) A particle-based representation characterizes the object state. The ohject state's predicted outcome following the executed robot actions computed alongside the ground truth object state using the MSE loss function train GNN. (b) For each skill, we apply random shooting to sample parameters within the action parameter space, utilizing the GNN predict object movement. We then select the action that brings closest to the desired state. Each skill executed sequence.
Long-Horizon Planning in Robotic Manipulation: Addressing long-horizon manipulation prob- lems presents considerable complexity. Hierarchical reinforcement learning (HRL) approaches ad- dress this issue by using a high-level policy to select optimal subtasks and low-level policies to execute them [27]. However, these methods face the challenges of the sim2real gap and the diffi- culty of real-world data collection, hampering their real-world transferability [28]. An alternative approach by Di Palo et al. [29] augments replay-based methodologies by integrating primitive skills for routine long-horizon tasks, though their task configurations lack versatillity Integrated task and motion planning (ITMP) combines discrete and continuous planning[30), blending high-level sym- bolic with low-level geometric reasoning for extended robotic action reasoning. Recent efforts by Lin et al. [31] have explored sequential manipulation of deformable objects through differential simulator and trajectory optimization. However, this work is only validated in simulation, and real- world deployment non-trivial due to the difficulty of obtaining gradients of state changes. To address this, we propose to apply trajectory optimization to GNN-based forward dynamics predic- tion modules, incorporating heterogeneous behavior primitives.
Approach
In our proposed system, a GNN first predicts system dynamics. Then, a primitive-augmentie tra- jectory optimization method achieves subgoals from a single demonstration, which is shown in Figure 2. Initially, the object state is represented as particles. We then train the GNN with the MSE loss between the predicted outcome following robot actions and the ground truth state. We use the random shooting explore the action parameter space and use the GNN's predicted results to se- lect the optimal action that aligns most closely with our desired state. The skills are executed in sequential manner, guiding our system to accomplish its tasks efficiently.
3.1 Learning Forward Dynamics via GNN
We define a dynamics function that describes a rigid-body system state S with M objects and N particles. We model each rigid object as a collection of uniformly distributed particles, offering a representation that is flexible and adaptable variations object shape, size, and geometry. The dynamics function expressed $ S x A + T. A represents the skill type and its associated parameters, and T denotes M rigid transformations containing the translation and rotation for each dynamic object. The future state of an object can be determined by applying sequence of these rigid transformations. We represent each rigid objects as uniformly distributed particles as such representations are versatile to object shape, object size, and object geometry.
Graph Construction:
Each object in the system is represented by its respective particles. We define the graph state s; Ee), where the graph's 's vertices O represent object's particles, and edges E, denotes the relations between particles. Each vertex 0,, (Fa,e C.a) consists of the particle's position Fy
3

---

and object's attributes Ci. including its dynamism (dynamic or static), the object it belongs to, its offset from the rigid body's center, and gravity. Edges & are formed when the distance between two vertices is less than a specified threshold. In our work, the relations are characterized by the physical interactions and geometric proximity between the particles signifying objects. Specifically, we introduce the following relations to adeptly encapsulate the complex dynamics in multi-object interactions: (1) Intra-object relations: Between different particles within the same object across different objects. (2) Gripper-to-objeet relations: Between particles from the objects and the robot's gripper. The edge relations are represented as ex = (ix. jk, Ca), in which ik, Jas denote the indices of the receiver and sender particles, respectively. The edge index is denoted by k, and nature of the relationship such as intra-object relation, or gripper-to-objer relation) is denoted by ck- Since our focus is predicting the motions of dynamic objects, we restrict node building to vertices associated with these dynamic objects. However, effectively model the interactions between dynamic and static objects (i.e, shelf and table), we choose to incorporate the particles of static objects during the construction of edges.
Message Passing: The features of all vertices and edges are processed through encoder multi- layer perceptrons (MLPs), resulting in latent vertex representations denoted as hę and latent edge representations represented as k.g. At each time step, we apply the following message function to the vertex feature and edge feature for several iterations to handle signal propagation.
E - ( 1 nD)) 1. + P Σ N
(1)
where the message passing functions pT and p² are MLPs. Subsequently, we apply an MLP decoder to the vertex feature processor's final layer output. The rigid transformation for each individual object is determined by calculating the mean of the decoder outputs corresponding to that object.
The control action, represented by the gripper's planned posi- tion and motion, is defined by partieles o,,t (Ii,, Vi,t, G.,). where Ii, denotes the current position of gripper, U1,t denotes the planned motion of the gripper. The particles associated with the gripper are subsequently encoded by the encoder. Additionally, we predict the future positions of the grip- per. The discrepancy between these predicted positions and the actual achieved positions serves as an auxiliary loss, which helps GNNs better understand the inherent physical laws and constraints.
Representing Action as Particles:
3.2 Control with the Learned Dynamics
In this section, we discuss the design of behavior primitives, and trajectory optimization algorithms used to generate parameters for different skills.
Behavior Primitives: We introduce the behavior primitives as higher level of temporal abstrac- tion to facilitate efficient planning. The behavior primitives simplify the task space by generating key poses for the system, which subsequently executes actions using Operational Space Control (OSC) [32] as lower-level controller. The GNN is used only to predict the system's state at the key poses of behavior primitives, which significantly reduces the number of forward passes and cu- mulative prediction error over time. Behavior primitives function serve as building blocks and can be easily extended for various manipulation tasks. We further specify a maximum execution time Tai for each behavior primitive. Our system integrates a collection of three primitives, encompass- ing both prehensile and non- non-prehensile motions. The primitives and their associated parameters are described as follows: (1) Sweeping: The robot sweeps objects the shelf using its end-effector, aiming to stand them upright. Sweeping is parameterized by the starting offset y in the shelf direc- tion, sweeping height h, sliding distance d, and the angle of gripper rotation 0 during the sweep. (2) Pushing: Pushing involves the robot nudging the object to establish potential grasp poses. The starting push position (x, y) and the distance of the push d are the parameters for pushing. (3) Transporting:
The robot picks up the object from the table, places it the shelf, and, necessary. adjusts its position through sliding. This skill is defined by parameters such as the starting off'set in the shelf direction y, the height of insertion h, the sliding distance d, and the gripper rotation angle 0 during the insertion process.

---

Goal-conditioned Trajectory Optimization: We optimize the parameters of given skill by min- imizing the mean square error (MSE) loss between the predicted particle positions and their corre- sponding positions as demonstrated. Keyframes collected during demonstrations can be denoted by g. We search for the skill parameter dp that minimizes the cost function representing the MSE between the predicted and target positions of the object particles. Mathematically, this optimization problem is represented as ap arg minay J(sr. .9). The resulting low-level control actions generated from the the skill which is parameterized by the skill parameters dg. Our dynamics net- work & makes forward predictions of the future state of the system after the execution of each skill. We employ trajectory optimization to find the skill parameters that yield the lowest cost.
Experiment Setup
Our experimental setup consists of both simulated and real-world environments. The simulated environment is built using Robosuite [33] and the MuJoCo physics engine [34], operating with 7-DOF Kinova Gen3 Robot. The real-world counterpart consists the same Kinova Gen3 Robot and a Robotiq 2F-85 gripper.
Data collection: We collect a dataset 300 episodes of action execution for each behavior primitive, where the actions are ex- ecuted based on randomly sampled skill parameters. For each episode, the robot randomly sampled parameters within its pa- rameter space and executed the skill in the simulator. We gather the key poses for each skill, subsequently training GNN to take the state at these key poses and the robot action and predict the future state at the subsequent key poses.
Camera
Shelf filled with various objocts
Target object grasping and inserting
Simulated environment: In our simulation, the shelf width is initialized to randomly vary within the range from 0.18 to 0.35 meters. We also randomly select the number of objects placed on the shelf, varying between two and four. The properties of each object, including size and density, were randomly gener- ated. All the objects are created with box shape. We use a SpaceMouse to teleoperate the robot to complete the task, and the ending poses of each skill were collected. These poses are then used as subgoals for each skill during execution.
Robot setup
(h) Objects used in this work
Figure 3: The experimental setup.
Real-world environment: Figure 3 illustrates our real-world environment setup. We use OAK-D Pro cameras, with a top-down camera to estimate the pose of the object for grasping and inserting, and a side-view camera to estimate the object's pose in the shelf. Objects placed on the table are always oriented perpendicular the table edge and positioned adjacent it. Shelf sizes of 0.18m and 0.35m tested, and objects in the shelf are placed randomized positions and orientations. A point cloud representation of each object created, including their sizes, positions, and orientations as the state representation.
We evaluated our dynamics model and manipulation outcomes in simulation based on the final prediction error, applying metrics such as Mean Squared Error (MSE), Earth Mover's Distance (EMD) [35], and Chamfer distance (CD) [36]. In real-world scenarios, success rates were computed for each setup. A success was defined as all boxes ending up within the shelf with their orientations falling within predefined threshold f.
Evaluation metrics:
Experimental Results
In this section, we evaluate our forward dynamics prediction model in both simulated and real-worke settings. We use a diverse range of objects and shelf dimensions to provide broad and challenging test. The results highlight our framework's 's potential for zero-shot sim2real transfer, demonstrating its applicability in handling real-world conditions without the necessity for prior real-world data.
5

---

Table 1: Dynamics Model Prediction Quantitative Results and Ablations. Our model consistently outper- forms RoboCraft in complex primitives such as 'Sweep" and 'Transport', evident in the lower MSE, EMD. and CD values. The Object-Level representation underperforms due to its lack of dense object information. In the simpler 'uus'' primitive, our model retains comparable MSE, EMD, and CD values within one standard deviation, indicating robust performance even in simpler scenarios.
Primitive
Method
MSE (mm)
EMD (mm) 7
CD (mm) &
Object-Level Repr
3.676 (+: 1.597)
75.015 (+ 14.452)
92,914 (+ 17.206)
Sweep
RoboCraft [14]
0.351 ( 0.222)
24.510
(#: 6,434)
33,277 (# 5.064)
Ours
0.287 (+ 0.185)
21.792
(# 5.533)
30.017 (# 3.259)
Object-Level Repr
3,765 (#: 0.76)
75.975 (# 10.927)
95,925 (# 14.802)
Push
RoboCraft
0.216 (+ 0.148)
12.509 (+: 3.494)
15.43 (+ 2.902)
Ours
0.292 (+ 0.179)
14.569 3.752)
15.046 (+ 3.085)
Object-Level Repr RoboCraft
5.861 (+ 3.106)
91.913 (+ 20.552)
113.263 (+ 23.168)
Transport
1.091 (+ 0.512)
42.162 + 10.317)
55.615 (= 9.997)
Ours
0.666 (+: 0.41)
31.232 (+ 9.108)
38.068 (#: 6.605)
Table 2: Quantitative Results of Control in Simulation. Our method consistently outperforms SAC, PPO, parameterized PPO. and heuristic-drive control, exhibiting markedly lower execution errors, illustrating the critical role of incorporating model for long-horizon stowing tasks.
Method
MSE (mm) 1
EMD (mm)
CD (mm)
SAC
265,411
465.639
368.571
PPO
87.479
266.554
173.522
Parameterized PPO
22.925
120.736
62.182
Heuristic
34.861
140.196
194.123
Ours
0.905
29.697
39.914
5.1 Dynamics Model Learning and Ablations
We trained our dynamics model using MSE loss. As part of our ablation study, we incorporated version of the GNN presented in [14], which we will refer to as "RoboCraft". It's important to note that this model does not encompass dynamic-static object interactions, nor does it utilize the auxiliary loss derived from gripper movements. Additionally, we introduced a baseline, "Object- Level Repr", using object-level representation by using a single GNN node to symbolize objects.
The quantitative results from the model learning are shown in Table The relatively small predic- tion errors suggest that our model is able accurately predict the interactions involved in the task; these include object/object interactions, environment/object interactions, and robot gripper/object interactions. The results further demonstrate that the model can effectively learn the rigid motions of the objects, resulting in only minor errors.
Compared to RoboCraft, the introduction of gripper movement information and dynamic-static ob- ject edge information improves the prediction accuracy of our GNN model, particularly in complex behavior primitives such as sweeping and transporting. Even in simpler behavior primitives like pushing, our GNN maintains a performance comparable to the RoboCraft, with the prediction er- ror remaining within one standard deviation, highlighting its consistent effectiveness. While the less specialized RoboCraft performs adequately in straightforward pushing skill, it encounters difficulties in more dynamic situations. In contrast, our model's 's advanced complexity and adaptability prove to be particularly advantageous in scenarios characterized by intricate dynamies, such as collisions and bounces between objects or with the environment.
5.2 Manipulation Results
Results in Simulation: We collect six demonstrations in the simulation. The keyframes from these demonstrations serve as the goal state for trajectory optimization, implemented with Random Shooting (RS). In our analysis, we evaluated Random Shooting, the Cross-Entrope Method, and Gradient Descent- - all of which exhibited similar performance. We present the results obtained from RS and denote them as "Ours" in the subsequent discussion. We use Proximal Policy Optimization (PPO) [37] and Soft Actor-Criti Critic (SAC) [38], both state-of-the-an model-free RL algorithms, as baselines. The choice of these algorithms aims to highlight the necessity of model for long-

---

(8) Nominal sctup
(b) With tiny object
(c) With heavy and trapezoid objocts
(d) With bottle grasped
(e) With slippery roller
Small shelf setup
Figure 4: Different Setups Used in Real-Worke Manipulation Experiments. Six setups represent wide range of conditions including different combinations of ohjects, shelf sizes, object dimensions, and shapes.
horizon stowing tasks. The simulation results are presented in Table 2. Each method's performance assessed using MSE, EMD, and CD as metrics. PPO and SAC utilize the negative MSE between the current and goal object states the reward function. In comparison to these methods, RS consistently yields lower execution errors across all metrics, which indicates its superior capability in minimizing the gap between actual and desired states. Model-free RL methods such as PPO and SAC struggle to perform effectively due to the large exploration space and the long-horizon nature of the task. Their effectiveness is further hampered by their lack of knowledge regarding the model of the environment. We also benchmark against version PPO with a parameterized action space based on behavior primitives and heuristic-driven approach devoid of learned dynamics. Our method outperforms both, demonstrating considerable performance advantage.
Results in the Real World: Our framework is tested in six different real-world setups, with each setup executed for ten test trails. We manually randomize the initial orientations and positions the objects within the shelf each trail. The object poses in these scenarios are identified using Scale- Invariant Feature Transform (SIFT) on the captured images. The various setups represent broad spectrum of conditions including different object combinations, shelf sizes, object dimensions, and shapes. This wide range of conditions is depicted in Figure 4.
In our experiments, we implement two distinct skill combinations for each setup: 3-skill set compris- ing sweeping, pushing, and transport- ing, and 2-skill set, which only in- cluded pushing and transporting. The term "heuristic" refers to process where humans fine-tune a relatively small parameter space and assign the tuned parameters the skills. The heuristic-basee approach likewise conducted utilizing all three skills: sweeping, pushing, and transporting.
Table 3: Real-World Success Rates. Performance evaluation six different setups using: our proposed method with two distinct skill sets skills and skills), RoboCraft learned dynamics, and heuristic-baat approach without dynamics prediction.
Success 1
Setups
Heuristic
RoboCraft
skills
skills
(a)
1/10
4/10
10/10
10/10
(b)
3/10
7/10
9/10
9/10
(c)
3/10
4/10
9/10
9/10
(d)
1/10
6/10
10/10
10/10
(e)
2/10
7/10
10/10
10/10
(f)
1/10
5/10
9/10
9/10
Average
18%
48%
95%
95%
Figure presents the success rates of the different control strategies. Our 3-skill method significantly outperforms the heuristic-basee approach with success rate of 95%, indicating the effective han- dling of varied setups by our dynamics prediction module. Interestingly, the 2-skill set also achieves the same 95% success rate, indicating that the robot's ability to understand the interactions between the gripper-held object and the objects within the shelf enables it determine the optimal position for insertion and placement. These high success rates demonstrate the effectiveness of our method.

---

Target
Slid out
Heuristic
Ours
(a) Sweeping
Heuristic
Slid out
Ours
(b) Transporting
Figure 5: Comparison of heuristic- method and our approach during the execution of sweep- ing skill transporting skill. method anticipates future states and arranges objects into upright posi- tions within the shelf, unlike the heuristic- based method which pushes objects of the shelf.
In contrast, the heuristic-basee strategy yielded an average success rate of only 18%. Despite be- ing trained solely with box-shaped objects, method generalizes effectively out-of-distribution objects, showing its versatility in variety of real-world conditions.
Figure 5 provides a qualitative comparison of the sweeping skill execution between the heuristic- based method and our approach. Our method, equipped with the ability to anticipate future states based on specific robot actions, is capable of sweeping and transporting objects into upright positions within the shelf. In contrast, the heuristic-based method tends to push objects out of the shelf.
6 Conclusion
In this work, we focus on stowing tasks wherein robot must manipulate a large, ungraspable flat object and subsequently stow within a cluttered shelf. The robot's task involves creating sufficient space within the cluttered shelf to accommodate the object appropriately. We introduce system that utilizes behavior primitives combination with forward dynamics prediction to achieve the task. We discuss the design choices for system and demonstrate its effectiveness in real robot scenarios. Moreover, show the system's ability generalize various stowing conditions.
Our work opens several potential avenues for future research. One promising direction involves developing the ability to composite skills and further reduce the sub-goals presented the demon- strations. Another that the design and definition of the behavior primitives library need additional exploration and research, which can enhance the adaptability and versatility of robotics systems in performing complex manipulation tasks.
Limitations: Our system currently has a few limitations. Firstly, it relies on manual human la- beling of ordered keyframes from demonstrations, which could potentially restrict scalability and deployment in larger and more complex scenarios. Secondly, we use box-shaped point clouds to represent objects during training and inference. This simplistic representation may not accurately reflect the geometrical properties of objects, especially in scenarios involving more complex inter- actions and contacts. Addressing these limitations, particularly improving object representation, presents promising direction for future research.

---

Acknowledgments
We thank Haochen Shi's tireless assistance with the GNN implementation, as well Neeloy Chakraborty, Peter Du, Pulkit Katdare, Ye-Ji Mun, and Zhe Huang for their insightful feedback and suggestions.
This work was supported by ZJU-UIUC Joint Research Center Project No. DREMES 202003, funded by Zhejiang University.
References
[1] R. Hogan and A. Rodriguez. Feedback control of the pusher-slider system: story of hybrid and underactuated contact dynamics. Workshop on the Algorithmic Foundations of Robotics, 2016.
[2] L. Manuelli, W. Gao, P.R. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In International Symposium of Robotics Research, 2019. URL https://apps http:////app.mmmticccccoooooooo
semanticscholat woacooorrrrrrooooooooooooo Terg/CorpusID-. 80628296.
[3] Luo, O. Sushkov, R. Pevceviciute, W. Lian, C. Su, Vecerik, N. Ye, S. Schaal, and J. Scholz. Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi: 10.11.677//SSS.22.....
[4] T. Zhao, Luo, O. O. Sushkov, R. Pevceviciute, N. M. O. Heess, J. Scholz, S. Schaal, and Levine. Offline meta-reinforcement learning for industrial insertion. 2022 International Conference Robotics and Automation (ICRA), pages 6386-6393, 2022.
[5] C.-Y. Chai, K.-F. Hsu, and S.-L. Tsao. Multi-step pick-and-place tasks using object-centric dense correspondences. In 2019 IEEE/RS International Conference Intelligent Robots and Systems (IROS), pages 4004-4011. IEEE, 2019.
[6] E. Johns, Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In IEEE International Conference on Robotics and Automation (ICRA), pages 4613 4619. IEEE, 2021.
[7] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
[8] B. Wen, W. Lian, K. Bekris, and S. Schaal. You only demonstrate once: Category-level ma- nipulation from single visual demonstration. In Robotics: Science and Systems (RSS), 2022.
[9] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, L. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL bb88660065445571117777711110 Paper.. pdf.
neurips www.rppp..cpppprr.iiiie.....
[10] A. Ganapathi, Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali, et al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515-11522. IEEE, 2021.
[11] H. T. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pages 347-363, Springer, 2.21.

---

[12] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. Learning predictive representations for deformable objects using contrastive estimation. In Conference on Robot Learning, pages 564-774, PMLR, 2021.
[13] P. Mitrano, D. McConachie, and D. Berenson. Learning where to trust unreliable models in an unstructured world for deformable object manipulation. Science Robotics, 6(54))eabd8) 70, 2021.
[14] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning to see, simulate, and shape elasto-plastic objects with graph networks. In Robotics: Science and Systems (RSS), 2022.
[15] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256-266. PMLR, 2022.
[16] Z. Huang, X. Lin, and D. Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation. In Robotics: Science and Systems (RSS), 2022.
[17] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019, URL http::///ppeeeeeeeeeee///ooororr
(18) Y. L., Wu, J.-Y. Y. Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pages 1211- IEEE, 2019.
[19] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[20] M. Chang, T. D. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional object-based approach to learning physical dynamics. In 5th International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:://ppererri/////// htttpp///ppaarrreeeeeee///o/aaaa
[21] A. Sanchez-Gonvalez, N. Heess, J. T. Springenberg. J. Merel, M. Riedmiller, R. Hadsell, and Battaglia. Graph networks as learnable physics engines for inference and control. In Inter- national Conference on Machine Learning, pages 4470-4479, PMLR, 2018.
[22] N. Funk, G. Chalvatzaki, B. Belousov, and J. Peters. Learn2assemble with structured repre- sentations and search for robotic architectural construction. In Conference on Robot Learning, pages 1401-1411. PMLR, 2022.
[23] T. Silver, R. Chitnis, A. Curtis, J. B. Tenenbaum, T. Lozano-Péree Pérez, and L. P. Kaelbling. Plan- ning with learned object importance in large problem instances using graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11962-11971, 2021.
[24] A. E. Tekden, Erdem, E. Erdem, M. Imre, M. Seker, and Ugur. Belief regulated dual propagation nets for learning action effects groups of articulated objects. 2020 IEEE In- termational Conference Robotics and Automation (ICRA), pages 10556-10562, 2019. URL
semanticscholat tttppc//ppp...mmm.cccccoooo..t org/CorpusID:: : 216450444.
[25] A. E. Tekden, A. Erdem, E. Erdem, T. Asfour, and E. Ugur. Object and relation cen- tric representations for push effect prediction. ArXiv, abs/2102.02100, 2021. URL https:
/ api www.nticscoooanrr www.aaaroorCCCCCCDD--C. 231786514.
10

---

[26] Y. Huang, A. Conkey, and T. Hermans. Planning for multi-object manipulation with graph neural network relational classifiers. Automation (ICRA), pages 1822-1829, 2022. URL https: httpp::/aap..ssnnniicchheee semasmicccccccooaar org/
2023 IEEE International Conference on Robotics and
[27] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In L. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference Robot Learning. CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learn- ing Research, pages 1025-1037. PMLR, 2019. URL v100/ gupta20a.... html.
mlr. prrss//
[28] A. Kadian, J. Truong, A. Gokaslan, A. Clegg. E. Wijmans, S. Lee, M. Savva, S. Cher- nova, and D. Batra. Sim2real predictivity: Does evaluation in simulation predict real- world performance? JEEE Robotics and Automation Letters, 5(4):6670-6677, 2020. doi: 10.1109/LRA.2220000033
[29] N. Di Palo and E. Johns. Learning multi-stage tasks with one demonstration via self-replay. In Conference on Robot Learning, pages 1 1180-1889. PMLR, 2022.
[30] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems, 4(1):265-293, 2021. doi:10. 1t01://ammmurroccooooo0000000000 //doi. orooi.oor//00. 1 canuuree-caatrol--911000------
URL https:
[31] X. Lin, Z. Huang. Y. Li, B. Tenenbaum, D. Held, and C. Gan. Diffskill: Skill abstraction from differentiable physies for deformable object manipulations with tools. The Tenth In- ternational Conference on Learning Representaations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[32] O. Khatib. A unified approach for motion and force control of robot manipulators: The op- erational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987. doi: 10. 11w.110//RR..... 199.///..198..019990000.. 1087068.
[33] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martin, A. Joshi, S. Nasiriany, and Zhu, robo- suite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. (00009.1229.. 12293, 2020.
[34] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033, 2012. doi: 10. (O00//RROS.0022.6388888
[35] Y. Rubner, C. Tomasi, and L. Guibas. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision (IEEE Cat. No. 18CH62136771) pages 59-66, 1998. doi: 10. 11.0..119900
[36] H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In Proceedings of the 5th International Joint Conference Artificial Intelligence Volume 2, IJCAI'77, page 659-663, San Francisco, CA, USA, 1977. Morgan Kaufmann Publishers Inc.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[38] T. Haarooja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum en- tropy deep reinforcement learning with a stochastic actor. In J. G. Dy and A. Krause, ed- itors, Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- chine Learning Research, pages 1865. PMLR, 2018. URL http: http::/proceedaaee /ttp///rroeeee... mlr, press/v80 httph/aaannnaaaaa html.

---

Appendices
A Additional Results
A.1
Generalization to Deformable Objects in Real-World Experiments
In effort to explore the adaptability of our method, we further extended our experiments con- sider deformable objects. Intriguingly, even though our methodology was trained predominantly using box-shaped rigid objects, it showcased commendable generalization capabilities. Neverthe- less, expected, the intricacies of deformable objects' dynamics, being distinct from rigid ones, posed challenges, resulting in somewhat diminished performance compared to other test cases.
Figure 6: Experimental setup with deformable objects.
Success Rate
Setups
Heuristic
RoboCraft
skills
skills
Deformable Objects
0/10
2/10
6/10
4/10
Table 4: Success rates when dealing with deformable objects.
A.2 Impact of Sample Size Prediction Performance
We examined the influence of training sample size on dynamics prediction performance. By varying number of training samples and maintaining consistent validation and test set, we observed relationship between sample size and prediction accuracy in Figure 7. Specifically:
For the push behavior primitive, performance improvements platcaued around 50 samples.
For the sweep behavior primitive, similar plateau was observed at 50 samples.
The transport behavior primitive exhibited steady performance gains until approximately 140 samples.
Implementation Details
B.1 Determining the Edge Formation Threshold in GNN
The optimal threshold for forming edge in graph representation empirically derived to strike balance between computational complexity and representational expressiveness. Just as with other hyperparameters essential for training our deep neural networks, we determined thresh- olds through extensive experimentation, primarily guided by validation loss. Depending on the spe- cific behavior being modeled, we adopted the following thresholds to ensure the edges encapsulate meaningful interactions:
Transport:
Intra-object relations: 0.15m
Gripper-to-object relations: 0.175m
12

---

Push
Transport
Push
Sweep
Transport
Mean Squared Eiror (mm)
0.6
6.4
0.3
0-
0.-
0.1
120
180
Dietance (mm)
Push
Sweep
Transport
120
22
20
150
100
18
100
60
14
20
40
90
100
120
100
150
200
100
120
140
160
160
Push
Sweep
Transport
Chamfer Distance (mm)
25.0
250
120
23.5
200
100
150
100
12.5
40
120
Number of Samples
Number of Samples
Number of Samples
Figure 7: Variation in prediction performance function of training sample size.
Push:
Intra-object relations: 0.1m
Gripper-to-ohject relations: 0.1m
Sweep:
Intra-object relations: 0.175m
Gripper-to-object relations: 0.175m
When the contact distance is excessively large, it leads the inclusion of significant number of edges that may not be relevant to the scene, which makes the model computationally expensive and introduces noise which can hinder both the training process and the quality of inference. A threshold that is too small may miss vital interaction relations. We tuned these thresholds to effectively capture nuances of object interactions to optimize the prediction accuracy.
B.2 Real Robot Experiment
Our perception pipeline uses two OAK-D Pro cameras to estimate the 6D (x, y, 7, roll, pitch, yaw) pose of objects. The side camera, oriented towards the shelf, determines the poses of the objects on the shelf. In contrast, the top camera determines the pose of the object situated on the table. We maintain a database encompassing comprehensive details of all objects in the experiment, including their size and texture. We approximate all objects cuboids, defined by three parameters: length, width, and height. Side and top view images of these objects are captured form collection of ground truth images. These images aid the identification of objects via the SIFT (Scale Invariant Feature Transform) feature matching algorithm. This algorithm is executed for each object in the
Perception:
13

---

scene to match keypoints, which are subsequently used to calculate the homography matrix. We then use this matrix to calculate the 2D coordinates and ID orientation for each object in the image space. Given the setup of our experiment, it suffices to estimate 3 degrees freedom (DOF) for all objects: (y, 2, roll) for objects on shelf and (x, y, yaw) for the object on the table. The other DOF is deterministic based on the known shelf location and object size. Incorporating the 3D pose of each object in the image space with prior knowledge of the experiment setup, obtain the 6D pose for all the objects in the scene for robot manipulation.
(a) SIFT featuring matching with top-down view camera
SIFT feature matching side view camera
(c) SIFT feature matching with side view cameea
Figure 8: SIFT results with setup-(f). The ground truth images collected before the experiment are shown in the upper-left corner each subfigure. The green lines show keypoints correspondence.
B.3 More information on the objects
Table 5: Object Dimensions
Length (m)
Width (m)
Height (m)
Object
Image
Domina ERAMULATED
Sugar box
0.089
0.040
0.177
10
Lint roller
0.110
0.050
0.230
Clorox bottle
0.065
0.255
0.055
CHEEZ-TT
Cheez-it box
0.152
0.047
0.190
Charger box
0.168
0.051
0.240
Continued on next page
14

---

Table 5 Continued from previous page
Length (m)
Width (m)
Height
Object
Image
(m)
STANOS
Book
0.203
0.028
0.245
Amazon box
0.210
0.288
0.045
VW
KETO
Keto box
0.193
0.259
0.046
Kellogg's box
0.190
0.045
0.284
WEGH PROTER
what Chex
Chex box
0.195
0.050
0.285
Cheerios
Cheerios box
0.195
0.052
0.285
120%
Fabie
Bran flakes box
0.198
0.297
0.057
Power drill box
0.250
0.073
0.279
Yellow plush toy
0.065
0.255
0.055
Blue plush toy
0.19
0.284
0.045
Continued on next page
15

---

Table 5 Continued from previous page
Length (m)
Width (m)
Height
Object
Image
(m)
Red plush toy
0.168
0.24
0.051
B.4 Subgoal Selection for Behavior Primitives
Figure 9 illustrates the typical subgoals selected during our real-world experiments. In our approach, subgoals serve as intermediate waypoints or milestones within the task. They provide sequence of specific, short-term objectives for the robot to achieve, ultimately guiding it towards the final desired outcome of the task. This structure helps in breaking down a complex, long-horizen task into more manageable segments, making easier for the robot execute and adapt.
(a) Initial setup.
(b) Sweeping subgoal.
Pushing subgoul.
(d) Transporting subgoal.
Figure 9: Typical subgoals selected during experiments. From left to right: (a) Initial setup, (b) Sweeping, (c) Pushing, (d) Transporting.
16


Reviews:
[{"review_id": "bq7IN2fwOa9", "reviewer": "Reviewer_qLwL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose, essentially, an extension to RoboCraft which models rigid-static object interactions (and gripper state) which they validate on a stowing task in both simulation and on a physical hardware. They estimate the dynamics of the scene (in a simplified uniform-density box-shaped representation)  via a graph neural network and evaluate several methods of planning (finding parameters for three parametric skills) with this dynamics, the most effective one being random shooting.", "summary_of_recommendation": "The amount of novelty is a bit low, however the expanded experiments make this paper valuable to the field. I advocate for acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Do the authors have a hypothesis for why this approach consistently underperform RoboCraft (even if only modestly) in the simple Push case?\n\n- Can the loop be closed during execution of each primitive, allowing for parameter re-estimation during execution?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper feels a little bit narrow, though it also serves as a limited and partial reproduction of parts of RoboCraft. The primary contribution compared to prior seems to be modeling dynamic/static interactions, which is good, but conversely this work does not evaluate settings with deformable objects or non-box-shaped representations.\n\nThat being said, the experimental domain (stowing) used in the paper is interesting with a good mix of difficulty and practical applicability. The author's action formulation was also reasonable and the results shown in real-world experiments are strong (95 percent) though the lack of a RoboCraft comparison in this setting weakens the result.\n\nThe paper would also benefit from additional ablation investigations in simulation, such as the effect of pose-estimation error and the dimensionality of the action parameter space on performance. It would be particularly interesting to see if random shooting still outperforms RL approaches such as PPO as the action parameterization becomes higher dimensional and more complex (I expect it would not, but it would be a helpful contribution for this to be evaluated).\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "IspUYb6J-r", "reviewer": "Reviewer_NnNZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a graph neural network based approach for placing objects on cluttered bins and shelves. One of the key challenges in their stowing task is that objects must be pushed or manipulated out of the way in order to clear space for them.\n\nThe world is represented as a bunch of interacting rigid bodies through a graph neural network (GNN). Each object is represented by particles (nodes in the graph). Edges are formed when distance is under a threshold. The action is also modeled as a trajectory of particles, over three primitives: sweeping, pushing, and transporting.\n\nThe authors built a simulated environment on Robosuite with varied shelf sizes and objects. They trained on 300 episodes, and evaluate on simulated and real robotics tasks with a variety of objects, and compare against heuristics and (in sim) against SAC and PPO.\n", "summary_of_recommendation": "I think the learned GNN-based dynamics model, the method of planning over primitives, and the interesting task with real-world verification all make this an interesting paper. I would be interested to see how the method could be applied to other tasks.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Do you need any additional primitives? How much engineering went into primitive design; could they be learned somehow?\n- How easily can these dense, particle based representations be created from RGB-D camera data? Do you need object models?\n- What affect do hyperparameters like contact distance, number of particles per object have?\n- What is the density of particles per object?\n- How long does it take to train? How does the amount of training data affect dynamics model performance?\n- How much did you need to vary the parameters of each primitive to learn a good dynamics model?\n- Why not compare to heuristic in simulation in addition to SAC and PPO?\n- How could you apply this to a wider range of tasks? Would it scale to tasks other than this shelf task?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think that the premise of the paper is interesting, and the task - while similar to many other rearrangement problems - is much more realistic. The GNN dynamics model seems novel, especially when applied to robot planning to manipulate in clutter. Generally, I really like the idea of using a dynamics model together with action primitives. The problem is also an interesting manipulation problem that involves a lot of object-object contact.\n\nThe GNN represents objects as a collection of particles. This is a different formulation from most GNN policies, and I think it's a very interesting one. However, it seems like there are a lot of hyperparameters involved - how many particules, when to create edges - and I wonder what the effect of these is. Likewise, the choice of key points in the trajectory, which the dynamics model uses to predict where objects will go, seems very important - this detail isn't really in the paper.\n\nThey train on 300 trajectories and test on object, including held-out objects with a different shape (it was only trained on boxes, but they test on bottles) - although they still represent it as a box during inference, which implies it might fail in other circumstances where shape matters more. I'm a bit surprised how little data the authors needed to train their dynamics model - 300 trajectories is not a lot! Although the action space is quite limited.\n\nThey show the method in simulation and the real world, and in particular they show it succeeding in a range of different challenging stowing tasks in the real world. In the end, I think the idea is straightforward, novel, and interesting. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "HF_BiEQVWhd", "reviewer": "Reviewer_pqt2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a graph neural network framework to address the task of stowing. Also, given the demonstration during training, this paper trains a graph neural networks-based dynamics model to model the interaction of multiple objects. With the learned dynamics, the paper uses a random shooting to choose the best action to achieve several keyframes sequentially to achieve the goal. Based on the training purely in simulation, this framework can generalize to the real-world without any finetuning.", "summary_of_recommendation": "This paper investigates a novel and interesting stowing task with some cool real-robot demos. However, this paper should discuss more related works like [1][2] to highlight the core contributions of this paper. Furthermore, the presentation of the paper is unclear and the paper should include some important details about the planning algorithm. Thus I recommend “weak reject”.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How does the author choose the keyframes or subgoals? \nThis seems like a very important assumption during both training and planning. \n\nMore details about the planning part. How do you optimize both the discrete part and continuous part of your skill primitives?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths of the paper: \n1: The stowing task is novel and important for the robot manipulation community. \n\n2: This approach is trained purely in simulation and can generalize to real-world scenarios. \n\n3: The real-robot demos look cool and interesting. \n\nWeakness of the paper:\n1: the author proposes a graph neural network framework to model the dynamics of multi-object interactions. Both dynamics learning and planning are pretty similar to [1]. The authors should discuss the difference with the previous work. \n\n2: The author highlights the contribution of long-horizon planning but does not cite enough related works like task and motion planning [2].  \n\n3: The paper is hard to follow. For example, the authors define subgoals at the beginning of the approach section without defining them. Also, the authors do not discuss the details of the loss function. If it’s limited by space, the authors should at least put the details of the loss function in the appendix. \n\n4: Planning is a very important part of the paper but the authors do not discuss that part clearly. The authors only mention “We employ trajectory optimization to find the skill parameters that yield the lowest cost.” How do you optimize both the discrete part and continuous part of your skill primitives? \n\n5: The authors claim “In our analysis, we evaluated Random Shooting, the Cross-Entropy Method, and Gradient Descent - all of which exhibited similar performance.” This seems weird to me, it’s clear that the cross-entropy method and gradient descent will perform much better than naive random shooting. My guess is that the assumptions of manually choosing good keyframes make the planning problem easy so even the naive random shooting works well. \n\nMinor points: \n1: Figure 2 is hard to understand. \n\n2: training time instead of training times.\n\n[1] Yixuan Huang, Adam Conkey, and Tucker Hermans. Planning for Multi-Object Manipulation with Graph Neu- ral Network Relational Classifiers. In IEEE International Conference on Robotics and Automation (ICRA), 2023.\n\n[2] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Perez, “Integrated task and motion planning,” Annual review of control, robotics, and autonomous systems, vol. 4, pp. 265–293, 2021.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "5IeNmDKXyt", "reviewer": "Reviewer_kz8m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper implements a pipeline that predicts object pose changes using Graph Neural Networks (GNN) in an environment with a variable number of objects. The system is designed to control the robot gripper, minimizing the expected and actual object pose in order to successfully execute a number of skills, including sweeping, pushing, grasping-transferring-releasing an object. For this, the system needs to learn GNN-based effect prediction on objects given action parameters, i.e. typical motions of objects in response to the actions. These are learned from random executions and human demonstrations. In the use case, the actions are chained in order to stow an object to a rack that includes other objects. The learning occurred in the simulator and was verified in the simulator and the real world. The authors provided results and comparisons with other methods.", "summary_of_recommendation": "The paper proposed a sound method that combines an existing GNN-based method with a simple action control optimizer. While I appreciate the writing, the working system, and the verification in the simulator and real world, the paper does not make a strong contribution to the field theoretically or as a very strong system in practice.\n\nUpdate: The paper now shows that their particle-based GNN outperforms the existing object-level GNN architectures. The paper now also includes deformable objects in experiments. Therefore, I increase my recommendation score as the paper now justifies its contribution better.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Please provide the novelty of the prediction part of this work considering other GNN-based predictors.\n- Please provide a comparison of the complete system with a more similar framework, for example, Tekden et al. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper is very well-written, easy to follow. \n- The idea of using GNN for predictive modeling of multiple object interactions is not new, as the authors indicated, but the authors missed some very related work [Tekden, 2020; Tekden 2021]. The proposed method uses particle-based object representation, whereas Tekden uses object-level representation in GNN. As all objects are assumed to be box-shaped rigid objects, does more-complex particle-based representation have any advantage?\n- As a general remark, although I believe that the framework is quite general, the paper writing and method description is too much geared towards the stowing task, which is, while very common, a specific task among many others. The authors should provide a more general approach.\n- The authors mention some challenges related to detecting contacts and friction, but they do not come back to these challenges in the rest of the paper.\n- The argument in lines 38-39 (related to failure due to human labeling) needs reference and justification. Most of this work is not self-supervised.\n- The argument in lines 83-84 also needs justification/reference.\n- Please explain what Random Shooting corresponds to.\n- In 103-104, the authors argue that their approach is flexible and adaptable to variations in object shape, size, and geometry. This is not rigorously justified in the experiments section. This can be shown by verifying the learned system with objects not observed during training. \n- The sub-goal explanation for Fig. 2 (l95) is not clear.\n- l113: The attributes include gravity; probably, it is supposed to be either density or mass.\n- l114: How did you set the threshold to form an edge?\n- l116: Please provide the complete set of relations.\n- l112: What does 'object it belongs to' mean? Do objects have ids?\n- The goal-conditioned trajectory optimization part is not clear. Please precisely explain the algorithm. How exactly the optimization occurs?\n\n- How is the data, collected in data collection, being used precisely? To train GNNs? Are 300 episodes sufficient to cover the wide range of object-action interactions? \n- How is the data obtained from the demonstrations used exactly? The authors mentioned they used teleoperation. How many teleoperations are done? \n- In the explanation of real-world experiments, point cloud representation is indicated to be created. How is this information used?\n\n- In the first experiment, the proposed method is shown to outperform the GNN-based baseline. However, the margin is small, and they are very similar methods with slight differences. Therefore the contribution of this result is not substantial.\n- In the second experiment, the proposed method is compared against a completely different approach, RL-based methods, where no details are provided. The authors need to compare theirs with an MPC-based method to show effectiveness.\n- How is the randomization realized in the third experiment (in the real world)? The assumption of complete object knowledge is a very strong one, and together with the object recognition step, the details should be moved from the appendix to the main manuscript. The baseline 'heuristic' approach is not clearly explained, and from the videos, it looks not a strong baseline. \n\n\nTekden et al. \"Belief regulated dual propagation nets for learning action effects on groups of articulated objects.\" ICRA 2020.\nTekden et al. \"Object and relation centric representations for push effect prediction.\" arXiv preprint arXiv:2102.02100 (2021).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree this paper proposes a novel model for representing skill-based dynamics using a graph neural network over particle-based representations of the object and robot. This is distinct from recent object-based GNN models and the experiments show the particle-based GNN outperforms existing object-level GNN architectures. This particle-based representation is highlighted in its ability to model deformable objects.

The model is evaluated on a stowing task, which all reviewers agree is an interesting and challenging task. As reviewer NnNZ points out is "is much more realistic" than most existing rearrangement planning papers.

I think this combination of strong architectural improvement with increased difficulty of task makes this an excellent paper for presentation that would help move the manipulation community forward towards more challenging and constrained multi-object manipulation tasks.
Decision: Accept (Oral)