Learning Geometric Reasoning Networks For Robot Task And Motion Planning
Authors: Smail Ait Bouhsain
Rachid Alami
Thierry Simeon
Venue: ICLR 2025 Poster (2025)
Keywords: Graph Neural Networks
Deep Learning for Robotics
Task and Motion Planning
Robot Manipulation Planning
Primary area: applications to robotics, autonomy, planning

Abstract: Task and Motion Planning (TAMP) is a computationally challenging robotics problem due to the tight coupling of discrete symbolic planning and continuous geometric planning of robot motions. In particular, planning manipulation tasks in complex 3D environments leads to a large number of costly geometric planner queries to verify the feasibility of considered actions and plan their motions. To address this issue, we propose Geometric Reasoning Networks (GRN), a graph neural network (GNN)-based model for action and grasp feasibility prediction, designed to significantly reduce the dependency on the geometric planner. Moreover, we introduce two key interpretability mechanisms: inverse kinematics (IK) feasibility prediction and grasp obstruction (GO) estimation. These modules not only improve feasibility predictions accuracy, but also explain why certain actions or grasps are infeasible, thus allowing a more efficient search for a feasible solution. Through extensive experimental results, we show that our model outperforms state-of-the-art methods, while maintaining generalizability to more complex environments, diverse object shapes, multi-robot settings, and real-world robots.

Published as a conference paper at ICLR 2025
LEARNING GEOMETRIC REASONING NETWORKS FOR ROBOT TASK AND MOTION PLANNING
Smail Ait Bouhsain, Rachid Alami & Thierry Siméon Laboratory for Analysis and Architecture of Systems (LAAS) National Center for Scientific Research (CNRS), Toulouse, France {sai tsaitbouhsa, a a alami, , s simeon ) llaas fr
ABSTRACT
Task and Motion Planning (TAMP) is computationally challenging robotics problem due to the tight coupling of discrete symbolic planning and continuous geometric planning of robot motions. In particular, planning manipulation tasks in complex 3D environments leads to a large number of costly geometric planner queries verify the feasibility of considered actions and plan their motions. To address this issue, we propose Geometric Reasoning Networks (GRN), a graph neural network (GNN)-based model for action and grasp feasibility prediction, designed to significantly reduce the dependency on the geometric planner. More- over, we introduce key interpretability mechanisms: inverse kinematics (IK) feasibility prediction and grasp obstruction (GO) estimation. These modules not only improve feasibility predictions accuracy, but also explain why certain actions grasps are infeasible, thus allowing a more efficient search for a feasible solu- tion. Through extensive experimental results, we show that our model outperforms state-of-the-ar methods, while maintaining generalizability to more complex en- vironments, diverse object shapes, multi-robot settings, and real-world robots.
INTRODUCTION
Task and Motion Planning (TAMP) (Garrett al.] 2021) robotics problem in which the goal is to find sequence of robot actions and their corresponding motions to transition an environment from an initial state to a goal state. most cases, the order of actions, their grounded parameters, and the feasibility of the corresponding motions are tightly coupled, requiring careful combination of symbolic task planning and continuous geometric planning. However, the resulting large search space leads to a combinatorial explosion. Moreover, every action considered must be validated geometrically to ensure feasibility, leading to numerous costly queries to the geometric planner. become a bottleneck for TAMP (Bouhsain et al. 20236).
Thus, geometric planning
demonstrate that learn- ing methods can help accelerate TAMP by providing fast geometric feedback the task planner. In particular, action feasibility prediction offers an efficient alternative to geometric planning during the TAMP process. In offline manipulation planning, can answer critical questions such which object can be picked, how to grasp it, where to place it, or how to free access to it. However, action feasibility prediction presents several challenges. A suitable representation of 3D environments is needed, as these may contain an arbitrary number of objects, along with action representations that capture varying parameters such as grasps. Furthermore, predictions must not only be fast and accu- rate but also interpretable to understand why an action is infeasible and how to rectify it (e.g. another object is blocking access to the desired one). Finally, action feasibility prediction must generalize to environments with numerous objects, objects of varying shapes, and multi-robot settings.
Previous works [Wells et al. 2019: Driess et al., (2020a) Khodeir al. 2023a
Existing approaches to action feasibility prediction often struggle with interpretability, scalability, and generalization across diverse environments. To address these limitations, we propose novel approach that leverages a GNN-based model for robot action and grasp feasibility prediction. Our method constructs graph representation of 3D environments, where fixed and movable objects represented as nodes, and edges capture spatial relationships and interaction constraints. Through this graph-based structure, leverage an Edge-Ennaaced Graph Attention Network (EGAT) to

---

Published as a conference paper at ICLR 2025
predict action and grasp feasibility for each movable object. A unique aspect of our approach is the introduction of two interpretability mechanisms: inverse kinematics (IK) feasibility predictions, which determine whether the robot can feasibly manipulate an object from different sides, and Grasp Obstruction (GO) predictions, which quantify how neighboring objects restrict access to grasps predict action
from different sides of the object. These interpretable features not only allow infeasibility, they also explain why a specific action fails, enabling more efficient planning.
The contributions of this paper are threefold: (1) We propose novel GNN-based model for efficient and accurate action and grasp feasibility prediction in complex 3D environments. (2) We introduce two mechanisms, inverse kinematics (IK) feasibility and grasp obstruction (GO) predictions, that im- prove feasibility prediction while enhancing its interpretability. hence task planning efficiency. (3) We provide comprehensive experiments showcasing our method's 's state-of-the-an (SOTA) perfor- mance, including evaluations of its interpretability and generalization capabilities. Our code is avail- able at:
com/ Sma geomet r reason i ngg www.nnetworrte git
RELATED WORKS
2.1
TASK AND MOTION PLANNING
Task and Motion Planning (TAMP) combines discrete symbolic task planning with continuous ge- ometric motion planning achieve robotic manipulation goals. Early approaches (Alami et al. 1990 Koga & Latombe 1994, Ahuaactin et al., 1998, Siméon et al. (004) view the problem pri- marily from a geometric perspective. These approaches build manipulation graphs that map valid grasps, placements, and connecting motions. Although, they suffer from a combinatorial explosion as the number of objects increases. Later, multi-modal motion planning (Hauser & Latombe 2010 Hauser & Ng. Teg-Thow-Hor Show-Hing] 2011) generalized these methods using masstraint-base graphs, but the complexity of constructing these graphs for cluttered environments remains challenge. Modern methods integrate symbolic task planning with geometric planners, using backtracking when actions are infeasible (Cambon et al. 2009, (Srivastava et al., 2014, Lagriffoul al.) 2014, Dantam et al., Garrett et al., 2018)) 8). However, in complex environments, the large number of queries to the geometric planner causes significant computational overhead, leading to slow planning processes.
2016
2.2 LEARNING FOR TASK AND MOTION PLANNING
Many recent works leverage learning methods to provide heuristics for TAMP. Chitnis et al. (2016)
apply reinforcement learning (RI.) to learn policies for task refinement. Xu et al. (021)) model both immediate and future affordances in a learned latent space. Some methods leverage GNNs
to identify the smallest set of objects for solving a planning problem (Silver et al. 2021 ), learning
inter-robot relations verify subgoal satisfaction (Huang et al. 2023), or predict dynamic object interactions (Chen et al. 2023). [Eisner et al. (2024) tackles the problem of precise relative object placement using an (B(3)-equvvarian learning. AAgi et al. (2023) learn task-agnostic policies for various robot skills and verify their feasibility. Other works learn to fully solve TAMP problems. Zhu al. (202T) | leverage two-level scene graph for neuro-symbolic task planning and graph-based motion generation. McDonald & Hadfield-Menell (202)) propose an imitation learning method that mimics a TAMP solver. Lin et al. (2022) introduce a GNN-based policy architecture trained on expert demonstrations. Haramati et al. (2024) propose a goal-conditioned RL framework using an object-centric image representation of the 3D environment. These methods tend to be problem- dependent, needing further training on unseen manipulation problems. Wang et al. (2024), Lin et al.) [203);; Huang et al.] (2024b) leverage large language models (LLM) to perform TAMP, incorporat- ing feedback from geometric planning failures predicted geometric infeasibility into the LLM to iteratively refine the solution. Although, these methods require large resources for running LLMs.
2.3 ACTION AND GRASP FEASIBILITY PREDICTION FOR MANIPULATION PLANNING
Action and grasp feasibility prediction is a fast growing research topic, which aims at accelerating common TAMP algorithms by reducing their dependency on the geometric planner. WWlls et al. (2019) pioneered this problem by proposing to Support Vector Machines (SVM) action and grasp feasibility classifiers. Though, this method is limited to simple environments with a fixed num- ber of objects. To tackle this issue, Driess et al. (2020b a): Xu et al. (2022) propose to use top-view
2

---

Published as a conference paper at ICLR 2025
Action Feasibillly Precictions
Gresp Types Feesibility Predictions
Predicted infessibility couse
Input 3D Scene
gross type (for the blue object
Access
Top
Front
Ree
Salutizo
Solution
Obstructed
Right
Left
Infeasilble
Atesp
Obotruction
retio
Solution
Clutter.
Top
Front
Reaa
Problem
Gistructed
Solution
Right
Feasiible
Infoesibbe
Gresp
0.5
Oostruction
ratio
Gestraciet
No IK Solition
Figure 1: Visualization of GRN predictions on two manipulation problems, Access (Panda arm) and Clutter (PR2 Robot, predictions shown for its right arm). A single query GRN outputs predictions for each movable object in the environment: Action feasibility, grasp types feasibility (two views), and the predicted infeasibility causes for each grasp type. For clarity, we show the predicted infeasibility cause for one object only (shown in blue), and distinguish two cases: (1) No IK solution: robot shown in red. (2) Grasp rype obstructed: we show all obstructing objects color gradient representing the obstructions ratio. Arrows show approach directions of grasp types.
depth images input to convolutional neural network (CNN). However, this approach is limited to tabletop problems. [Bouhsain et al. (2023a] generalize this method to 3D environments by us- ing 5 depth images from different
views. Bouhsain et al. (2024) extend this method mesh objects and multi-robot problems. Yang et al. (2022) also use multiple views, combined with text descriptions of actions and predicates input transformer. Such image-based representations suffer from occlusions, which can hurt feasibility prediction accuracy. Some methods use 3D voxel grids (Park al. 2022) or pointclouds (Huang et al. 2024a) to represent 3D scenes. Though, they suffer from high inference times and are, hence, limited to environments with few objects. Other works such as Kim et al. (2022); Khodeir al. (2023a by; Sung et al. (2023) tackle this issue by representing 3D environments using scene graphs, leveraging GNNs predict the success of geo- metric planning steps from prior search experiences. These methods, however, lack interpretability and can not provide feedback why actions are infeasible.
PROBLEM DESCRIPTION
In this work, we address the problem of predicting action and grasp feasibility for offline manipula- tion planning in 3D environments. The goal is determine whether robot can successfully plan actions, preciscly picking placing objects, and which grasps allow their collision-free motions, while accounting for inverse kinematics constraints and potential grasp obstructions.
3.1 OFFLINE MANIPULATION PLANNING CONTEXT
Our approach is tailored for offline manipulation planning tasks in 3D environments, containing fixed objects and movable objects. We assume the shape, dimensions, and pose of all objects are fully known, and that all objects remain static unless moved by the robot. A pick action involves grasping a movable object, while a place action refers to placing it at target pose. From a motion planning perspective, these actions symmetrical (Wells al. 2019 Bouhsain et al. 2023h)
3

---

Published as a conference paper at ICLR 2025
Assuming they start and end at the same home configuration of the robot, a place action a specific pose is the reversed pick action from that pose. Throughout this work, the term "action" " refers to both pick and place actions. Planning action first requires sampling grasps. For each grasp, an inverse kinematics (IK) solver computes the corresponding robot configurations. If one exists, is checked for collisions before motion planner computes the robot's full trajectory. This process involving grasp sampling, IK solving, collision checking, and motion planning is computationally expensive and incurs significant overhead during TAMP, especially in cluttered 3D environments.
We define G = (Top, Front, Rear, Right, Left} as set of 5 grasp types, each one representing a subspace of grasps related the side from which the object is grasped. We focus on axis-aligned grasps, such that each type is the continuous set of grasps for which the end-effector's ' axis is parallel to one of the object's principal axes a specific direction. This representation is similar to those introduced in previous works (Wells et al. 2019 Driess et al. 2020b
Bouhsain et al. 2023a b).
3.2 FEASIBILITY PREDICTION
This work aims to reduce the dependency on the complex geometric planning process involved in offline manipulation planning. Given an environment E and object of interest O E, the goal is to predict both the feasibility Fa (O) E) E I of picking or placing O at its pose in E, and the feasibility of each grasp type, F (O, E), Vg E G] E RRS.
Moreover, we aim to estimate the cause of infeasibility for each grasp type. We focus on two primary factors contributing to infeasibility: (1) the absence of a valid inverse kinematics (IK) solution for all grasps within grasp type Kg [Kg[ (O, E).Yg Vg E g] E R and (2) the obstruction of grasps by each neighboring objects, represented as the ratio Pg of obstructed grasps:
Ps = (0,(',''), Vgeg, VO' EN(O E RRxMW(()
where (NO) denotes the distance-based neighborhood of O and [[O) its cardinality. In addition to helping explain infeasibility and providing insights into the constraints imposed by the environ- ment, these predictions are also used in the feasibility prediction process. In summary, the task at hand is to learn two classification functions fr, fnx and regression function fp s.t.:
Fa Fg
fr(O, E, Kgg PG)
where
Kig f=(0, E) and Pg = f0(O F)
(2)
GEOMETRIC REASONING NETWORKS
We propose Geometric Reasoning Networks (GRN), a three-module GNN-based neural network which takes as input a graph representation of the environment, and outputs the action and grasp types feasibility for each movable object, as well as inverse kinematics feasibility (IK) and grasp ob- struction (GO) predictions cf. Figure D}. The main challenge of learning methods in a manipulation planning context finding an appropriate representation of the 3D environment which can con- tain an arbitrary number of objects. Previously proposed image-based representations suffer from occlusions as the number of objects in the environment increases, which can significantly impact performance. In this paper, we tackle this issue by representing 3D scenes graphs where nodes represent fixed and movable objects and edges represent geometric relationships between objects.
4.1 3D SCENE REPRESENTATION
Given an environment E, we construct directed graph (V,E) where each node corresponds to an object in E, fixed or movable. Nodes have a feature vector x = [1, w, h, x. 2,,3.22 07 where (1, w, h) are the length, width and height of the object's bounding box, (a,, y, 2) represent the position of the object in the environment, and 0 is the object's orientation w.r.t its z axis. The position and orientation of the object are expressed in the reference frame of the base of the robot. This allows a straightforward generalization to multi-robot settings simply by switching the reference frame to the base of the robot of interest. For each node u corresponding to a movable object O, we add a self-loop edge (u u) E E, as well directed edge (v + u) E from each node corresponding to a neighboring (fixed or movable) object O' E (OO). Two objects are considered neighbors if the euclidean distance between their closest points along the (æ, y) plane is lower than
4

---

Published as a conference paper at ICLR 2025
Self-loop edges features
Update soth-oop edges festures
Geometric Rensoning Networks (BRN)
Nodes features
IK Feasibility Module
Kg
Movable
MLP + Sigmold
Action
F.
Updetec Graph
Feasibility Mocule
Fiee
Proximity edges
Gresp Obstruction
EGAT Sigmoid
features
Proviity
Module
MLP HardTanh
Pg
Enviroment
Ladate proxinity
Graph
object noce
Movable nodo
Forword pass
Self-loop edge Prowineyy edge
Backwaard pass
Figure 2: Complete GRN architecture. A scene graph is constructed from the input 3D environ- ment. Node features of movable objects are given to IK feasibility module which outputs are used to update self- edge features. The concatenated features of nodes linked through proximity edge are fed to the GO module to get grasp obstruction estimations, which are appended to proximity edge features. Finally, the updated graph is given to the AGF module to predict the action and grasp types feasibility for each movable object the environment.
user-defined threshold. Since our approach is GNN-based, where edges are used to determine which features to aggregate, the more edges are present in the input graph, the higher the number of computations Hence, the permissiveness of this threshold creates a tradeoff between accuracy and inference speed. Finally, self-loop edges and proximity edges are differentiated using a one-hot encoded feature vector such that esst = [0, 1)" and e, [1, 0)", where eat and ep represent the feature vectors for self-loop edge and a proximity edge respectively. The obtained graph is given as input to GRN, shown in Figure 2 which is comprised three modules.
4.2 INVERSE KINEMATICS FEASIBILITY PREDICTION
The first module is the Inverse Kinematics (IK) feasibility prediction module. This submodel is binary classifer, composed of a 4-layer Multi-Layer Perceptron (MLP) with ReLU activation func- tions, followed by Sigmoid activation function. It takes input the feature vector x, of each node u E corresponding to a movable object, and simultaneously outputs the predicted inverse kinematics feasibility for each of the five grasp types in G:
Kig (u)
(MI.
E [0, 1|5
Each prediction Kig where g corresponds to the presence of valid inverse kinematics solution for at least one grasp in g. This can be viewed as predicting whether the robot can reach a specific side of the object its pose in the environment, without taking into account any other object. Since this information is valuable for action and grasp types feasibility prediction, we incorporate it to the previously constructed graph by concatenating the obtained predictions Kg to the features of self-loop edges resulting in ex)
[est || Kg] E RR,, where denotes the concatenation operator.
4.3 GRASP OBSTRUCTION ESTIMATION
The second stage is Grasp Obstruction (GO) estimation, which regression module consisting of 4-yayer MLP with ReLu activation functions, followed by HardTTanh activation function which bounds the output between and 1. It takes input the concatenated features of each pair of nodes (v + u) linked through a proximity edge. For each grasp type g E G, it outputs the estimated ratio of grasps of u that are obstructed by the object corresponding to v, which is defined as the number of obstructed grasps divided by the total number of grasps of specific type:
Pg (u, v) = HardTanh HarddTanhMMLPcoo//
xx,))) [0,
(4)
This submodel not only predicts whether object obstructs grasps of another, it also gives insight into how much it blocks access to it. In a manipulation planning context, this allows to rank obstruct- ing objects and plan accordingly. Similarly to IK feasibility predictions, this information impacts directly the feasibility of actions and grasp types. Thus, we concatenate the grasp obstruction esti- mations Pg the edge features of proximity edges obtain ej (ep PG] R..
5

---

Published as a conference paper at ICLR 2025
4.4 ACTION AND GRASP FEASIBILITY PREDICTION
Once our graph is constructed and enriched through IK feasibility and GO predictions, it is given to the action and grasp feasibility (AGF) prediction module. As mentioned previously, IK feasibility and grasp obstructions, which constitute our edge features, have a direct impact on action and grasp type feasibility. However, the classic graph attention network (GAT) (Veličković al. (018)) uses edge features simply for computing attention weights, and does not take them into account during the aggregation process. This causes the model to lose valuable information when applied to tasks where edge features are as important as node features. In this work, we propose to use the Edge- Featured Graph Attention Network (EGAT) (Wang et al., 2021) which, in addition to using edge features compute attention, leverages both node and edge features during the aggregation process.
The third submodel first computes embeddings h. and euv of node and edge features using two fully-connected layers such that hn Ww. and euv We.e where Wh. We are weight matrices. Then, EGAT computes the multi-head attention each edge as:
Lsppa'LeayyReLUU((... h, (i.l))) || h ()))
Ouu [kEN(W)U(u)
(5)
Note that this formulation is slightly different from the proposed by Wang et al. (2021). We adapt
the attention computation introduced by |Boody et al. (2022). which fixes the static attention problem of the standard GAT. Once the multi-attention weights computed, our model computes a weighted average of the concatenated node and edge embeddings, followed by LeakyReLU activation:
h', LeakyReLUU
Σ
Our [he eurl
"EA(c)U(u)
Finally, the obtained vector is passed through a 2-layer MLP followed by sigmoid activation func- tion that outputs the action and grasp types feasibility predictions for movable object node u:
(u) Fg(u)
Sig
4.5 TRAINING STRATEGY
Our proposed GRN model is trained in a supervised manner by first pre-training each module sep- arately. The IK feasibility prediction submodel is trained using a binary cross entropy loss denoted CIK while the GO estimation module trained using the mean square error loss Lgo- Regarding the AGF prediction module, it is trained using binary cross entropy loss Lp. using the ground truth of IK feasibility and grasp obstructions as edge features. The complete GRN model is then fine-tuned using a weighted sum of the previously defined losses such that:
L = LF + LIk + n.Lo0
(8)
where 17 is a weighting factor allowing a balanced order of magnitude across classification and regression. This training strategy is inspired by the one proposed by Chen et al. (2020).
During training, our 3D scene representation allows two data augmentation methods: dimensions switch and rotation. They both take advantage of the symmetry of bounding boxes. Indeed, switch- ing the length and width of bounding box, then applying a 3 rotation around the z axis, keeps the geometric properties of the environment unchanged, resulting in a new node feature vector x = [w,, 1, h, x. y, z, + TJ. The same goes for applying a æ rotation without any dimensions switch such that æ = [r, w, h, x, y. 2, 0 + ./7" These can be applied to fixed and movable objects. For the latter case, the labels associated with grasp types need to be interchanged, since switching the dimensions and/or rotating the object changes which side is considered as the front for example.
5 EXPERIMENTS
We conduct a series of experiments in order to evaluate the performance of our proposed method compared to existing approaches, and showcase the generalization capabilities of our approach. We

---

Published as a conference paper at ICLR 2025
conduct our experiments using mainly the Franka Emika Panda, which is degrees-of of-freedom (DOF) robotic arm with a parallel jaw gripper. Since our model is robot-centric, meaning that is specific the robot it is trained for, showcase the applicability of our method to other robotic manipulators by running experiments on the Willow Garage PR2 robot as well. The latter is a dual-arm robot where each arm has DOFs and parallel jaw gripper, and telescoping spine.
5.1 DATASETS
Our model is trained and evaluated on fully synthetic data. We generate a number of datasets follow- ing the method described in Appendix B Each one characterized by a number of movable objects as well as minimum and maximum numbers of structures (e.g, rack, counter, basket) and obstacles. They are also characterized by the robot used during data annotation.
Panda-3D-4: This is dataset is composed of 3D environments containing 4 movable objects, 1 to 4 structures and 0 to 4 obstacles and annotated using a Panda robot.
Panda-Tabletop-4: In order to conduct fair comparison to tabletop methods Wells et al. (2019);
Driess et al. (2020b), we generate dataset consisting of tabletop environments with O structures, movable objects and up to 4 obstacles, all placed on the same support surface as the robot's 's base.
PR2-3D-4: We generate this dataset to showcase the applicability of our approach to other robotic arms as well multi-robot settings. It is generated using the same parameters as the Panda-3D dataset, and annotated for the right arm of PR2 robo I We consider that the base of robot is fixed to the ground, and that the telescopic spine is one of the DOFs of the arms.
In order to quantitatively measure the generalizability of our approach to environments containing a higher number of objects than training environments, generate three additional test sets denoted Panda-3D-10, Panda-3D-15, Panda-3D-20, each composed of 1'000 environments containing 10, 15 and 20 movable objects respectively. These environments also contain a higher number of fixed objects, with a number of structures ranging from to 8, as well as 2 to 4 obstacles. Moreover, we increase the dimensions" range used during data generation resulting in larger objects.
5.2 BASELINES
We compare our proposed approach to multiple baselines and methods proposed or adapted from previous works on action and grasp feasibility prediction for manipulation planning.
MLP: This is simple baseline which uses 4-layer MLP that takes as input the feature vector æ of an object to predict action and grasp feasibility, without considering the rest of the environment.
Feasibility-SVM (F-SVM): Introduced by Wells et al. (001)), this method uses multiple SVMs to predict the action and grasp types feasibility prediction tabletop environments containing movable objects represented using hand-crafted feature vectors.
Deep Visual Heuristics (DVH): This method, proposed by Driess et al. (2020b). represents envi- ronments using top-view depth images, then uses a CNN to predict the feasibility of an action using grasp type. We adapt DVH to output the feasibility of the action and all grasp types simultancously.
Action and Grasp Feasibility Prediction Network (AGFP-Net): Bouhsain et al. (203b)) propose this method as an extension of DVH to 3D environments, using 5 depth images corresponding to different views of the scene as input to the CNN.
Feasibility GATT (F-GAT); This baseline is an adapted version of the methods proposed by Silver et al. (2021) and Khodeir et al. (2023a b). It represents environments as graphs where nodes repre- sent objects, and edges represent symbolic relationships (c.g object on table). Nodes features are the dimensions and pose of objects and edge features are one-hot-encodings of the different relationship types. GAT is then used to predict action and grasp feasibility.
Teasibility-GCC (F-GCN): This baseline uses the same scene representation F-GAT, except that GAT is replaced with Graph Convolution Network (GCN), which does not use edge features.
As explained is Section 4.1 model can he applied to different arms of the same type simply by ex- pressing the objects' poses in the frame of reference of the considered arm.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with SOTA methods trained and tested on different datasets. For grasp types
feasibility prediction, the mean (# standard deviation) of F1 scores of the grasp types are reported.
Datasel
Panda-3D-4
Vanda-Tabletop-
PR2-3D-4
Task
Action (F
Grasp (FT)
Action (FI)
Grasp (FT)
Action (FT)
Grasp (F1)
F-SVM
0.884
0.415 0.220)
MLP
0.784
0.558 0.089)
0.911
0.696 (# 0.121)
0.750
0.574 (#: 0.104)
DVH
0.840
0.718 0.108)
0.961
0.865 (= 0.073)
0.808
0.622 (#: 0.179)
AGFPNet
0.882
0.806 0.065)
0.964
0,916 (1: 0.032)
0.836
0.655 (#: 0.238)
F-GCN
0.836
0.721 (+ 0.057)
0.955
0,879 (+ 0.042)
0.791
0.680 (+ 0.075)
F-GAT
0.867
0.796 (+: 0.052)
0.961
0,904 (+: 0.034)
0.827
0.764 (+: 0.061)
GRN (Ours)
0.939
0.940 (= 0.009)
0.976
0,976 (= 0.004)
0,908
0.903 (#: 0,013)
5.3 EVALUATION METRICS
We use the F1 score the evaluation metrie for classification tasks, namely action, grasp types and IK feasibility predictions. For or Grasp Obstruction (GO) predictions, we use the Mean Absolute Error measure performance. For clarity, we report the mean and standard deviation of predictions across the different grasp types.
RESULTS
6.1 COMPARISON TO PRIOR WORK
Table T shows that our proposed model outperforms all prior works both action feasibility and grasp types feasibility predictions, and on all datasets. CNN-based methods, DVH and AGFP-Net, fall short compared our approach, with difference in F1 score the Panda-3D-4 of 10% (resp- 5.7%) for action feasibility prediction, and 22.6% (resp. 13.7%), for grasp type feasibility predic- tion. Indeed, image-based scene representation suffers from occlusions due to the 3D nature of the environment, resulting in inaccurate predictions for occluded objects. GNN-based methods, on the other hand, represent 3D environments using scene graphs. However, the performance of our ap- proach compared to F-GCN and F-GAT shows that careful design of the graph and its connectivity needed. (Silver et al. (202T) and Khodeir et al. (2023a]b) b) connect nodes using symbolic facts. Our method uses geometric relationships between objects to connect the graph. This allows our model to achieve an score up to 10.3% higher than other GNN-based baselines on action feasibility prediction, and up to 21.8% higher grasp types feasibility prediction Panda-3D-4.
Comparing the standard deviations across Fl scores of each grasp type shows that our proposed method has more consistent performance across the different grasp types than other models. This due to our data augmentation method and the two interpretation mechanisms. Switching the di- mensions then/or rotating object implies interchanging grasp types annotations, which ensures balanced labels across grasp types. This method is not applicable to CNN-based methods 2 Addi- tionally, the use of IK feasibility predictions and GO estimations edge features allows each grasp type feasibility prediction to be informed, which is not the case for other GNN-based methods.
In robotic manipulation planning, feasibility prediction must not only be accurate, must also have low inference time and memory footprint. Sable@reeort the number of param- eters in previous models compared to GRN, well as the inference time of each model on 3D scene containing Panda robot, 4 movable ob- jects and 15 fixed object. The inference time in- corporates the complete prediction process from the model's input construction to the output, for each movable object in the environment. For reference, we also report the planning time of an off the shelf geometric planner.
Table Comparison the number of parameters and inference time on 3D environment with movable objects and 15 fixed objects (4 queries).
Inference time (ms)
Nb
Model
Parameters
Geometric planner
1500
MI.P
0.6
269'830
DVH
25
11'225'282
AGFPNet
150
347585'350
F-GCN
4.25
1'057'798
F-GAT
5.0
2'636'800 806
GRN (Ours)
5.5
2'259'472
Switching the dimensions then rotating an object results in the input images.
8

---

Published as a conference paper at ICLR 2025
Table Ablation Study on the Panda-3D-4 dataset. For each task related to grasp types, we report the mean (#t standard deviation) across all grasp types.
Task
Action (F1)
Grasp (FI)
IK (FI)
GO (MAE)
w/o IK, wo GO
0.868
0,811 (# 0.043)
w/o GO
0.872
0,811 (# 0,046)
0.995 (# 0.001)
w/o IK
0.937
0.933 (+ 0.011)
0.029 (+ 0.003)
Full model w/ GAT
0.928
0,924 (# 0.013)
0.995 (1: 0.001)
0.029 (#: 0.003)
Full model w/o data aug.
0..15
0.903 (+ 0.013)
0,990 (+ 0.001)
0.044 (+ 0.002)
Full model (Trained from scratch)
0,932
0.925 0.011)
0,994 (1: 0.001)
0.038 (#: 0.002)
Full model (Ours)
0.939
0.939 (# 0.009)
0.995 (+ 0.001)
0.028 (# 0.003)
Results show that, compared to previous works, our method yields the most accurate predictions while being one of the models with the lowest inference times and memory footprints. Furthermore, GRN has a 99.6% lower inference time than traditional geometric planning. In an offline manipula- tion planning context, where the number of feasibility checks can reach tens of thousands of queries, this difference in computational cost can significantly reduce planning time.
6.2 ABLATION STUDY
In order to justify the choices behind our neural network architecture and training strategy, we duct an ablation study on the Panda-3D-4 dataset. Results reported in Table showcase the im- portance of the proposed interpretation modules. Our full model shows 7.1% % gain in performance compared to the one without IK feasibility and GO predictions. A more in-depth analysis shows that the grasp obstruction estimation module the most important, while IK feasibility prediction yields a slight improvement in performance. Moreover, the improved performance obtained using EGAT instead of classical GAT Veličković et al. (018)) shows that incorporating edge features in the atten- tion computation and the aggregation process helps action and grasp types feasibility prediction. We conduct two ablations to demonstrate the effectiveness our training strategy. Training the model without the proposed data augmentation method, yields lower performance all tasks, particu- larly on grasp types feasibility prediction and GO estimation with 3.6% difference in F1 score, and 1.6 % in MAE. Finally, training the full model from scratch, rather than rre-training each module before fine-tuning the complete network, yields a slightly lower performance across all tasks.
6.3 GENERALIZABILITY EVALUATION
Applicability to other robots. Table shows our model's performance on the PR2-3D-4 compared to prior methods. The results show that GRN achieves better performance than the state-of-the of-thh-art on robots with various kinematics. A consistent decrease in performance can be noticed across all methods (ours included), compared when trained on the Panda-3D-4 dataset. This due the smaller number of training data of the PR2 dataset and the harder kinematics of the PR2 robot.
Generalizability to more complex environments. Table 4 reports the performance of our method and the baselines on the Panda-3D-10 Panda-3D-15 and Panda-3D-20 test sets, when trained on the Panda-3D-4 dataset. Results show that, although there is a decrease in Fl-scores compared to the one obtained on Panda-3D-4, GRN maintains a good performance on 3D environments with a higher number of fixed and movable objects, with an Fl FI-score on Panda-3D-20 of 0.89 for action feasibility prediction, and 0. 0.903 for grasp types feasibility prediction. Particularly, our model shows a better generalization capability on the latter than previous CNN-based or GNN-based methods.
6.4 APPLICATION TO TASK AND MOTION PLANNING
Although the integration of our method into task and motion planner outside the scope this paper, we develop a simple single-shot planner (cf. Appendix C. to showcase the power of GRN's predictions and its interpretation mechanisms, as well as the planning performance gain yielded by approach. From single query to GRN, it proposes geometrically feasible plan to class of manipulation planning problems, where the goal is to move single object initially placed in complex setting. We compare this algorithm to the non-informed TAMP planner proposed by

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation of the generalizability 3D environments with a higher number of objects
compared to SOTA methods, when trained on the Panda-3D-4 dataset.
Test set
Pan1a-3D-10
Panda-3D-15
Panda-3D-20
Task
Action (F1)
Grasp (F1)
Action (F'I)
Grasp (F1)
Action (F1)
Grasp (F1)
MLP
0.773
0.624 0.028)
0.766
0.616 (#: 0.046)
0.768
0.609 (#: 0.047)
DVH
0.820
0.697 0.115)
0.819
0.686 (#: 0.127)
0.825
0.676 (:: 0.140)
AGFPNet
0.858
0.770 0.079)
0.862
0.768 (1: 0.078)
0.864
0.755 (1: 0.087)
F-GCN
0.794
0.633 0.036)
0.771
0.595 (+ 0.044)
0.764
0.565 0.055)
F-GAT
0.829
0.738 0.026)
0.826
0.725 (#: 0.032)
0.825
0.715 (#: 0.038)
GRN (Ours)
0.894
0.909 (+: 0.012)
0.891
0.906 (+: 0.013)
0.890
0.903 (+: 0.012)
Table 5: Performance of GRN planner compared to a non-informed planner on the Access and Clutter problems. Results are average over 10 runs on 10 different instances of each problem.
Success
Planning time (s)
Nb Geometric
Problem
Method
Rate ( (%)
Plamner Calls
Bouhsain et al. 20244
100%
26.5
41.1
Access
GRN planner
90%
3.17
Bouhsain et al. 2024
100%
558.9
89.5
Clutter
GRN planner
100%
14.8
7.2
Bouhsain et al. (024)) on two different problems shown in Figure The first is the Access problem, where single Panda robot has to move small bottle, to which number of fixed and movable objects block access. The second a multi-robot Clutter problem, in which both arms of the PR2 robot collaborate pick an object surrounded by grasp-obstructing objects. Objects used in these problems are a mix of box-shaped and mesh objects. Jshooss that despite its simplicity, the GRN-based planner not only achieves 90% and 100% success rate but also reduces planning time by 88% on the Access problem, and 97% on the Clutter problem. This is possible thanks to the two interpretation mechanisms, which allow the planner to reason over why an action or a grasp type is infeasible, and recursively decide which objects should be moved to rectify that. We also test GRN planner in real-world setups, on both Panda and a PR2 robots. Objects' poses are estimated using separate perception module. The proposed model is able to accurately predict action and grasp feasibility, as well as reasons of infeasibility, from estimated objects' poses in both setups, allowing the planner to compute feasible solutions that are successfully executed by the robots.
DISCUSSION AND FUTURE WORK
In this work, we propose a framework for action and grasp feasibility prediction 3D environments. Leveraging GNN-based neural network and two interpretation mechanisms, our model predicts the fcasibility of pick or place actions, different grasp types, and the infeasibility cause for each grasp type from a scene graph representation. Results demonstrate that our approach outperforms state- of-the-art methods, generalizing better to complex environments and robots. Additionally, through IK feasibility and grasp obstruction predictions, class of manipulation planning problems can be solved with single query to our model, significantly reducing planning time compared to traditional TAMP planners. Our design also enables straightforward extensions to diverse object shapes via bounding boxes and multi-robot feasibility predictions due to the robot-centric nature of GRN.
Currently, our method is limited to infeasibility caused by missing IK solutions or grasp obstructions, and it struggles with problems stemming from infeasible robot motion. Future work will include graph pooling layers to evaluate motion infeasibility across the entire scene graph. Additionally, grasp obstructions are currently estimated as blocked ratio, which does not indicate which grasps are obstructed. Representing obstructions as blocked object regions could improve interpretability. While bounding boxes work well for common objects, they face challenges with large or complex shapes. A potential extension is using multi- boutdiibundig box representations, treating sub-boxes as separate nodes. Alternatively, embeddings from off- fff-th shape encoders (e.g., meshes point clouds) could enhance node features, though computational efficiency remains a challenge. Finally, also aim at integrating our model into more sophisticated TAMP algorithm.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partially supported by the EU-funded project euROBIN under grant agreement no. 101070596 (https: sttps://wet //www//ww......//////////////.// werrbiiinn n-pro eect. We also would like thank Baptiste Merlau for his help with the real-world experiments.
REFERENCES
Christopher Agia, Toki Migimatsu, Jiajun Wu, and Jeannette Bohg. Stap: Sequencing task-agnostic policies. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7951- 7958. IEEE, 2023.
Juan Manuel Ahuactzin, Kamal Gupta, and Emmanuel Mazer. Manipulation planning for redundant robots: practical approach. The International Journal of Robotics Research, 1998.
Rachid Alami, Thierry Simeon, and Jean-Paul Laumond. A geometrical approach to planning ma-
nipulation tasks. the case of discrete placements and grasps. In The fifth international symposium on Robotics research, 1990.
Smail Ait Bouhsain, Rachid Alami, and Thierry Simeon. Learning to predict action feasibility for task and motion planning in 3d environments. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023a.
Smail Ait Bouhsain, Rachid Alami, and Thierry Simeon. Simultaneous action and grasp feasibility prediction for task and motion planning through multi-task learning. In 2023 IEEE/RSJ Interna- fional Conference on Intelligent Robots and Systems (IROS), 2023b.
Smail Ait Bouhsain, Rachid Alami, and Thierry Simeon. Extending task and motion planning with feasibility prediction: Towards multi-robot manipulation planning of realistic objects. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In Interna- tional Conference on Learning Representations, 2022.
Stephane Cambon, Rachid Alami, and Fabien Gravot. hybrid approach to intricate motion, ma- nipulation and task planning. The International Journal of Robotics Research, 2009.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Con- ference on Robot Learning, pp. 66-75. PMLR, 2020.
Haonan Chen, Yilong Niu, Kaiwen Hong, Shuijing Liu, Yixuan Wang. Yunzhu Li, and Kather- ine Rose Driggs-Camplln Campbell. Predicting object interactions with behavior primitives: An applica- tion in stowing tasks. 7th Annual Conference on Robot Learning. 2023.
Rohan Chitnis, Dylan Hadfield- Hadeield-Menell, Abhishek Gupta, Siddharth Srivastava, Edward Groshev, Christopher Lin, and Pieter Abbeel. Guided search for task and motion plans using learned heuris- tics. In 2016 IEEE International Conference on Robotics and Automation (ICRA), 2016.
Neil T Dantam, Zachary K Kingston, Swarat Chaudhuri, and Lydia E Kavraki. Incremental task and motion planning: A constraint-basee approach. In Robotics: Science and systems, volume 12, pp. 00052. Ann Arbor, MI, USA, 2016.
Didier Devaurs, Thierry Siméon, and Juan Cortés. Enhancing the transition-based rt to deal with complex cost spaces. In 2013 IEEE international conference on robotics and automation, pp- 4120-4125. IEEE, 2013.
Danny Driess, Jung-Su Ha, and Marc Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. In Robotics: Science and Systems 2020, 2020a.
Danny Driess, Ozgur Oguz, Jung-Sh Su Ha, and Mare Toussaint. Deep visual heuristics: Learning feasibility of mixed integer programs for manipulation planning. In 2020 IEEE International Conference Robotics and Automation (ICRA), 2020b.

---

Published as a conference paper at ICLR 2025
Ben Eisner, Yi Yang, Todor Davchev, Mel Vecerik, Jonathan Scholz, and David Held. Deep se (3)equivariaan geometric reasoning for precise placement tasks. In The Twelfth International Conference on Learning Representations, 2024.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint AXXiv:9033022443 2019.
Caelan Reed Garrett, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Ffrob: Leveraging symbolic planning for efficient task and motion planning. The International Journal of Robotics Research, 2018.
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4(1) ((1):265-293) 2021.
Michael Görner, Robert Haschke, Helge Ritter, and Jianwei Zhang. Moveit! task constructor for task-level motion planning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 190-196. IEEE, 2019.
Dan Haramati, Tal Daniel, and Aviv Tamar. Entity-centric reinforcement learning for object ma- nipulation from pixels. In The Twelfth International Conference on Learning Representations, 2024.
Kris Hauser and Jean lean-Claude Latombe. Multi-modal motion planning in non-expansive spaces. The International Journal of Robotics Research, 2010.
Kris Hauser and Victor Ng-Thow-Hing Randomized multi-modal motion planning for a humanoid robot manipulation task. The International Journal of Robotics Rexearch, 2011.
Haojie Huang, Owen Lewis Howell, Dian Wang. Xupeng Zhu, Robert Platt, and Robin Walters. Fourier transporter: Bi-equivariant robotic manipulation in 3d. In The Twelfth International Con- ference Learning Representations, 2024a.
Yixuan Huang, Adam Conkey, and Tucker Hermans. Planning for multi-object manipulation with graph neural network relational classifiers. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1822-1823 IEEE, 2023.
Yixuan Huang, Christopher Agia, Jimmy Wu, Tucker Hermans, and Jeannette Bohg. Points2plans: From point clouds to long-horizon plans with composable relational dynamics. arXiv preprint arXiv:2408. 14769, 2024b.
Mohamed Khodeir, Ben Agro, and Florian Shkurti. Learning to search in task and motion planning with streams. IEEE Robotics and Automation Letters, 2023a.
Mohamed Khodeir, Atharv Sonwane, Ruthrash Hari, and Florian Shkurti. Policy-guided lazy search with feedback for task and motion planning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3743-3749. IEEE, 2023b.
Beomjoon Kim, Luke Shimanuki, Leslie Pack Kaelbling, and Tomás Lozano- Pérez. Representation, learning, and planning algorithms for geometric task and motion planning. The International Journal of Robotics Research, 2022.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412,6988 2014.
Yoshihito Koga and J-C Latombe. On multi-arm manipulation planning. In 1994 IEEE International Conference on Robotics and Automation, 1994.
Fabien Lagriffoul, Dimitar Dimitrov, Julien Bidot, Alessandro Saffiotti, and Lars Karlsson. Ef- ficiently combining task and motion planning using geometric constraints. The International Journal of Robotics Research, 2014.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345-1365, 2023.
12

---

Published as a conference paper at ICLR 2025
Yixin Lin, Austin S Wang, Eric Undersander, and Akshara Rai. Efficient and interpretable robot manipulation with graph neural networks. IEEE Robotics and Automation Letters, 7(2):2740- 2747, 2022.
Michael James McDonald and Dylan Hadfield-Menell. Guided imitation of task and motion plan- ning. In Conference on Robot Learning, 2022.
Jia Pan, Sachin Chitta, and Dinesh Manocha. Fel: A general purpose library for collision and proximity queries. In 2012 IEEE International Conference on Robotics and Automation, pp. 3859-3866. IEEE, 2012.
Suhan Park, Hyoung Cheol Kim, Jiyeong Back, and Jacheung Park. Scalable learned geometric feasibility for cooperative grasp and motion planning. IEEE Robotics and Automation Letters, 2022.
Guillaume Sarthou. Overworld: Assessing the geometry of the world for human-robot interaction. IEEE Robotics and Automation Letters, 83)):1774-1890, 2023.
Tom Silver, Rohan Chitnis, Aidan Curtis, Joshua Tenenbaum, Tomás Lozano-Péree Pérez, and Leslie Pack Kaelbling. Planning with learned object importance in large problem instances using graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2021.
Thierry Siméon, Jean-Pate Laumond, Juan Cortés, and Anis Sahbani. Manipulation planning with probabillistic roadmaps. The International Journal of Robotics Research, 2004.
Siddharth Srivastava, Eugene Fang, Lorenzo Riano, Rohan Chitnis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible planner-independent inter- face layer. In 2014 IEEE international conference on robotics and automation (ICRA), 2014.
Yoonchang Sung, Zizhao Wang, and Peter Stone. Learning to correct mistakes: Backjumping in long-horizon task and motion planning. In Conference on Robot Learning, pp. 2115-2124. PMLR, 2023.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Zhu, and Hangxin Liu. Llm" 3: Large language model-based task and motion planning with motion failure rea- soning. Multi-moda Foundation Model meets Embodied Al Workshop in the 41 st International Conference on Machine Learning, 2024.
Ziming Wang. Jun Chen, and Haopeng Chen. Egat: Edge-featured graph attention network. In Artificial Neural Networks and Machine Learning-ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part I 30, pp. 253-264. Springer, 2021.
Andrew M Wells, Neil T Dantam, Anshumali Shrivastava, and Lydia E Kavraki. Learning feasibility for task and motion planning in tabletop environments. IEEE robotics and automation letters, 2019.
Danfei Xu, Ajay Mandlekar, Roberto Martín-Martin Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Deep affordance foresight: Planning through what can be done in the future. In 2021 IEEE Internutional Conference Robotics and Automation (ICRA), 2021.
Lei Xu, Tianyu Ren, Georgia Chalvatzaki, and Jan Peters. Accelerating integrated task and motion planning with neural feasibility checking. arXiv preprint arXiv:2203. 105688 2022.
Zhutian Yang, Caclan Reed Garrett, and Dieter Fox. Sequence- based plan feasibility prediction for efficient task and motion planning. arXiv preprint arXiv:2211.01577 2022.
Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, and Yuke Zhu. Hierarchical planning for long- horizon manipulation with geometric and symbolic scene graphs. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6541-6548 IEFF, 2021.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Scene Graph Construction. Scene graphs are built using the graph representation from Pytorch Geometric (Fey & Lenssen 2019). Nodes correspond to fixed and movable objects from the en- vironment. Self-edge loops are added for nodes corresponding to movable objects. When adding proximity edges, since evaluating the distance between two objects closest points can be computa- tionally expensive, we consider two objects O and O' as neighbors if the euclidean distance between their centers is lower than a threshold c ro + rox + K, where ro and ror are the bounding cylin- der radii of O and ('' respectively, and K is constant set to 0.6. This constant is chosen twice the length of the last link of the robot. Using this definition, the neighborhood of an object is easy to compute and captures enough of its surroundings for accurate action/grasp feasibility prediction.
Model Architecture. The IK feasibility prediction and GO estimation modules are 4-Layer MLPs with hidden size of 512. The AGF module, on the other hand, has a hidden size of 256, attention heads and one pressage-passingg step. Exceptionally, when training on the PR2-3D-4 dataset, we use hidden size of 256 for the GO module it yields better results.
Training Details. The three modules are implemented in Pytorch Geometric (Fey & Lenssen 2019) and trained using the Adam optimizer (Kingmal 2014).. During the pre-training stage, each module is trained for 100 epochs. We use batch size of 8192 and a learning rate of 0.001 for the IK feasibility classifier and the GO estimator. For AGF classifier, we set the batch size to 2048 and the learning rate to 0.0001. During the fine-tuning stage, the complete GRN model is trained for 100 epochs with a batch size of 2048 and learning rate of 0.0001. The model is trained on an Intel(R) (R) W-2223 CPU @ 3.60GHz workstation, with an NVIDIA RTX A5000 GPU. The full training process takes approximately hours.
Inference Time Decomposition. Table 6 shows the inference time decomposition of our proposed model. The total inference time of GRN is 5.5 ms in average, with the most significant portion spent on scene graph construction with an average time cost of 3 ms. During this step, time is mostly spent on evaluating the distance between pairs of objects add proximity edges between their correspond- ing nodes. The IK feasibility and GO modules take each computation time of 0.5 ms. Finally, the AGF module takes 1.5 ms output the action and grasp feasibility prediction modules.
Table 6: Decomposition of the inference time of GRN on a 3D environment with 4 mov- able objects and 15 fixed objects.
Step
Inference Time
Scene Graph Construction
3 ms
IK Feasibility Module
0.5
GO Module
0.5 ms
AGF Module
1.5 ms
Total
5.5 ms
Rack
Bar
Counter
Basket
Figure 3: Visualization of the different types of structures used during data generation.
14

---

Published as a conference paper at ICLR 2025
B DATA GENERATION AND ANNOTATION METHOD
Data Generation. Our data generation process consists of generating a number of 3D environments containing a random number of support surfaces, movable objects and obstacles within a specified range. We define types of structures shown in Figure (1) a rack which is structure containing a varying number shelves separated by a gap, and a holder on each side or each corner, (2) a bar which is a L--saped structure with support surface and one holder, (3) basket which consists of a support surface and sides surrounding it, and (4) a counter which generating an environment, we first randomly sample number of structures within the specified range. We then choose one of the predefined structures, before randomly sampling its dimensions and pose. After collision checking using the FCL library (Pan al.) 2012). all objects composing the structure are added to the environment as fixed objects. These can be either support surfaces or obstacles (e.g holders). This process is repeated until the number of structures sampled is reached.
large block. When
Given the fact that our approach represents objects using their bounding boxes, it is sufficient to gen- erate dataset consisting of box-shaped objects only. After sampling number of movable objects to add, we randomly sample the dimensions of the object with a specified range. In order to incorpo- rate difficulties encountered in real-world scenarios, we define three methods for object placement sampling: (1) the first is random placement in which support surface and a pose within its bounds are randomly sampled, (2) proximity placement in which the object is placed in the neighborhood of randomly chosen object, (3) the third is the underneath placement in which the object is placed underneath randomly chosen support surface. We check the sampled object placement for colli- sions before adding it to the environment. We then repeat the process until the sampled number of movable objects reached, Finally, a number of fixed box-shaped obstacles is randomly added to the scene, before storing it.
Figure 4: Visualization of generated environments from the (top) Panda-3D-4, (middle) Panda-3D- 20, and (bottom) PR2-3D-4 test sets
15

---

Published as a conference paper at ICLR 2025
Using this method, generate the datasets presented in Section 5.1 The Panda-3D-4 dataset consists of training set containing 70 000 scenes, a validation set of IO 000 scenes, and a test set of 20'000 scenes, each generated using different random seed to ensure that the environments different across all three sets. The Panda- Tabletop-4 and PR2-3D-4 are each composed of 25'000 training scene, 5' 000 validation scenes and 10'000 test scenes. Figure A shows environments from different datasets.
Data Annotation. Data annotation can be done using any off-the-shelf geometric planner. In this work, we use an adapted version of Moveit Task Constructor (Görner et al. 2019), with the KDL plu- gin for IK computation. FCL (Pan al. 2012) for collision checking and, Bidirectional Transition- Based Rapid Random Tree (Devaurs et al. 2013) for motion planning. Since pick and place actions are symmetrical, they can be considered as equivalent. Thus, we only annotate pick actions. Also, we do not focus trivial infeasibility cases due to the object being larger than the gripper. Hence, during annotation, we allow collisions between the object to pick and the gripper fingers. This also allows the handling of mesh objects such mug wine glass, for which feasible grasps exist even the bounding box larger than the gripper's ' maximum width.
For each movable object in generated environment, query the geometric planner to plan pick action from its placement in the environment. First, we uniformly sample a number of grasps depending the size of the object. For each sampled grasp, which are then checked for collisions. Finally, the motion planner is queried for each collision-fie IK solution until feasible collision-fres trajectory is found. If solution is found, the action is annotated as feasible. In parallel, we set the feasibility label of each grasp type E G to 1 if at least one grasp in g feasible. Similarly, grasp belonging to it has a valid inverse kinematics solution. Finally, we record all grasp-obstructing objects well the ratio of grasps obstructed per grasp type. Data annotation takes approximately 68 hours to complete on the Panda-3D-4 dataset.
compute up to 8 IK solution,
the IK feasibility label of a grasp type
if at least one
Figures JanurresJand@&Shoov the distributions of obtained labels for the Panda-3D-4 and PR2-3D-4 training sets. For the former, the action feasibility annotations are balanced, while grasp types feasibility imbalanced with more infeasible cases. Regarding reasons of infeasibility, Figure [5c] shows balanced distribution between failures due IK feasibility and grasp obstructions. Cases where
120000
Numberr of datpooma
1B0000
B0000
60000
S00t000
Grasp
20000
Feasble
Infensible
Mobion Panning Fatlure
Fallure
Pesslble
Infeasible
1bp
Front
Pear
Rith
Laft
Ta
hrnnt
Bear
Righ
Left
Action Feasibilly
typa
Grasp typ
(a) Number of feasible and infeasi- ble actions
(b) Number of feasible and infeasi-
(c) Distribution of failure causes
ble per grasp type
per grasp type
Figure 5: Annotations statistics for the Panda-3D-4 training set.
60000
100000
Carpopts
Number of datapoinda
90000
60000
3
40000
Mocion Plarni Fallure
Pallure
20000
40000
10130
Infeadible
20000
Fess Bie
Foex/Ole
Infews Die
Tront
Left
Tront
Stanet
Acuon Imesibiily
Ureap sype
Lihatio bypwe
(a) Number of feasible and infeasi- ble actions
(b) Number feasible and infeasi- cases per grasp type
(c) Distribution of failure causes
per grasp type
Figure 6: Annotations statistics for the PR2-3D-4 training set.
16

---

Published as a conference paper at ICLR 2025
infeasibility is due to motion planning failure, the other hand, appear rarely in the dataset. The PR2-3D-4 dataset has a more pronounced imbalance towards infeasibility, the latter being caused more often by the absence of inverse kinematics solutions than grasp obstructions, showcasing the more complex kinematics of the PR2 robot.
C APPLICATION TO TAMP
C.I GRN PLANNING ALGORITHM
Algorithm D shows the pseudo-code for the GRN-based planner. Given the GRN predictions on the initial state of the environment, feasibility threshold and grasp obstructions threshold, it recursively builds task plan by moving objects when it feasible, or trying to free access to grasp types based grasp obstruction and IK feasibility information otherwise. Once task plan is found, we query the geometric planner used during data annotation to verify its geometric feasibility, and plan the corresponding robot motions. For simplicity, Algorithm [ shows the planner used for the single robot problem only. For the multi-robot Clutter problem using both arms of the PR2 robot, choose the robot with the higher feasibility prediction execute feasible actions. If an action is infeasible for both arms, we consider both of them when trying to free grasp types.
Algorithm GRNPlanner
Input: Fa, Fo, Kg: Ps. Teo, O
GRN predictions, feasibility threshold, grasp obstrnction threshold, desired object
1: plan f-
2: if FO)) >
3: feasibleGrasps
4: for each G such F((O) > TF
5:
feasibleGrasps feasibleGrasps Ug
6: end for
7:
8:
return plan
9:
else
10:
Blockere PerGruspT upe
Vy
11: (reeableGraspl) Grasp'lypps ypes +-
12:
for each G
13:
if (O)
(O) < Tp then freeahleGrayyTyps
14:
freeableGraepTTape
15:
else
16:
for O' (O) such that (0, Teo do
17:
O' is fixed then freeableGraepT upes + /reeableGraspTyppe \.8 break
18:
19:
20:
else
21:
end BlockersPerrraapee upelgi BlockersPerGrospll ypely) of
22:
23:
24:
end if
25:
end for
26:
if freeableGraspT ypes empty then
27:
return
28:
29:
for each E frocableGraspiTTpa do
30:
subplan -
31:
grasy/Type ypeeyppeeeerr Freed +- True for each O' E BBoodeersPerrrraapppppp do
32:
33:
result +- GRN Planner (F,, (F., Fe, Po. Teo, O*)
34:
if result is empty then
35:
grasp/ parssppyyerrrree Freed False break
36:
37:
else
38:
subplan subptan.appendd(yer append(reeutt))
39:
end end for if grll ype Freed is True then
40:
41:
42:
Man.append (ppend(sndppann)) (sbbpla)) return plan end end for
43:
44:
45:
46:
grasp ype yperrreed Freed is False then return
47:
48:
end if
49:
50:
end if
17

---

Published as a conference paper at ICLR 2025
Figure 7: Rollout of the solution found by the GRN-based planner on the Access problem.
Figure 8: Rollout of the solution found by the GRN-based planner the Clutter problem.
18

---

Published as a conference paper at ICLR 2025
C.2 WORLD EXPERIMENTS
We test the BRRN-based planner in two real-world setups, using Panda arm and a PR2 robot. We use perception module based on Overworld (Sarthou (023) to identify objects and estimate their poses. The shapes of the object are known and are not subject to detection. The GRN-based planner was able to generate geometrically feasible plans on both the Access and Clutter problems, which were successfully executed by the robots. Figures 7 and g show the rollouts of both executions. In the Access problem, the desired task is simply to pick and place the bottle object, the planner has to find the solution plan that needs to move all obstructing objects to get the bottle. In the Clutter problem, the goal is to simply pick the object in the middle of the clutter and place it at the center of the table. In the same way, the planner has to find a solution that frees of the grasp types of the object first. Although these tasks might seem simple, they represent challenging TAMP problem with high combinatorial complexity. Indeed, in both problems, multiple objects not only block access to the goal object, they also block access to each other. The number of geometrically feasible solutions is very limited, requiring the planner to find the specific order in which to move the objects and which grasp to choose at each step. As shown in Figures a the Access problem requires Pick-Place actions to be solved, while the Clutter problem requires 7 Pick-Place actions to be solved.
C.3 ADDITIONAL SIMULATION EXPERIMENTS
Access problem with a large number of objects. We conduct an additional experiment in simu- lation to test the scalability of our method to problems requiring long-horizon planning. As in the previous problems, the goal is move single object (shown in pink in Figure D.. which a total of 27 objects block access. This task requires removing at least 18 objects to access the desired one, resulting in a total of 19 necessary Pick-Place actions. Figure shows GRN's predictions on the initial state of this problem. The model accurately predicts that, initially accessing the desired object infeasible, and that only two blockers can be moved, using single grasp type each. For simplic- ity, we show the obtained reasons of infeasibility of the pink object only. However, after the same initial query, our model also predicts the reasons of infeasibility for all the blockers as well. The non-informed TAMP algorithm from Bouhsain et al. (2024) completely fails to solve this problem, even after 2 hours of planning, while the GRN-based planner find a feasible solution in under 15s.
Action Feasibility Prodictions
Grasp Types Feasibility Predictions
Predicted Infeasibility cause grasp type (for the pink object
Input 3D Scene
Top
Front
Rear
No & Solution
No IK Solution
Cbstruzted
Right
Left
Feasible
Infeesilble
Arasp
Olas
telio
No IK Salution
Figure 9: Visualization of GRN predictions the initial state of larger version of the Access problem.
Inter-robot Handover. Our proposed approach can also be used to predict the feasibility of han- dover actions between two robots. Indeed, given the robot-centric nature of our model, we can divide a handover action into two separate Pick-Place actions, and query GRN for each robot individually to predict the feasibility of each sub-action. One of the main chalienges of handover tasks is to determine where to perform the object exchange. Our model can be used to efficiently find feasible handover positions by evaluating different sampled positions. In order to showcase this ability, we define the Handover problem, in which two Panda robots have to perform a handover task for a single object. A wall separates the two robots leaving only 3 "windows" through which the object can be exchanged. We sample a number of potential handover positions that lie in the intersection between the two robots workspaces, and query GRN evaluate their feasibility. As shown in Figure
19

---

Published as a conference paper at ICLR 2025
Action Feasibli ity Predictions for different sampled handover posibions
Grosp Types Feesiblliy Predictions for different sampled handover positions
Probiom
Figure 10: Visualization of GRN predictions on different sampled handover positions in the Han- dover problem.
10 our model is able to accurately predict that the wall blocks most of the sampled positions, and that only those in front of the windows are feasible. One key advantage of our method is that all the sampled positions can be evaluated simultaneously by adding a node to the input scene graph for each one, while making sure that proximity edges not created between these nodes. This enables an efficient search for feasible handover positions, requiring a single query to GRN. In contrast, traditional methods need to query the geometric planner for each sampled position independently, resulting in long planning times.
D INTERPRETABILITY ANALYSIS
To assess the interpretability of our model, we conduct an evaluation using a 3D environment con- taining two objects of interest for which the action and grasps are initially infeasible, illustrated in Figure 11 Based on GRN's predicted infeasibility causes, we systematically modify the envi- ronment observe how our model's predictions change in response. This allows to evaluate the interpretability of our method by showcasing how the predicted reasons of infeasibility explain ac- tion and grasp feasibility predictions. and how changing the input scene based on these infeasibility causes affects the outputs of the model.
Figure lla shows GRN's predictions on the initial 3D environment, and the predicted reasons of infeasibility for the first object of interest, For simplicity, we show infeasibility causes for the Top and Rear grasp types only. The obtained predictions show that the action involving the object of interest is infeasible, with Top grasps being obstructed by the shelf, while Rear grasps are obstructed by two movable objects. Figure TTb shows that increasing the height of the shelf removes the obstruction of the Top grasp type, with GRN predictions indicating that Top grasps and the action are now feasible. For the Rear grasp type, Figure I1e shows that removing only one of the two obstructing objects does not make the grasp type feasible. Removing the second object well is necessary to allow access to the object of interest, as shown in Figure 11d
Regarding the second object of interest, the corresponding reasons of infeasibility are shown in Figure TTel Due to the object's 's unreachability by the robot, all grasp types are infeasible due to the absence of IK solutions. Fi shows that decreasing the distance between the object and robot allows reachability, with GRN predictions showing that the action as well as the Top and Rear grasp types become feasible.
This evaluation demonstrates GRN's strong interpretability capabilities. By predicting specific in- feasibility causes, namely grasp obstructions and IK infeasibility, our model's feasibility predictions are not only informed, but also interpretable. Moreover, these explanations provide actionable in- sights into how modifying the environment affects feasibility. Its outputs consistently align with expected changes, sustaining GRN's ss reliability and coherence. These interpretability features are critical for real-world task and motion planning, where understanding the reasons behind infeasibil- ity is as important as the feasibility predictions themselves.
20

---

Published as a conference paper at ICLR 2025
3D Scene
Action Feasibiliit
Grasp Type Feasibility
Top and Rear infeasibility Causes
Object of est
Obstructed
Obstructed
(a) An action involving the object of interest is infeasible, Top grasps obstructed by the shelf, Rear grasps obstructed by movahle objects.
Top
Rear
Feasible
Obstructed
(b) With higher shelf height, Top grasps become feasible, which makes the action feasible
Top
Rear
Object of Interest
Obstructed
Obstructed
(c) Removing only one of the two obstructing objects does not free access to the object via Rear grasps
Top
Rear
Ocject of Interest
Obstructed
Feasible
(d) Removing both obstructing objects makes Rear grasps feasible, hence the action becomes feasible
Top
Rear
Object
Solution
No IK Solution
(c) The object of interest is unrcachable by the robot, making the action and grasps infcasible due to the absense of IK solutions
Top
Rear
Object
Feasible
Feasible
(f) Making the object closer to the robot allows reachability. thus the action and grasps become feasible.
Figure 11: Interpretability evaluation GRN's predictions on an example 3D environments.
21

---

Published as a conference paper at ICLR 2025
E ROBUSTNESS TO NOISE
We evaluate the robustness of GRN by introducing varying levels of noise to the input data. Specifi- cally, noise is added the node features, i.e. the dimensions and poses of all objects in the environ- ment. The noise is sampled from a normal distribution with the following parameters: (i) 1cm, 1 Noise Level: Noise with mean of 0 and standard deviation of cm for bounding box dimensions and translation coordinates, and 1 for object orientation, (ii) 2cm, 2 Noise Level: Noise with a mean of 0 and standard deviation of em for bounding box dimensions and translation coordi- nates, and 2" for object orientation. Table 7 presents the performance of GRN on the Panda-3D-44 test set under these noise conditions.
Table 7: Performance of GRN on the Panda-3D-4 test set with different levels of noise.
No Noise
1cm,
2cm, 2
Action (F1)
0.939
0.912
0.864
Grasp (FI)
0.940 (+: 0.009)
0.891 (#: 0.020)
0.820 (#: 0.029)
IK (F1)
0.995 (+: 0.001)
0.985 (#: 0.001)
0.971 (#: 0.002)
GO (MAE)
0.028 (# 0.003)
0.039 (# 0.003)
0.057 (# 0.003)
While there is a slight decrease in performance noise levels increase, GRN maintains a good accuracy in both action and grasp feasibility predictions. Importantly, even under the highest noise condition (2 cm, 29), our method still achieves better performance than the one yielded by most baselines on noise-free inputs, shown in Table | More precisely, under both noise levels, GRN outperforms MLP, DVH and F-GCN on action and grasp feasibility prediction, even when these baselines are given noise-free inputs. Under the (1em, 19) noise level, our model also provides more accurate predictions than the ones obtained by AGFPNet and F-GAT under zero noise. Under the highest noise level, GRN achieves similar performance to noise-free F-GAT and a slightly worse performance than zero-noise AGFPNet on action feasibility prediction, but still yields a better performance than both on grasp type feasibility prediction. Also, comparing standard deviations shows that even when noise is applied, GRN results a more consistent prediction quality across grasp types.
Furthermore, we highlight that in addition to a better accuracy under different noise level, our model also provides richer information thanks to our two interpretation modules. 7 that GRN maintains a good performance across noise levels on IK fcasibility and Grasp Obstruction (GO) predictions as well. These results emphasize the robustness of GRN to real-world imperfections, while still delivering state-of-the-ae performance compared to other methods.
shows
F VISUALIZATIONS
In this section, we present a visual comparison between the predictions of our model, GRN, and the ground truth. Figure 12 illustrates GRN's predictions in test environment from Panda-3D- 4, while Figure [3 showcases predictions from Panda-3D-10. In these figures, action feasibility predictions and ground truth are represented through object colors in the top images; green indicates feasibility for pick or place actions, while red denotes infeasibility. The feasibility of different grasp types visualized through the coloration of the corresponding object's faces. Finally, the reasons of infeasibility are shown for one of the objects in the environment (in blue), for each grasp type. There are two types of infeasibility causes, IK infeasibility (with the robot depicted in red) and grasp obstructions (where obstructing objects are displayed in full opacity, with their colors reflecting obstruction ratios). These comparisons highlight that GRN accurately predicts action and grasp feasibility for most objects. It also effectively identifies the reasons for infeasibility and estimates grasp obstruction ratios with minimal error. A notable exception occurs in the 10-object environment, where the Rear grasp type of one object is incorrectly classified as infeasible. Figure details the predicted infeasibility reasons for this misclassification. corresponding Rear grasp type plot reveals slight overestimation of the obstruction ratio for large block, which likely contributes to the error. However, GRN correctly predicts the feasibility of other grasp types for the same object, including the feasible Top grasp type, ensuring accurate action feasibility predictions overall.
closer examination of the
22

---

Published as a conference paper at ICLR 2025
3D Scene
Predictions
Ground Truth
Action Feasibllity
Grasp types Feasibility
Top
Top
Reason Innfasibblltty per type
Obstructed
Obstructed
(for the blue object)
Front
Front
Roar
Obstructed
Obstructed
Obstructed
Obstructed
Right
Left
Right
Obstructed
Obstructed
Obstructed
Obstructed
0.5
Feasible
Inteasible
Grasp Obstruction
Figure 12: Visualization of GRN prediction compared to the ground truth test environment from Panda-3D-4.
23

---

Published as a conference paper at ICLR 2025
3D Scene
Predictions
Ground Truth
Action Foasibility
Grasp types Feasibility
Top
Top
Reason infeasibiitt per type
Obstructed
Obstructed
(for the blue object)
Front
Rear
Fregt
Rear
No IK Solution
Obstructed
No IK Solution
Feasible
Right
Left
Right
Left
Feasible
No IK Solution
Feasible
No IK Solution
0.5
Feasible
Infeasible
Grasp Obstruction ratio
Figure 13: Visualization of GRN prediction compared to the ground truth test environment from Panda-3D-10.
24


Reviews:
[{"review_id": "b38m95GMVt", "reviewer": "Reviewer_LeQs", "summary": "This paper presents Geometric Reasoning Networks (GRN), a graph neural network-based approach to enhance efficiency in Task and Motion Planning (TAMP) for robotic manipulation. An incremental GNN-based model is introduced with two mechanisms: inverse kinematics (IK) feasibility prediction, and grasp obstruction (GO) estimation using full-state knowledge. Experimental validation demonstrates GRN's applicability to various environments and configurations.", "questions": "I've listed my main concerns under _weaknesses_ section, and here are some further questions (some overlapping with the points above):\n\n- **Self-loop Edge Representation in 4.1**: What is the rationale for using the self-loop edge to store IK feasibility instead of treating it as a node feature? How does this choice affect the overall model performance or its ability to generalize to unseen environments?\n\n- **Effectiveness and Necessity of the IK Module**: In the ablation study (Table 3), the results show minimal difference between the full model and the one without the IK module. Given this, is the IK module truly necessary? What are the specific cases where IK adds value? Also, could you provide the inference time for the IK module to understand its overhead?\n\n- **Access Task Success Rate in Section 6.4**: The success rate for the Access task in Table 5 is slightly lower than Bouhsain et al. (2024). What are the main reasons for this reduction? Are there particular scenarios where GRN struggles, or is this a result of experimental variance?\n\n- **Inference Time in Table 5**: Could you provide more detailed inference times for Table 5, including how the times break down between GRN predictions and geometric planning? Understanding the computational cost of each component would provide a clearer picture of where time savings are realized.\n\n- **Handling Dynamic Scenes**: The current assumptions restrict the model to static scenes, which is not always practical in real-world tasks. How would this approach be extended to handle dynamic environments?\n  \n- **Real-world Evaluation of Long-horizon Tasks**: The paper would benefit from showing how GRN scales to more challenging multi-step tasks, such as multi-robot collaboration or handling non-prehensile manipulation. Can the authors comment on future steps for validating GRN on these scenarios?\n\n- **Simplified Grasp Types**: The use of only five grasp types is limiting. Are there plans to extend the grasp representation, or can the authors justify why five types were sufficient for the tasks tested in this paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The study tackles a significant challenge in TAMP arising from the bottleneck in geometric planning within complex 3D environments and proposes a novel, though incremental, architecture to predict action and grasp feasibility to reduce the need for costly geometric planning queries.  \n\n- The model is more generalizable in complex environments compared to prior works (Table 4).\n\n- The proposed model is evaluated and compared against prior works and an ablation study is conducted.\n\n- The authors conducted real-robot experiments, and shared their code.", "weaknesses": "Below is the list of the issues I observed, and my suggestions for this work:\n\n1. **Strong Assumptions on Object Knowledge and Static Scene Conditions**: The model assumes complete knowledge of object shape, dimensions, and pose, as well as static conditions where objects remain stationary unless moved by the robot (lines 153–154). This is a considerable simplification for real-world scenarios, where partial observability and dynamic conditions are typical. By contrast, some baselines used for comparison assume partial observability and predict feasibility based on images rather than full state knowledge, making these comparisons less directly fair. Addressing this by either relaxing these assumptions or comparing only with baselines under similar assumptions would improve applicability and fairness.\n\n2. **Limited Integration and Short Planning Horizons**: While some baselines are fully integrated within TAMP solvers, GRN’s integration is relatively basic and tested only on shorter-horizon problems. The scenarios addressed in the experiments are limited in complexity, lacking tasks that demand multi-step, long-horizon planning (e.g., removal of obstructing objects or inter-robot handovers). Expanding the evaluation to more complex, long-horizon planning tasks would better demonstrate GRN's practical utility in handling realistic TAMP challenges.\n\n3. **Improvement for TAMP**: Since the paper do not include a full-fledge TAMP solver, it remains unclear how it improves traditional/existing solvers. It'd have been really useful to see how many geometrical computations were skipped (so computational time was gained) when GRN was used vs. not used. Inference time comparison to some of the baselines is not fair, as they use image-based input while this work relies on a significantly smaller full-state information of the environment. \n\n4. **Incremental Advancement and Limited Impact of IK Infeasibility Module**: While GRN introduces small adjustments on Edge-Featured Graph Attention Network (EGAT), these additions are incremental and rely heavily on prior EGAT methods. Furthermore, as shown in Table 3, the IK infeasibility module does not significantly impact overall performance. Adding a deeper analysis or improvement in these areas could strengthen the contribution. It seems like GO by itself is already a good estimator of feasibility. This could also be due to problem settings where there might not be (enough) cases when there is no grasp obstruction IK was still infeasible (e.g., due to reachability). A more nuanced analysis of scenarios where IK feasibility alone might be crucial (e.g., environments with tighter spaces or more complex robot configurations) that would help clarify when this module provides substantial benefits.\n\n5. **Restricted Grasp Representation**: The model simplifies grasp feasibility by considering only five grasp types, which may limit its effectiveness in real-world settings where grasp variety and adaptability are crucial. Even for grasping box-shaped objects requires 24 (or 20 if it lies flat on a surface) different configurations with a parallel jaw gripper, much higher than 5 as used in this work. Exploring a more flexible grasp representation that could handle diverse object shapes and placements would make the model more versatile for practical robotics applications.\n\n6. **Limited Interpretability Support**: While the authors claim that GRN allows interpretation of feasibility (or infeasibility) of actions, this claim lacks follow-up discussion or specific experimental support. Including experiments or qualitative analysis to illustrate interpretability (e.g., showing how grasp obstruction information directly affects planning decisions) would make this feature more convincing and actionable. \n\n7. **Clarity and Organization of Paper Structure**: The organization could be improved for readability and logical flow. For example, Figure 2 is never referenced in the text, and Figure 1, which appears on page 3, is not mentioned until Section 6.4 on page 10. Figures in the Appendix (e.g., Figures 7 and 8) are not referenced or explained, leaving the reader without context for understanding their relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhtxzDAmIb", "reviewer": "Reviewer_edpJ", "summary": "The paper presents a graph neural network model for efficient Task and Motion Planning (TAMP). It uses a scene graph to represent 3D environments, incorporating Inverse Kinematics (IK) and Grasp Obstruction (GO) modules. Experiments show its superiority over state-of-the-art methods in accuracy, generalization, and speed. It reduces the reliance on computationally expensive geometric planners while remaining robust across diverse robots and complex scenes.", "questions": "1. How would the performance of GRN be affected if the Inverse Kinematics (IK) and Grasp Obstruction (GO) modules were replaced with traditional methods, such as standard inverse kinematics solvers or simpler distance-based obstruction checks? Specifically, how would this change impact prediction accuracy, interpretability, and computational efficiency in various environments?\n\n2. What is the maximum level of environmental complexity that GRN can handle effectively? For example, how many fixed nodes or objects can it support while maintaining accuracy and efficiency? Could GRN scale to environments with thousands of nodes, akin to point cloud representations, and still perform well in highly cluttered scenes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It incorporates explainable modules within the GNN, such as inverse kinematics and grasp obstruction estimations, to predict feasibility. This interpretability enhances decision-making by providing insights into why certain actions are infeasible.\n\n2. Experiments demonstrate that GRN significantly lowers computational costs compared to traditional geometric planners, making it a more efficient solution for real-time TAMP applications.", "weaknesses": "1. The method's reliance on simplified bounding box representations and discrete grasp feasibility may limit its effectiveness in real-world TAMP problems, especially in dynamic, cluttered environments with irregular objects or sensor noise.\n\n2. The method only assesses discrete feasibility without motion planning, restricting its ability to handle complex cases where motion feasibility [1] is crucial. \n\n[1] Scaling Multi-Modal Planning: Using Experience and Informing Discrete Search", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7aFKw2RAC", "reviewer": "Reviewer_FvER", "summary": "The paper introduces Geometric Reasoning Networks (GRN), a GNN model for predicting action and grasp feasibility. Technical Innovations of the paper includes:\n- Novel scene representation using directed graphs\n- Edge-Enhanced Graph Attention Network (EGAT) that leverages both node and edge features\n- Ability to explain why actions are infeasible, enabling more efficient planning", "questions": "1a. How imbalanced is the dataset? Is it nearly composed of 50% feasible cases and 50% infeasible cases? It would be appreciated if some ratio of the dataset could be shared as a table.\n\n1b. Since RRT is a random algorithm, I think it doesn't necessarily mean one task is truly infeasible, if RRT couldn't generate a plan. What is the definition of the feasibility in this paper's scope?\n\n2. Regarding training, why is pre-trained needed before jointly fine-tuning? Was there any training instability if we directly train all the modules?\n\n3. The generalization experiment is interesting. What I'm expecting is that the graph representation should be powerful enough to achieve a significantly better generalization. However, it seems that the MLP method is also good enough to generalize to 20 obstacles. Is there any specific reason why this is the case? Or are we also facing the imbalanced dataset issue here? Sampling a feasible case for 20 obstacles sounds challenging, right?\n\n4. I am not very familiar with TAMP problem. Just as a discussion, in general, when would a learning-based feasibility checker be preferred compared to a learning-based end-to-end planner, that directly generates a plan? And when vice versa?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Outperforms state-of-the-art methods in both action and grasp feasibility prediction\n2. Lower inference time compared to traditional geometric planning\n3. Generalization to more complex environments and different robot types\n4. Writing is very clear and detailed", "weaknesses": "1. Some details for the experiment setting needed to be discussed further to correctly evaluate the approach. See the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B05laaJQGc", "reviewer": "Reviewer_WQrW", "summary": "This paper proposes to use GNN-based methods to predict action and grasp feasibility, which can speed up the geometric planner in task and motion planning. Furthermore, the authors propose to predict the inverse kinematics feasibility and grasp obstruction to improve the interpretability. Through quantitative and qualitative experiments, the authors verify the effectiveness of the proposed approach.", "questions": "1: how would your proposed approach compare to related works in geometric feasibility reasoning [1, 2, 3] and graph neural networks in task and motion planning [4, 5, 6]? \n\n2: Is there any noise when you estimate the object's shape and pose in the real world? If there is noise, would your proposed approach be robust to the noise? \n\n3: Could you train one more for all robots? Training one model for each robot limits the generalization ability of your proposed system.  \n\n4: For the Panda-3D-4 dataset, the MLP achieves a 0.558 F1 score but the Panda-3D-20 achieves a 0.609 F1 score, why does MLP perform better in a more complex dataset? This is weird. \n\n5: In Table 5, why does your proposed approach perform worse than the baseline in the “Access” problem? Why does the baseline always achieve a 100% success rate? I guess your task is too easy.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1: This paper is well-motivated. The efficiency of task and motion planning is a significant issue and how to improve it is an important research question. \n\n2: Predicting geometric dependencies and feasibility with a graph neural network makes sense to me. \n\n3: This paper contains an extensive experiments section and compares the method to several baselines, which is nice. \n\n4: The paper shows some real-world demos.", "weaknesses": "1: My first concern is the limited discussion and comparison to related works in geometric feasibility reasoning [1, 2, 3] and graph neural networks in task and motion planning [4, 5, 6]. In particular, [2] also uses feasibility-related predicates to improve the efficiency of the task planning. \n\n[1]: K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345–1365, 2023.\n\n[2]: Y. Huang, C. Agia, J. Wu, T. Hermans, and J. Bohg. Points2Plans: From Point Clouds to Long-Horizon Plans with Composable Relational Dynamics, ArXiv, 2024. \n\n[3]: C. Agia, T. Migimatsu, J. Wu, and J. Bohg. Stap: Sequencing task-agnostic policies. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7951–7958. IEEE, 2023.\n\n[4]: Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers. In IEEE International Conference on Robotics and Automation (ICRA), 2023.\n\n[5]: H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning to see, simulate, and shape elastoplastic objects in 3d with graph networks. The International Journal of Robotics Research. \n\n[6]: H. Chen, Y. Niu, K. Hong, S. Liu, Y. Wang, Y. Li, and K. R. Driggs-Campbell, “Predicting object interactions with behavior primitives: An application in stowing tasks,” in 7th Annual Conference on Robot Learning, 2023.\n\n2: The videos and figures for this paper are hard to follow. For example, in Figure 1, I can only understand green represents feasible and red represents feasible. However, there are other colored objects like blue ones. Furthermore, why robots are sometimes grey and sometimes red?  \n\n3: This paper makes several strong assumptions including known object shapes and poses, and all objects remain static except the robot grasps them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)