Learning Geometric Reasoning Networks For Robot Task And Motion Planning

Authors: Smail Ait Bouhsain, Rachid Alami, Thierry Simeon
Primary area: applications to robotics, autonomy, planning
Keywords: Graph Neural Networks, Deep Learning for Robotics, Task and Motion Planning, Robot Manipulation Planning

Abstract:
Task and Motion Planning (TAMP) is a computationally challenging robotics problem due to the tight coupling of discrete symbolic planning and continuous geometric planning of robot motions. In particular, planning manipulation tasks in complex 3D environments leads to a large number of costly geometric planner queries to verify the feasibility of considered actions and plan their motions. To address this issue, we propose Geometric Reasoning Networks (GRN), a graph neural network (GNN)-based model for action and grasp feasibility prediction, designed to significantly reduce the dependency on the geometric planner. Moreover, we introduce two key interpretability mechanisms: inverse kinematics (IK) feasibility prediction and grasp obstruction (GO) estimation. These modules not only improve feasibility predictions accuracy, but also explain why certain actions or grasps are infeasible, thus allowing a more efficient search for a feasible solution. Through extensive experimental results, we show that our model outperforms state-of-the-art methods, while maintaining generalizability to more complex environments, diverse object shapes, multi-robot settings, and real-world robots.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729721146839

thread_type:
official_review

<thread_object>
creator: reviewer_wqrw

<content>
summary:
This paper proposes to use GNN-based methods to predict action and grasp feasibility, which can speed up the geometric planner in task and motion planning. Furthermore, the authors propose to predict the inverse kinematics feasibility and grasp obstruction to improve the interpretability. Through quantitative and qualitative experiments, the authors verify the effectiveness of the proposed approach.

strengths:
1: This paper is well-motivated. The efficiency of task and motion planning is a significant issue and how to improve it is an important research question. 

2: Predicting geometric dependencies and feasibility with a graph neural network makes sense to me. 

3: This paper contains an extensive experiments section and compares the method to several baselines, which is nice. 

4: The paper shows some real-world demos.

weaknesses:
1: My first concern is the limited discussion and comparison to related works in geometric feasibility reasoning [1, 2, 3] and graph neural networks in task and motion planning [4, 5, 6]. In particular, [2] also uses feasibility-related predicates to improve the efficiency of the task planning. 

[1]: K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345–1365, 2023.

[2]: Y. Huang, C. Agia, J. Wu, T. Hermans, and J. Bohg. Points2Plans: From Point Clouds to Long-Horizon Plans with Composable Relational Dynamics, ArXiv, 2024. 

[3]: C. Agia, T. Migimatsu, J. Wu, and J. Bohg. Stap: Sequencing task-agnostic policies. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7951–7958. IEEE, 2023.

[4]: Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers. In IEEE International Conference on Robotics and Automation (ICRA), 2023.

[5]: H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning to see, simulate, and shape elastoplastic objects in 3d with graph networks. The International Journal of Robotics Research. 

[6]: H. Chen, Y. Niu, K. Hong, S. Liu, Y. Wang, Y. Li, and K. R. Driggs-Campbell, “Predicting object interactions with behavior primitives: An application in stowing tasks,” in 7th Annual Conference on Robot Learning, 2023.

2: The videos and figures for this paper are hard to follow. For example, in Figure 1, I can only understand green represents feasible and red represents feasible. However, there are other colored objects like blue ones. Furthermore, why robots are sometimes grey and sometimes red?  

3: This paper makes several strong assumptions including known object shapes and poses, and all objects remain static except the robot grasps them.

questions:
1: how would your proposed approach compare to related works in geometric feasibility reasoning [1, 2, 3] and graph neural networks in task and motion planning [4, 5, 6]? 

2: Is there any noise when you estimate the object's shape and pose in the real world? If there is noise, would your proposed approach be robust to the noise? 

3: Could you train one more for all robots? Training one model for each robot limits the generalization ability of your proposed system.  

4: For the Panda-3D-4 dataset, the MLP achieves a 0.558 F1 score but the Panda-3D-20 achieves a 0.609 F1 score, why does MLP perform better in a more complex dataset? This is weird. 

5: In Table 5, why does your proposed approach perform worse than the baseline in the “Access” problem? Why does the baseline always achieve a 100% success rate? I guess your task is too easy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730409839392

thread_type:
official_review

<thread_object>
creator: reviewer_fver

<content>
summary:
The paper introduces Geometric Reasoning Networks (GRN), a GNN model for predicting action and grasp feasibility. Technical Innovations of the paper includes:
- Novel scene representation using directed graphs
- Edge-Enhanced Graph Attention Network (EGAT) that leverages both node and edge features
- Ability to explain why actions are infeasible, enabling more efficient planning

strengths:
1. Outperforms state-of-the-art methods in both action and grasp feasibility prediction
2. Lower inference time compared to traditional geometric planning
3. Generalization to more complex environments and different robot types
4. Writing is very clear and detailed

weaknesses:
1. Some details for the experiment setting needed to be discussed further to correctly evaluate the approach. See the questions.

questions:
1a. How imbalanced is the dataset? Is it nearly composed of 50% feasible cases and 50% infeasible cases? It would be appreciated if some ratio of the dataset could be shared as a table.

1b. Since RRT is a random algorithm, I think it doesn't necessarily mean one task is truly infeasible, if RRT couldn't generate a plan. What is the definition of the feasibility in this paper's scope?

2. Regarding training, why is pre-trained needed before jointly fine-tuning? Was there any training instability if we directly train all the modules?

3. The generalization experiment is interesting. What I'm expecting is that the graph representation should be powerful enough to achieve a significantly better generalization. However, it seems that the MLP method is also good enough to generalize to 20 obstacles. Is there any specific reason why this is the case? Or are we also facing the imbalanced dataset issue here? Sampling a feasible case for 20 obstacles sounds challenging, right?

4. I am not very familiar with TAMP problem. Just as a discussion, in general, when would a learning-based feasibility checker be preferred compared to a learning-based end-to-end planner, that directly generates a plan? And when vice versa?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703487371

thread_type:
official_review

<thread_object>
creator: reviewer_edpj

<content>
summary:
The paper presents a graph neural network model for efficient Task and Motion Planning (TAMP). It uses a scene graph to represent 3D environments, incorporating Inverse Kinematics (IK) and Grasp Obstruction (GO) modules. Experiments show its superiority over state-of-the-art methods in accuracy, generalization, and speed. It reduces the reliance on computationally expensive geometric planners while remaining robust across diverse robots and complex scenes.

strengths:
1. It incorporates explainable modules within the GNN, such as inverse kinematics and grasp obstruction estimations, to predict feasibility. This interpretability enhances decision-making by providing insights into why certain actions are infeasible.

2. Experiments demonstrate that GRN significantly lowers computational costs compared to traditional geometric planners, making it a more efficient solution for real-time TAMP applications.

weaknesses:
1. The method's reliance on simplified bounding box representations and discrete grasp feasibility may limit its effectiveness in real-world TAMP problems, especially in dynamic, cluttered environments with irregular objects or sensor noise.

2. The method only assesses discrete feasibility without motion planning, restricting its ability to handle complex cases where motion feasibility [1] is crucial. 

[1] Scaling Multi-Modal Planning: Using Experience and Informing Discrete Search

questions:
1. How would the performance of GRN be affected if the Inverse Kinematics (IK) and Grasp Obstruction (GO) modules were replaced with traditional methods, such as standard inverse kinematics solvers or simpler distance-based obstruction checks? Specifically, how would this change impact prediction accuracy, interpretability, and computational efficiency in various environments?

2. What is the maximum level of environmental complexity that GRN can handle effectively? For example, how many fixed nodes or objects can it support while maintaining accuracy and efficiency? Could GRN scale to environments with thousands of nodes, akin to point cloud representations, and still perform well in highly cluttered scenes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731530912

thread_type:
official_review

<thread_object>
creator: reviewer_leqs

<content>
summary:
This paper presents Geometric Reasoning Networks (GRN), a graph neural network-based approach to enhance efficiency in Task and Motion Planning (TAMP) for robotic manipulation. An incremental GNN-based model is introduced with two mechanisms: inverse kinematics (IK) feasibility prediction, and grasp obstruction (GO) estimation using full-state knowledge. Experimental validation demonstrates GRN's applicability to various environments and configurations.

strengths:
- The study tackles a significant challenge in TAMP arising from the bottleneck in geometric planning within complex 3D environments and proposes a novel, though incremental, architecture to predict action and grasp feasibility to reduce the need for costly geometric planning queries.  

- The model is more generalizable in complex environments compared to prior works (Table 4).

- The proposed model is evaluated and compared against prior works and an ablation study is conducted.

- The authors conducted real-robot experiments, and shared their code.

weaknesses:
Below is the list of the issues I observed, and my suggestions for this work:

1. **Strong Assumptions on Object Knowledge and Static Scene Conditions**: The model assumes complete knowledge of object shape, dimensions, and pose, as well as static conditions where objects remain stationary unless moved by the robot (lines 153–154). This is a considerable simplification for real-world scenarios, where partial observability and dynamic conditions are typical. By contrast, some baselines used for comparison assume partial observability and predict feasibility based on images rather than full state knowledge, making these comparisons less directly fair. Addressing this by either relaxing these assumptions or comparing only with baselines under similar assumptions would improve applicability and fairness.

2. **Limited Integration and Short Planning Horizons**: While some baselines are fully integrated within TAMP solvers, GRN’s integration is relatively basic and tested only on shorter-horizon problems. The scenarios addressed in the experiments are limited in complexity, lacking tasks that demand multi-step, long-horizon planning (e.g., removal of obstructing objects or inter-robot handovers). Expanding the evaluation to more complex, long-horizon planning tasks would better demonstrate GRN's practical utility in handling realistic TAMP challenges.

3. **Improvement for TAMP**: Since the paper do not include a full-fledge TAMP solver, it remains unclear how it improves traditional/existing solvers. It'd have been really useful to see how many geometrical computations were skipped (so computational time was gained) when GRN was used vs. not used. Inference time comparison to some of the baselines is not fair, as they use image-based input while this work relies on a significantly smaller full-state information of the environment. 

4. **Incremental Advancement and Limited Impact of IK Infeasibility Module**: While GRN introduces small adjustments on Edge-Featured Graph Attention Network (EGAT), these additions are incremental and rely heavily on prior EGAT methods. Furthermore, as shown in Table 3, the IK infeasibility module does not significantly impact overall performance. Adding a deeper analysis or improvement in these areas could strengthen the contribution. It seems like GO by itself is already a good estimator of feasibility. This could also be due to problem settings where there might not be (enough) cases when there is no grasp obstruction IK was still infeasible (e.g., due to reachability). A more nuanced analysis of scenarios where IK feasibility alone might be crucial (e.g., environments with tighter spaces or more complex robot configurations) that would help clarify when this module provides substantial benefits.

5. **Restricted Grasp Representation**: The model simplifies grasp feasibility by considering only five grasp types, which may limit its effectiveness in real-world settings where grasp variety and adaptability are crucial. Even for grasping box-shaped objects requires 24 (or 20 if it lies flat on a surface) different configurations with a parallel jaw gripper, much higher than 5 as used in this work. Exploring a more flexible grasp representation that could handle diverse object shapes and placements would make the model more versatile for practical robotics applications.

6. **Limited Interpretability Support**: While the authors claim that GRN allows interpretation of feasibility (or infeasibility) of actions, this claim lacks follow-up discussion or specific experimental support. Including experiments or qualitative analysis to illustrate interpretability (e.g., showing how grasp obstruction information directly affects planning decisions) would make this feature more convincing and actionable. 

7. **Clarity and Organization of Paper Structure**: The organization could be improved for readability and logical flow. For example, Figure 2 is never referenced in the text, and Figure 1, which appears on page 3, is not mentioned until Section 6.4 on page 10. Figures in the Appendix (e.g., Figures 7 and 8) are not referenced or explained, leaving the reader without context for understanding their relevance.

questions:
I've listed my main concerns under _weaknesses_ section, and here are some further questions (some overlapping with the points above):

- **Self-loop Edge Representation in 4.1**: What is the rationale for using the self-loop edge to store IK feasibility instead of treating it as a node feature? How does this choice affect the overall model performance or its ability to generalize to unseen environments?

- **Effectiveness and Necessity of the IK Module**: In the ablation study (Table 3), the results show minimal difference between the full model and the one without the IK module. Given this, is the IK module truly necessary? What are the specific cases where IK adds value? Also, could you provide the inference time for the IK module to understand its overhead?

- **Access Task Success Rate in Section 6.4**: The success rate for the Access task in Table 5 is slightly lower than Bouhsain et al. (2024). What are the main reasons for this reduction? Are there particular scenarios where GRN struggles, or is this a result of experimental variance?

- **Inference Time in Table 5**: Could you provide more detailed inference times for Table 5, including how the times break down between GRN predictions and geometric planning? Understanding the computational cost of each component would provide a clearer picture of where time savings are realized.

- **Handling Dynamic Scenes**: The current assumptions restrict the model to static scenes, which is not always practical in real-world tasks. How would this approach be extended to handle dynamic environments?
  
- **Real-world Evaluation of Long-horizon Tasks**: The paper would benefit from showing how GRN scales to more challenging multi-step tasks, such as multi-robot collaboration or handling non-prehensile manipulation. Can the authors comment on future steps for validating GRN on these scenarios?

- **Simplified Grasp Types**: The use of only five grasp types is limiting. Are there plans to extend the grasp representation, or can the authors justify why five types were sufficient for the tasks tested in this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732503476443

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank all reviewers for their constructive feedback and valuable insights. We appreciate the reviewers' acknowledgment of the **novelty and clarity of our work**, as well as the **efficiency, generalizability, and interpretability** of our approach.

Based on the reviewers’ feedback, we have made the following modifications to our manuscript:

- **Improved Figures:** We revised the manuscript to make our figures easier to understand and properly referenced them in relevant sections.
- **Updated Related Works:** We updated our related works section to include recent research in geometric feasibility reasoning and GNNs for task and motion planning, as suggested by Reviewer WQrW.
- **Enhanced Table 5:** We included the number of geometric planner queries in Table 5, addressing Reviewer LeQs’s suggestion.
- **Inference Time Analysis:** We added an inference time decomposition of our method in Appendix A.
- **Dataset Details:** We provided additional details about our datasets and data annotation in Appendix B, including label distributions for the Panda-3D-4 and PR2-3D-4 datasets.
- **Problems complexity:** We emphasize the difficulty of the **Access** and **Clutter** problems in Appendix C.2.
- **New Experiments:** We added two new experiments in Appendix C.3, demonstrating our method's ability to handle **complex long-horizon problems** and **inter-robot handover tasks**.

We believe these changes significantly enhance the strengths and clarity of our paper while further showcasing the impact of our contributions. We hope these improvements align with the reviewers’ expectations.

As the discussion phase nears its conclusion, we kindly ask all reviewers for their reassessment and encourage them to share any remaining concerns they might have.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel framework for learning Geometric Reasoning Networks (GRN) to address task and motion planning (TAMP) challenges in robotic manipulation. The method combines geometric reasoning with deep learning techniques to improve efficiency and reduce dependency on traditional geometric planners. The authors integrate the GRN within a TAMP framework, focusing on long-horizon planning problems and demonstrating their approach's efficacy in both simulated and real-world environments.

The paper makes strong claims about the generalizability of its learned representations and the ability to handle complex reasoning tasks that were previously challenging for geometric planners. The experimental results show improvements over baseline methods in terms of success rates, planning efficiency, and computational cost, making a convincing case for the practical utility of the proposed method.

**Strengths:**

1. The proposed GRN introduces a novel combination of geometric reasoning with task planning, a critical step forward in addressing the limitations of traditional TAMP approaches.

2. The authors provide extensive evaluations, including comparisons to baseline geometric planners and ablations that demonstrate the contributions of individual components. The experiments are performed across diverse environments, adding credibility to the results.

3. The integration of GRN within a broader TAMP framework demonstrates its scalability and applicability to real-world tasks, showcasing both simulation and real-world results.

**Weaknesses:**

1. While the results are strong in structured and static scenarios, the paper does not sufficiently explore the method's applicability to dynamic environments or tasks involving significant real-time changes.

2. The reviewers noted that the paper's claims about generalizability are not fully substantiated, as the evaluations focus primarily on specific planning scenarios. Broader validations would further strengthen the impact.

3. Although the paper includes comparisons with traditional geometric planners, additional comparisons with alternative learning-based TAMP methods could provide a more comprehensive evaluation.

4. The real-world experiments, while impactful, are limited in scale and complexity. Expanding these evaluations to larger or more diverse scenarios would better support the paper's claims.

**Reasons for Acceptance:**

The paper addresses a critical challenge in TAMP, offering a novel solution that integrates geometric reasoning with deep learning. The proposed framework shows strong empirical results and demonstrates its potential applicability to real-world tasks. While there are some limitations in terms of generalizability and dynamic task exploration, the strengths outweigh these concerns. The method is a valuable contribution to the field, and the revisions made during the rebuttal significantly improved the paper's quality and clarity.