Title: Learning Spatial-Aware Manipulation Ordering
Venue: NeurIPS 2025
Accepted: yes
Keywords: object manipulation

cluttered environment
Abstract: Manipulation in cluttered environments is challenging due to spatial dependencies among objects, where an improper manipulation order can cause collisions or blocked access. Existing approaches often overlook these spatial relationships, limiting their flexibility and scalability. To address these limitations, we propose OrderMind, a unified spatial-aware manipulation ordering framework that directly learns object manipulation priorities based on spatial context. Our architecture integrates a spatial context encoder with a temporal priority structuring module. We construct a spatial graph using k-Nearest Neighbors to aggregate geometric information from the local layout and encode both object-object and object-manipulator interactions to support accurate manipulation ordering in real-time. To generate physically and semantically plausible supervision signals, we introduce a spatial prior labeling method that guides a vision-language model to produce reasonable manipulation orders for distillation. We evaluate OrderMind on our Manipulation Ordering Benchmark, comprising 163,222 samples of varying difficulty. Extensive experiments in both simulation and real-world environments demonstrate that our method significantly outperforms prior approaches in effectiveness and efficiency, enabling robust manipulation in cluttered scenes.


Peer reviews:
comments: SUMMARY:
In this paper, the authors address the problem of the order of manipulation, which is crucial in tasks such as picking up postal parcels. The robot must determine which item to handle first and which to handle next.

STRENGTHS AND WEAKNESSES:
The topic is useful, and the performance is good. The method considers both spatial and temporal contexts, which should be valuable for this problem. However, when examining the framework, it seems that the authors treat the problem primarily as a computer vision task, similar to how appreciation order is used to detect objects in an image. I wonder if this problem has been considered from the perspective of object detection. If Yes. Whether this method could be applied to the context in this paper to achieve even better results.

QUESTIONS:
1. The attributes are very important for representing the object. How are these attributes obtained? I think the authors should provide more details on this process.
2. What are the extrinsic attributes? What do the arrows with different colors represent, and how are they obtained?
3. In Fig. 2, do you directly combine all the embeddings together? How are the circle and rectangle combined? I believe there are several fusion methods; could the authors clarify which add is used here?
4. How is fusion implemented in the extrinsic module?

LIMITATIONS:
See weakness
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OrderMind, a spatial-aware framework for learning object manipulation order in cluttered scenes. The idea is to directly learn manipulation priority from RGB-D inputs using a combination of a k-NN spatial graph (for object-object and object-robot relationships) and a temporal ordering module that ranks objects by learned priority scores. Labels for supervision are generated using a vision-language model (VLM), but enhanced with spatial priors (like object isolation or stackability). The authors also built a large-scale benchmark (160k+ samples) and show both strong sim and real-world performance compared to heuristics, VLM pipelines, and two-stage baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The problem that author explores is real and under-explored by other works. OrderMind speedup on slow multi-stage pipelines by jointly encoding spatial reasoning and prioritization in one model. Use of k-NN spatial graphs and attention-based temporal priority modeling is a well-motivated and effective architectural choice. No manual annotation needed. Proposed a new benchmark. Both qualitative and quantitative results show consistent improvements across difficulty levels and across simulation/real-world settings.\n\n**Weaknesses**: Spatial priors help, but the generated order labels may still reflect unintended biases or fail in edge cases. Some analysis on this would help.\nThe authors miss details on VLM prompting: What exactly is the prompt format for order generation? Is there chain-of-thought prompting or just spatial descriptor input? Since the VLM is a core component for supervision, this should be shown.\nIn addition, the author should also discuss the label noise come from VLM. What happens when VLM labels are wrong or inconsistent? The model likely learns around it, but it'd be good to quantify how robust it is.\n\nQUESTIONS:\nCan you share examples of the prompts used for VLM ordering supervision? Was any spatial descriptor, e.g., relative pose or bounding box info, passed into the prompt?\nThe VLM-generated labels are critical to training. How consistent are these labels across different scenes and object types? Have you tested generalization to new objects or unseen environments beyond YCB?\nWhat kind of errors does the model tend to make in hard real-world scenes (e.g., with deformable or occluded bags)? Qualitative failure examples would be valuable.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe agent should be able to look at a scene, understand 3D relationships between objects, and then take a series of manipulation actions to pick up unobstructed objects efficiently.\n\nSTRENGTHS AND WEAKNESSES:\nThe application domain is straight forward to understand and the task is of real utility. There are a number of points where I'm confused by the experimental setup and goals. \n\nWhat I understand, is that a 2D image is mapped to 3D with biases and preferences like picking up two objects near each other is prioritized and the evaluation is done over varying amounts of clutter until models begin to fail. \nWhat I don't understand is why a heuristic policy (or simple pipeline) would ever fail.  Additional motivation, analysis, and insights would be greatly appreciated.\n\nQUESTIONS:\n1. I apologize to the authors if I've missed something critical but, its easy and fast to predict bounding boxes and masks, so it should be easy to simply rank unoccluded objects and then sort by proximity. Particularly because we are only working with overhead views.  Why is robot pose necessary here or even a bespoke architecture?\n\n2. The fact that in hard scenarios the results only drop to 76% seems to validate my concern that the task is not that difficult to code up? Is this due to the myriad number of ways to succeed? Is there really a good reason to choose one ordering over another? \n\n3. The real world examples appear incredibly simple.\n\nLIMITATIONS:\nSee questions -- more generally I'm unclear on what the constraints are on the system, what the failures look like, what would be necessary to address them, etc.\n\nOne angle I could see is that there's an efficiency concern which limits the use of large VLMs but then I would still expect to see a few simple mask/bbox/yolo/... style baselines that can run at very high frequencies.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors propose a neural network-based approach for the manipulation task of ordered object picking. The use an image backbone on RGB-D images to predict object bounding boxes together with attributes. On top of the bounding boxes and attributes, a neural network is trained to predict picking priority score, which is then used to create an execution plan. The approach is validated both in simulation and real-world experiments, and shows impressive performance in picking success rate, significantly outperforming VLM baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
- a solution to an important real-world robotic problem is proposed
- the solution is relatively simple and does not depend on heavy VLM compute at inference time
- experimental validation consists of tests both in simulation in real world and shows high success rate. I find fascinating how state-of-the-art VLM are able to achieve reasonable performance on an unseen robotics task, yet cannot outperform a task dedicated solution, reinforcing the "jack of all trades, master of none" claim for LLM.
- the system runs at 6-21 FPS which is additionally impressive.

QUESTIONS:
- What is the architecture of the image backbone? I could not find details neither in the paper nor in supplementary.
- does OrderMind create a single plan and execute it in simulation and real-world experiments, or replan every few frames? in case of the latter, are plans stable for rearrangement? is there index shuffle during replan?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03