Learning Spatial-Aware Manipulation Ordering
Authors: Yuxiang Yan
Zhiyuan Zhou
Xin Gao
Guanghao Li
Shenglin Li
Jiaqi Chen
Qunyan Pu
Jian Pu
Venue: NeurIPS 2025 poster ()
Keywords: object manipulation
cluttered environment
Primary area: applications
TL;DR: OrderMind is a spatial-aware framework for manipulation ordering in cluttered environments. It learns object manipulation orderings by encoding spatial relationships. It outperforms existing methods in both simulated and real-world tasks.

Abstract: Manipulation in cluttered environments is challenging due to spatial dependencies among objects, where an improper manipulation order can cause collisions or blocked access. Existing approaches often overlook these spatial relationships, limiting their flexibility and scalability. To address these limitations, we propose OrderMind, a unified spatial-aware manipulation ordering framework that directly learns object manipulation priorities based on spatial context. Our architecture integrates a spatial context encoder with a temporal priority structuring module. We construct a spatial graph using k-Nearest Neighbors to aggregate geometric information from the local layout and encode both object-object and object-manipulator interactions to support accurate manipulation ordering in real-time. To generate physically and semantically plausible supervision signals, we introduce a spatial prior labeling method that guides a vision-language model to produce reasonable manipulation orders for distillation. We evaluate OrderMind on our Manipulation Ordering Benchmark, comprising 163,222 samples of varying difficulty. Extensive experiments in both simulation and real-world environments demonstrate that our method significantly outperforms prior approaches in effectiveness and efficiency, enabling robust manipulation in cluttered scenes.

Learning Spatial-Aware Manipulation Ordering
Yuxiang Yan 1*
Zhiyuan Zhou
Xin Gao'
Guanghao Li
Shenglin Li
Jiaqi Chen
Qunyan Pu²
Jian 1.
Fudan University
Shanghai YinCheng Intelligent CO., LTD
Stanford University
{yxyan22, zhouzy25) fudan edu. en, jianpu@fudan. edu.cc
Abstract
Manipulation in cluttered environments challenging due spatial dependencies among objects, where an improper manipulation order can cause collisions blocked access. Existing approaches often overlook these spatial relationships, limiting their flexibility and scalability. To address these limitations, OrderMind, a unified spatial-aware manipulation ordering framework that directly learns object manipulation priorities based on spatial context. Our architecture integrates spatial context encoder with temporal priority structuring module. We construct spatial graph using k-Nearest Neighbors to aggregate geometric infor- mation from the local layout and encode both object-object and interactions to support accurate manipulation ordering in real-time. To generate physically and semantically plausible supervision signals, we introduce a spatial prior labeling method that guides vision-language model produce reasonable manipulation orders for distillation. We evaluate OrderMind on our Manipulation Ordering Benchmark, comprising 163,222 samples of varying difficulty. Extensive experiments in both simulation and real-world environments demonstrate that our method significantly outperforms prior approaches in effectiveness and efficiency, enabling robust manipulation in cluttered scenes.
propose
Introduction
Object manipulation | in real-world environments is fundamental capability for robots to interact with diverse objects to perform tasks 4. In cluttered environments, objects are often closely packed, partially occluded, or physically constrained by one another [5]. The order in which objects are manipulated can significantly impact both task efficiency and scene stability. Improper ordering may lead to collisions or structural collapse (6). Effective manipulation ordering requires understanding how the spatial layout of objects constrains their interactions. Capturing spatial conditions and inferring manipulation order is essential for enabling safe robotic manipulation.
Most existing approaches for manipulation in cluttered environments do explicitly model ma- nipulation ordering. As shown in Fig. [ [), some systems [7-9 systems apply heuristic algorithm after object recognition. These pipelines are generally limited in their ability to generalize and tend to break down when spatial relations vary across different scenes. More recent methods, shown in Fig. [b), leverage large vasion-l models to guide the manipulation sequence through explicit prompts. While these approaches are flexible, they suffer from long inference times [10], often spanning several seconds, making them unsuitable for real-time deployment. These limitations motivate the need for unified framework that can directly learn spatial-awan manipulation ordering with low latency and high adaptability.
"These authors contributed equally to this work. i Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Object
Order Maninulation
Spatial-Aware Munipulation Ordering
Percepiion
Heuristic
Perception
Explicit Network
Model
Algorithm
Model
Emonder
Spatial
Temporal Priority Structuring
Rocoder
Emender
Decoder
Enonderr
Decodar
Context
Understanding
Robotic
Robotic
Rabotic
Spatial Prior
Order Align
Poss
Pose
Pose
Order Labelling
(a) Heuristic Pipeline
(b) Two-stage Framework
(c) Our Unified Model
Figure Comparison of various designs for manipulation ordering. (a) Heuristic pipelines use hand-crafted optimization algorithms for post-processing. but struggle with generalization. (b) Two-stage frameworks utilize separate networks to predict manipulation order, resulting in increased latency and reduced efficiency. (c) Our unified spatial-aware framework jointly learns spatial representation and manipulation ordering. achieving both high accuracy and real-time performance.
As illustrated Fig. [e), we propose OrderMind. unified framework that learns spatial aware ma- nipulation priorities directly from cluttered environments. It combines spatial context understanding module and a temporal priority structuring module infer manipulation ordering based on local ge- ometry and interactions. A k-Nearest Neighbors algorithm-based spatial graph is proposed to capture object-object and manipllat relations, supporting efficient spatial layout understanding. We introduce spatial prior labeling method that guides vision-language model to generate plausible orderings. This enables training without manual labels. We modify a vision-centric backbone [ for spatial context understanding. By integrating object understanding and order prediction into single inference pass, the model achieves accurate and efficient manipulation ordering, maintaining spatial consistency across diverse cluttered scenes while supporting real- sell-time performance.
To facilitate further assessment, we build the first large-scale simulation Manipulation Ordering Benchmark, which is explicitly designed for the manipulation ordering challenge and comprises 163,222 samples across diverse difficulty modes. We propose set of evaluation metries designed to thoroughly measure model performance, taking into account the validity of inferred manipulation sequences. Additionally, we conduct comprehensive experimental validation both simulated and real-world settings. Our findings confirm the strength of the proposed approach, achieving real-time manipulation ordering in cluttered environments.
Our contributions are summarized as follows: (i) We propose a unified spatial- aware manipulation ordering framework for cluttered environments, which addresses the fundamental challenge of fully leveraging spatial information to learn manipulation ordering. (ii) To implement our proposed strategy, we introduce the OrderMind architecture, which consists of spatial context understanding module and temporal priority structuring module. This design enables efficient, real-time inference of manipulation order. (iii) Extensive experiments in both simulation and real-world robot setups demonstrate the effectiveness and robustness of our method across diverse cluttered scenes.
2 Related Works
2.1 Models for Robotic Manipulation
With diverse modeling paradigms ranging from vision-language reasoning to physics-based dynamics, recent models for robotic manipulation aim to bridge perception and control through both learned and analytical representations. Some vision-language models, such as CI.IPort [12] and PaL--EE [3] extend manipulation capabilities by conditioning on language and integrating multimodal information from vision and proprioception. SayCan [14] introduced affordance-
language grounding. RT-1 [5] and its successor RT-2 [16] re- re-frame actions as text tokens and co-finetune on internet-scale lasion-language corpora, unlocking chain-of Mhain-of-thhughh thought reasoning for semantic and arithmetic object commands. VIMA [17 generalises prompt engineering to robotics for strong zero-shot performance.
ReKep 18 leveraged the power of VLM, specifically GPT-4o 119 to effectively reason about 3D rotations. achieved this by expressing spatial relationships as arithmetic operations between keypoints. Moreover, advances in perception method provide scene reconstruction [20 and object understanding [21] that can used to preprocess visual inputs before manipulation. SeeDo 23 utilizes VLM with chain-of-thouge prompting and visual cues from video tracking to enable effective

---

reasoning for robotic task planning. SpatialVLM 24 enhanced VLM spatial reasoning capabiliiiess
by training on large-scale generated spatial Visual Question Answering dataset. Such perception-
driven priors enhance spatial awareness 25 26 and improve the reliability of downstream control.
RoboMamba [27] integrated a vision encoder with Mamba [28] and Large Language Model to obtain both visual common sense and robotic-related reasoning abilities. Although VLM- models possess general reasoning capability, they are relatively slow and cannot meet the real-time requirements of robotic manipulation tasks.
2.2 Manipulation Relationship Network
Early methods in visual relationship detection focused on building prpiicttpredicat--bbjeed triplets and combining linguistic priors or global context for joint learning. Visual relationship prediction model [29] learned the appearance of objects and predicates individually and used the relationship
embedding space learned from language detect visual relationships. SCG (30) applied interactive message passing for deep object-relation interactions. VRL [3I] used a reinforcement learning- -bsed method to explore multiple relationships and attributes sequentially. CDDN Network [32] utilized word semantics and visual scene graphs to capture the interactions between different object instances for more accurate predictions. Graph R-CNN [3 leverages object-relationship regularities to
generate scene graphs from detected objects in an end-to-end manner. Motifnet (34 staged detection
results and relationships establish rich context for subsequent prediction.
To further integrate the physical order of actions into robotic manipulation. VMRN 335 proposed a novel pooling layer to predict manipulation relations in parallel. GVMRN 36 adopted GCNs to enhance contextual reasoning and filter irrelevant pairs, boosting inference robustness. MRG [37 used affordance representation get the potential manipulation relationships of arbitrary scene. VeriGraph (88 generated high-level task plans for robot object manipulation based on the scene graph. D3G 39 detected objects and reasoned relationships for grasp planning simultaneously by leveraging a transformer-hoe
detector and graph layers. Although these models are capable of generating object relation graphs that can be used for manipulation tasks, they still focus on perceiving relationships between objects, ignoring the relationships between objects and the robot, and still require further post-processing obtain the manipulation order.
3 Methodology
As illustrated in Fig. 2 we propose OrderMind, unified framework for learning spatial-awan ma- nipulation ordering cluttered scenes. OrderMind integrates scene comprehension with manipulation ordering, enabling robots to perform real-time manipulation ordering in cluttered environments. This section describes how we define order-aware representation (Sec. 3.1.. how to learn the spatial-aware manipulation ordering (Sec. 3.2). and how to construct reliable order annotations (Sec. 3.3)
3.1 Problem Formulation
We define the manipulation ordering challenge as learning mapping f I x SE - O, where I I HRHWx x1 denotes the space of RGB-D images, SF(3) represents the pose space of the robotic end-effector, and = E). The output O comprises two key components: a set of object representations O {or, (0,,02,- ... ......... o,). and manipulation sequence Σ {01, 02,........ J that defines the operational priority order.
Each object representation o; integrates both intrinsic attributes such as extent and category, and extrinsic attributes such as position and orientation, collectively capturing the spatial context necessary for reasoning about manipulation order. The model assigns each object continuous priority score 8; E IR based on these spatial-awan representations, and the sequence Σ is derived by sorting these scores. This approach enables fine-grained differentiation between manipulation priorities rather than simple ordinal rankings, particularly valuable when certain objects require significantly higher operational precedence.

---

Image Backbone
Proposal Head
RGB
Object Rol Queries
T
Ordering
Extrinsic Attributes
Module
Depth
Image Feature
Intrinsic Attributes
Ordered Object Detection
Spatial Context Understanding
Temporal Priority Structuring
Max
K.
Effector Pose
Relative Attributes Extrinsic Embodding
N
Cross- Atfention Block X
Priority Senre
Order Sorting
Extrinsic Attributes
- Fusion(f..f)
Object Order Queries
Self Atlention
Mi.
Intrinsic Attributes
Intrinsic Embedding
ResNet Block
Object Rol Queries
Visual Embedding
Concatenate Elementwiise Add
Max Max Pooling
Latent Features
Image Global Token
Figure Main framework of OrderMind. OrderMind captures spatially manipulation order- ing representations that encode both object-object relationships and object-manipulator interactions in cluttered environments. The spatial context understanding module aggregates spatial information from local geometry, while the temporal priority structuring module integrates local object features with global scene context. OrderMind predicts the manipulation priority for each object, and by obtain the manipulation order.
sorting these priorities,
3.2 Spatial-Aware Manipulation Ordering
To support optimal prediction of manipulation order, design both spatial context understanding module and temporal priority structuring module, which tightly integrate spatial-aware arrangements and manipulation ordering.
Spatial Context Understanding. Occlusions and inter-object support in cluttered scenes make necessary to reason about spatial and physical relationships rather than visual appearance alone. To enable accurate manipulation ordering cluttered environments, we focus on extracting spatial-aware representations that capture both local and global scene structure. We represent each object using its 3D bounding box center and associated intrinsic and extrinsic attributes. The intrinsic attributes describe inherent properties such semantic class and physical extent, while the extrinsic ones define the object's pose in the world coordinate system. The object centers collectively form a sparse point cloud representing the 3D scene layout. Based this point cloud, we construct spatial graph where nodes correspond to objects and edges encode geometric proximity.
To model inter-object spatial dependencies, localized spatial graph around each object. As shown in Fig. 3 we treat the center point of each object node in the graph and connect each object to its neighbors with edges. Each node in this graph incorporates both intrinsic and extrinsic attributes. For each center p;, messages from its spatial neighbors N& (p:) aggregated to form spatial-aware features that encode local geometric structure, as described in Equation []
apply k- k-Nearest Neighbors strategy to build
Fusion (fr. fj) M(Linea((Cnncatefffff f()),, Vp, E (p.)),
(1)
This design captures how physical proximity and spatial alignment influence the feasibility of manipulation actions. In line with PointNet-style architectures [40] feature aggregation is performed using pooling M across neighborhoods to produce compact object-level embeddings.
In addition to object-object interactions, we model the spatial relationship between each object and the robot. This is encoded by computing the relative transformation between each object's pose and the end-effector's current state, providing important cues for operational accessibility. These robot-centric features are integrated into the object representation to further refine the spatial-aware

---

Message Propagation
Hash Tablc
Lift to 3D
Center Query
Linear & Max
Spatial Graph Construction
Neighbor Queries
Updated Query
Figure 3: Details of Spatial Context Understanding Module. A spatial graph is constructed
from object-centric point clouds to capture scene layout in cluttered environments. Local geometric structure is extracted using a k-Nearest Neighbors strategy, and spatial-aware features are aggregated through neighborhood pooling to support manipulation ordering.
embedding. By jointly encoding object geometry, visual cues, and robot-relative spatial structure, our method constructs rich spatial-aware representation that supports effective manipulation ordering in cluttered environments.
Temporal Priority Structuring. Once spatial-aware features are extracted, the model transitions from structural perception to manipulation-oriente ordering by learning to prioritize objects according to their contextual importance. To this end, we design an attention mechanism to dynamically integrate object-specific information with scene-level cues.
Firstly, a set of object ordering tokens extracted from the image encoder is aggregated via global max pooling form a high-level scene representation, denoted as G. This global context encodes the layout of the scene, capturing occlusion patterns, object stacking, and spatial symmetry. Each object token then refined through self-attention, which incorporates inter-object interactions and aligns the object tokens with one another. Formally, the attention flow is defined as:
IQ's G + Self-AAt((QQ,, GJ), Q" +- Cross--AttQ'' [G, (G,F),
(2)
where [ denotes the concatenate operation, F denotes the original visual features and Q' represents updated object embeddings after intra-token refinement. Secondly, the cross-aftention module allows each object token to query both the global context G and its associated visual representation F, enabling the model extract manipulation-rellvan cues while maintaining spatial coherence. Finally, the ordering token are decoded into a set of priority scores, which represent the manipulation priority for each object under the current spatial layout.
The output Q" encodes spatial-aware priority for each object, implicitly modeling manipulation priority under geometric and physical constraints. By leveraging structured attention, method enforces consistency between local feasibility and global accessibility, resulting in manipulation orderings that are both spatially grounded and temporally executable.
Preferential Order Alignment. To learn a spatially-aware manipulation ordering from dis- crete order annotations, we first align predicted and ground-truth objects via bipartite match- ing. Let o and ó denote ground-truth and predicted objects, respectively. We formulate the assignment using the Hungarian algorithm, where object similarity serves as the matching cost
á arg min Σ Lhungariaa (oi, (0)) When the object correspondence â established, we align the
prediction of ordering preference with ground truth order by comparing pairwise relations. Instead of directly predicting discrete ranks, the model leams to produce continuous scores & to implicitly align sorted order with the annotated order. The objective is expressed as follows:
N
N
Londer ΣΣ
Wijk [o,<<o)) log(1 (1 + exp($5(1) - $a())).
J=1 k=1
where Wikk = log(1 + +1o, ox) is a logarithmic weighting term that emphasizes ordering consistency for object pairs with large ground-trufe rank differences.
This formulation allows the model to infer object priority through relative comparisons in the score space, rather than committing to exact rank values. The transition from discrete index annotations to continuous score predictions enables the model to capture more reasonable preferences in manipula- tion order, especially in cases where certain objects dominate the ordering due to spatial accessibility
5

---

or structural dependencies. By sorting these predicted scores, the final output ordering naturally aligns with spatial constraints and task demands.
3.3 Spatial Priors Order Labeling
Learning spatial-aware aware manipulation ordering relies on well-structured annotations that reflect object interaction priorities in cluttered environments. In our framework, these manipulation orders generated by a VLM [42]. To enhance the quality and consistency of these labels, we introduce two spatial priors to serve as auxiliary signals during the generation process. These priors influence how the large model interprets spatial relationships, ensuring that the predicted orderings align with common physical and operational patterns observed in cluttered scenes.
This prior encourages the early manipulation of objects that are spatially separated from others on the horizontal plane. In cluttered environments, such independent objects can be accessed more safely, as their removal is unlikely to cause physical interference with neighboring items. For each object o, O, compute its projected area A, by projecting its 3D bounding box onto the horizontal support surface. The pairwise distance between two objects is defiined as d(A; Aj) in Ilp; pyll2 2 pi E Ac, Pj E A₁ ). An object is considered spatially independent when the minimum distance from A; to all other Aj is no less than threshold т, which is selected based on the size of the suction end-effector and required safety margin:
Independence Prior.
1, if min d(A,((,, Aj) j71
Tindep (Oi; O)
0, otherwise
where objects satisfying this condition are generally safe to manipulate first, providing useful spatial cues for generating physically consistent manipulation order labels.
This prior captures vertical accessibility by identifying objects that are not occluded from above. In cluttered environments, vertical stacking is frequent, and selecting objects from the top layers helps avoid disturbing other items and maintains the scene's 's stability. For object o, O and its 3D bounding box B;, we define zmax(o.) (o.) = sup {z (x, y, 2) E B.} as the height of its topmost point. The volume above o, is defined as the upward extension of its projected area A;
Local Optimality Prior.
Vabove (o.) (z. y, 2) R (z, E A, A > 2max mmax(o:)) (o:))
(5)
An object is locally optimal when no other object intersects with this vertical space:
1, if vk + i, Bg O Valove =
Tromost (o;, 0)
(6)
0, otherwise
where objects that satisfy this criterion are likely to be directly accessible from above. This prior provides useful spatial context to help the model generate orders that respect scene stability and reduce structural disruption during manipulation.
Experiments
This section evaluates framework for its effectiveness in manipulating ordering in cluttered scenes. Section 4.1 4.1 provides the details related to the image datasets and implementation. Section presents the results obtained with our framework and compares them with other related approaches. Finally, section contains ablation studies to investigate the effectiveness of each proposed component.
4.1 Experimental Settings
4.1.1
Environments and Task
We constructed datasets involving complex robotic manipulation scenarios in both simulated and real- world environments. For simulation, we utilized Pybullet [43 as the engine and designed cluttered scenes using the YCB object set [44). Three levels of difficulty were established: Easy with 24 objects, Medium with 36 objects, and Hard with 60 objects. The simulation dataset consists of 161,722 RGB-D images for training, 1,500 images for validation. Closed-loop testings are performed within the Bulllt simulator. For real- world experiments, we collected 26,324 images for training and 6,581 for validation. Closed loop testings are directly conducted in real-world cluttered environments.

---

Table I: Manipulation performance under easy, moderate, and hard settings in the simulation environment. RC, OD and SR refer to residual count, object disturbance and success rate, respectively. Privilege indicates models are accessible to objects' ground truth information. N/A indicates that the model's parameter size is not reported.
Fasy
Pramework
Privilege
#Param.
FPS
4PT-4 19
4.5&1.0
2.380088
9033264
9,321.3
5,0E.0
77.9.8.4
168822
12082.
71,258.3
NEA
0.1
Clande 3.7 Sonter 3P
0.740.6
2.069.4
92.1-7.1
22211]
4.511.1
34.0=7.1
53522
10.4225
77.98.3
NEA
0.1
Gamini-2
0078003
1640.7
92.499.7
12215
484215
5464108
5350.1
12.45.0
78.5416.5
NEA
0.1
Gemma-3 33
1.28122
222312
58.3:112.6
2.811-
49418
63.2-108
7.588
11223..3
706:9933
27B
0.1
Owent.s-Vi 5-Vi
0.900.0
V253193
2.7615
458111
1:7887
114334
70.81100
778
0.1
21.0
58.2211099
48222
704488.
YOLOx
3.482.1
7492622
11.9
UniDe:30 Uomiinne
3.183.6
2.580.8
494:117:0
4.5-2.
4381.
530-79
9.7325
8,43366
46.12109
15.31
0.8
UniDer Dieinne
2.345.0
2.410.7
40.3-113.
4.--1.9
5.591.5
348-TA
14.133
128139
24.933
15.81
0.8
UniDecD--GPPE GPF-6
7.283.6
20060.8
42.8.7.1
1200:25
35:00.
T9
23.033
9.23322
35.955.6
ISM-NNA
0.1
L InieetDDCCardeee 2-5pen- 7-Sonnet
4.78::.6
2.8.:111
399:6.
6:5-2.9
0::00-3
1601119
185510
16.281.5
30.7557
ISM-N/A
0.1
UnDecDD Gemizi 2,5
3.2.2..0
3.063..2
45:5:20.1
5.6:3.5
$.5218
46.21122
165137
131122.1
26.622
ISM-NNA
0.1
3.140.9
40:3-150
5.5-9.3
33911.
43-2-677
153+50
135133
158-559
ISM+2716
3.543.2
38.-:11.
7.--26
7.1-1.7
1999:000
28.3-5.
ISM-7218
UniDecDD
2,5 23.2
2.963.1
38.3810.1
7.161.7
5.561.5
11,421.33
43.1294
OrderMiid Mini
L081.
94.26766
3.321.5
3,6410
896627.1
54513
7,243.4
90.429.5
21.3
OrderMiad
0.4:8.5
96.5533
14:085
4,3115
95.4:5.1
The task is to pick up objects sequentially in scenes with densely stacked items. requires maintaining a high success rate while minimizing disturbance to surrounding objects. OrderMind replans after ensure robustness against disturbance from object interactions. The robot
each manipulation interacts with objects using a suction-based end-effector. An RGB-D camera, positioned in a top- down orientation above the workspace, captures visual input at resolution of 1408 x 1024.
4.1.2 Training and Implementation Details
We trained model
single RTX 4090 GPU using batch size of 24. The optimizer is AdamW 44 with an initial learning rate of x 10-1 and a weight decay of 0.01. The learning rate follows cosine annealing schedule [46. Training proceeds in two stages. The first stage focuses
on pre-training the image backbone [47 48) for 50 epochs. In the second stage, the full model is
trained end-to-end for 10 epochs, enabling the system to learn order-aware manipulation directly from sufficient spatial representation.
4.1.3 Evaluation Metrics
We evaluate performance using three metrics: higher success rates, lower object disturbance, and fewer residual objects, which reflect more effective manipulation order.
Success Rate. This metric is defined as the ratio of successful pick-ups to total attempts. In simulation, pick is considered successful when the 6D pose of the end-effector pe. including both its center and orientation, closely aligns with the target object pose Patj- Specifically, success is confirmed when the positional difference satisfies l|p. Patjl 2 0.05.
Residual Count. This metric tracks the number of objects left outside the robot's reachable workspace at the end of task. A smaller count indicates that the robot maintained optimal manipulation ordering during the task. While all objects start within reach, due to unintended displacement during may shift out of range and remain unprocessed.
manipulation.
Object Disturbance. In cluttered environments. manipulating one object can cause disturbance to nearby objects. After each action, the total displacement of surrounding objects is measured. Lower disturbance indicates that a well-ordered manipulation sequence is being operated by the robot, ensuring the scene remains stable during each step of order-aware manipulation.
4.2
Experimental Results
4.2.1
Simulation Experiments
Tab. presents a comprehensive evaluation of manipulation performance in cluttered environments across four experimental settings: heuristic pipeline, two-stage framework, privilege information- based two-stage framework, and our proposed unified spatial-aware framework. Privilege information
7

---

Table Comparison of order stability and success rate under different replanning intervals. Order-
Mind maintains lower Levenshtein Distance (LD) and higher Success Rate (SR).
Interval
Framework
Privilege
LDI
SRT
GT+SPH
1.5
96.3
YOLO11-seg+SPP
X
3.6
77.5
YOLO11-det-SPP!
X
1.8
79,8
OrderMind (Ours)
x
1.7
89.6
5 GT + SPH
J
5.2
85.5
YOL011-seg+SH
X
6.6
75.3
YOLO11-det+SPP
X
72.2
OrderMind (Ours)
X
3.5
82.3
10
+ SPH
V
7.5
85.5
YOLO11-seg+SPP
X
9.3
70.6
X
9.3
69.0
OrderMind (Ours)
X
4.7
80.5
refers to models having direct access ground truth object pose information in the simulation environment. We derive our Spatial Priors (Sec. 3.3 as the Spatial Priors Heuristic (SPH) for complete comparison. The speed of VLMs calculated by using OpenRouter [55].
We first evaluate the upper bound of vision-language reasoning using VLMSs with privileged in- formation. Even with privileged information, the best model achieves only 92.5% success in easy setting and 78.5% in hard setting. Next, we test heuristic pipelines using YOLOv1 (53] and UniDet3D [54], selecting objects by Spatial Priors Heuristic, confidence, and distance the end effector. Although UniDet3D and YOLO has fewer parameter counts, it is limited to 0.8 FPS due to its point cloudbbaeed architecture. We also evaluate two-stage frameworks where UniDet3D results are passed to VLMs for ordering, which achieves only 0.1 FPS and cannot meet the call-time demand.
Our OrderMind framework learns order-aware manipulation directly from visual inputs, enabling precise sequence planning and stable performance in cluttered environments. Its core strength its spatial understanding, which anticipates cascading effects to minimize disturbance surrounding objects. OrderMind yields superior performance as it achieves 96.5% success rate in easy mode, outperforming privileged VLMs, and maintains 95.4% success rate in hard mode. In contrast, Gemini-2.5's performance drops to 78.5%, which demonstrates that conventional VLMs struggle to infer stacking relations and plan effective sequences. Ultimately, OrderMind's is ability to learn spatial order and minimize disruption provides a clear advantage in cluttered robotic tasks. Furthermore, the lightweight OrderMind-Mini model delivers superior accuracy in real-time (21.3 FPS) with only 35.2M parameters, confirming the framework's strong potential for real-world deployment.
Order Stability Analysis. To examine the effect of replanning frequency plan stability and task performance, we compare OrderMind against heuristic baselines under various replanning intervals. We the Levenshtein Distance (LD) quantify plan stability, where larger LDD indicates greater order reshuffling between consecutive plans.
As shown in Tab. 2 our method delivers more stable plans and higher success rate compared to heuristic baselines. With infrequent replanning. these baselines produce large order reshuffling reflected by larger LD, and suffer drop in success. However, our approach consistently outperforms the two-stage framework in all tested intervals, achieving both a lower LD and a superior success rate.
We attribute this superior performance to our unified design that jointly learns spatial representations and manipulation order. Heuristic methods are often greedy, i.c., they select the best immediate action but fail to foresee long-term consequences. In contrast, OrderMind distills knowledge from VLMS and integrates rich visual and geometric context. This allows to generate globally coherent and far-sighted plans, making them inherently more stable and successful.
Robustness under Simulated Label Noise. To assess our OrderMind's robustness unexpected noise from VLMs such as hallucinated spatial relations, incorrect labels, or omissions, we conducted study by randomly perturbing certain percentage of pairwise orders in the VLM labels.

---

Figure 4: Visualization of manipulation ordering results under easy, moderate, and hard settings. 3D bounding boxes are demonstrated and manipulation orders are marked at the center of each object.
21
14
22
10
20
12
24
8
15
13
25
23
10
Figure 5: Visualization of real-world manipulation ordering. The left figure shows results from experiments conducted laboratory setting, while the right figure presents results obtained in an actual factory environment. Predicted order is annotated on each object in the scene.
Tab. 3 reports success rates on scenes of varying difficulty under different noise ratios. The per- formance degrades slightly with moderate noise, demonstrating the robustness our method to VLM label noise, thanks to both label ensemble strategy and the unified learning of perception and ordering. However, under the extreme 70% noise condition, the performance drops substantially. This likely occurs because, at such a high noise ratio, the model fails to extract meaningful information and effectively degenerates into random policy. In moderate mode, the success rate also declines due to perception noise-induced errors.
Table 3: Comparison of success rates under varying levels of simulated label noise.
Noise Ratio
Privilege
3R-Easyt
SR-Modf
SR- ER-Hardf
85.35
83.19
83.66
0.1
X
85.10
82.71
79.13
0.2
80.08
79.00
76.30
0.5
78,88
76.91
73.81
0.7
75.99
70.36
67.31
GT+random
75.50
75.58
68.88
4.2.2 Real World Experiments
Tab. 4 shows the performance of our OrderMind model in real- world experiments across three levels task difficulty. As the task difficulty increases from easy to hard, the success rate decreases from 93.3% to 76.6%, while the residual count rises from 0.2 to 3.0. This reflects the growing challenge of planning and execut- ing actions in denser and more constrained object arrangements. Despite this, the model continues to demonstrate strong ability to complete tasks with relatively high success rates and limited residual objects.
Table 4: Performance of Order-
Mind in real-world closed-loop test- ing under three difficulty modes.
Mode
RC
SR (%)
Easy Moderate Hard
0.2
93.3
2.0
78.5
3.0
76.6
Fig. 5 iillustrates the predicted manipulation sequences in real-world scenes, with numbers directly annotated on the objects to indicate their execution order. The results show that our model can effectively understand both object isolation and stacking relationships. Guided by the learned sequential policy, the robot interacts with objects efficiently and safely, while preserving scene stability. This capability is essential for enabling reliable and autonomous robotic operation in real-world environments.

---

Failure Analysis. Reviewing 30 minutes of robotic arm operation, we found four main failure sources. Perception eπrоrs were primary contributor, with 21% % from inaccurate 3D rotation and another 15% from object center point misidentification. For challenges with deformable objects, an inability to find suitable suction surfaces caused 21 % of the failures. Furthermore, incorrect manipulation ordering led to object disturbances, accounting for 39%. Other factors, like poor synergy between the robotic arm and the camera, accounted for 4%.
4.3 Ablation Study
We conduct an ablation study to assess the contribution of each core component in our framework, as demonstrated Tab. Starting from bascline with 5.0 residual objects, 5.4 total displacement and success rate of 76.1 12, we observe steady improvements with the addition of each module.
Introducing either SCU TPS individually boosts the success rate to 81.0% and 80.5% while also re- ducing residual objects and disturbance. Their combi- nation yields further gains, with success rate reaching 87.7%, suggesting that accurate feature association and context- selection complement each other improving manipulation quality. Adding the SPOL component leads to the most substantial improve- ments. For example, the combination of SCU and TPS lifts the success rate to 91.4% and reduces the number of residual objects to 2.0, demonstrating the importance of learning informed action sequences in cluttered environments.
Table 5: Ablation study of proposed modules mod- erate task level. SCU, TPS, SPOL denote spatial understanding, temporal priority structuring, spa- tial priors order labeling, respectively.
SCU
TPS
SPOL
RC +
OD J
SR(%) f
5.0
5.4
76.1
3.6
81.0
5
80.5
3.5
4.4
87.7
2.0
4.0
91.4
90.5
95.3
Finally, the full model with SCU, TPS, and SPOL achieves the best performance with only 1.0 residual object, and minimal disturbance at 1.4. These results highlight the strong synergy among components -eaturr association, object selection, and order planning together lead to efficient and stable manipulation.
Conclusion and Limitation
In this work, we presented OrderMind, a framework for learning spatial-aware manipulation ordering tailored to robotic tasks in cluttered environments. OrderMind addresses this by integrating spatial context understanding and temporal priority structuring, enabling reliable and efficient manipulation without reliance on idealized inputs or environment models. We demonstrate that OrderMind achieves high success rates, minimal object disturbance, and low residuals across both simulated and real- world cluttered environments, all while operating in real time. We found that while large models provide strong general scene understanding and ordering capabilities, their reasoning remains limited when confronted with the fine-grained spatial dependencies and object stacking present in cluttered settings.
Broader Impact. OrderMind can improve the reliability and autonomy of robots in cluttered environments, enhancing efficiency and safety sorting, warehouse logistics, and assistive robotics. It can reduce manipulation errors, making automation more sustainable and effective. The spatial- aware encoding ability also enables easier deployment in unstructured settings, broadening access to intelligent automation in healthcare, manufacturing, and services, while inspiring research in human-robot collaboration.
Limitations. Despite these advantages, several limitations remain. The current system assumes stable scene during execution. which may restrict its ability to adapt to dynamic environments. Besides, the accuracy of order-aware manipulation prediction relies on precise 3D attribute estimation, which remains challenging under severe occlusions common cluttered environments.
10

---

References
[1] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi, "Manipulathor: A framework for visual object manipulation," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 4497-4506.
[2] H. Yin, A. Varava, and D. Kragic, "Modeling, learning, perception, and control methods for deformable object manipulation," Science Robotics, vol. 6, no. 54, p. cabd8803, 2021.
[3] M. Shridhar, L. Manuelli, and D. Fox, "Perceiver-actor: A multi-task transformer for robotic manipulation." in Conference Robot Learning. PMLR, 2023, pp. 785-799.
[4] J. Eom, S. Y. Yu, W. Kim, C. Park, K. Y. Lec, and K.-J. Cho, "Mogrip: Gripper for multiobject grasping in pick-and-plaa tasks using translational movements of fingers," Science Robotics, vol. 9, no. 97, p. eado3939,, 2024.
[5] M. Danielczuk, A. Mousavian, C. Eppner. and D. Fox, "Object rearrangement using learned implicit collision functions," in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 6010-6017.
[6] A. Xu. K. Gao, S. W. Feng, and J. Yu, "Optimal and stable multi-layer object rearrangement on tabletop." in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 2078-2085.
[7] D. Saxena and M. Likhachev, "Planning for manipulation among movable objects: Deciding which objects go where, in what order, and how," in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 33, 2023, pp. 668-6668 666.
[8] J. A. Haustein, K. Hang. J. Stork, and D. Kragic, "Object placement planning and optimization for robot manipulatste manipulators," in 2019 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019, pp. 7417-7424.
[9] H. Karami, A. Thomas, and F. Mastrogiovanni, "A task and motion planning framework using iteratively deepened and/or graph networks," Robotics and Autonomous Systems, vol. 189, p. 104943, 2025. [Online]. Available:
[10] I. Kapelyukh, Y. Ren, I. Alzugaray, and E. Johns, "Dream2real: Zero-shot 3d object rearrangement with vision-language models," in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 4796-4803.
[11] X. Zhou, D. Wang, and Krähenbühl, "Ohjects points," arXiv preprint arXiv:1904.078500 2019.
[12] M. Shridhar, L. Manuelli, and D. Fox, "Cliport: What and where pathways for robotic manipulation," in Conference on robot learning. PMLR, 2022, pp. 194-906.
[13] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q Vuong. T. Yu et al., "Palm-e: An embodied multimodal language model," in International Conference on Machine Learning. PMLR, 2023, pp. 8469-8488.
[14] M. Ahn, A. Brohan, N. Brown, Y. Chehotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al., "Do as i can, not i say: Grounding language in robotic affordances," arXiv preprint arXiv:2204.0 0160000,019,..... 2022.
[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar. J. Dabis, C. Finn, K. Gopalakrishnan. K. Hausman, A. Herzog, J. Hsu et al., "Ri-1: Robotics transformer for real-world control at scale," arXiv preprint arXiv:22 12.02227.. 2022.
[16] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn al., "Rt-2: Vision- kanguage-action models transfer web knowledge to robotic control," arXiv 2023.
preprint Xiv::23077
[17] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen. L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, "Vima: General robot manipulation with multimodal prompts," arXiv preprint arXiv:2210.03094, vol. no. 3, p. 6, 2022.
[18] W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei, "Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation," arXiv preprint arXiv:2409.01652, 2024.
[19] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint wrXiv:2303.08774, 2023.
11

---

[20] Y. Yan, B. Liu, J. Ai. Q. Li, R. Wan, and J. Pu, "Pointssc: A cooperative cloud benchmark for semantic scene completion." in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 17 17027-170344
point
[21] J. Guo, X. Gao, Y. Yan, G. Li, and J. Pu. "Dark-isp: Enhancing raw image processing for low-light object detection," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.
[22] X. Gao and Pu, "Deep incomplete multi-view learning via cyclic permutation of vaes." in The Thirteenth International Conference on Learning Representations, 2025.
[23] B. Wang. J. Zhang. S. Dong, I. Fang. and C. Feng. "Vlm see. robot do: Human demo video to robot action plan via vision language model." arXiv preprint arXiv:2410.08792. 2024.
[24] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L.. Guibas, and F. Xia, "Spatialvim: Endowing vision- language models with spatial reasoning capabilities." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14455-114665
[25] G. Li, Q. Chen, S. Hu, Yan, and Pu, "Constrained gaussian splatting via implicit tsdf hash grid for dense rgb-d slam," IEEE Transactions on Artificial Intelligence 2025.
[26] G. Li, Y. Can, Q. Chen, X. Gao, Yang, and Pu, "Papl-slam: Principal axis-anchored monocular point-line slam," IEEE Robotics and Automation Letters, 2025.
[27] J. Liu, M. Liu, Z. Wang, L. Lee, K. Zhou, An, S. Yang. R. Zhang, Y. Guo, and S. Zhang, "Robo- mamba: Multimodal state space model for efficient robot reasoning and manipulation." arXiv preprint arXiv:2406. 04039, 2024.
[28] A. Gu and T. Dao, "Mamba: Linear-time sequence modeling with selective state spaces," arXiv preprint arXiv:2312.00752. 2023.
[29] C. Lu, R. Krishna, M. Bernstein, and I.. Fei-Fei, "Visual relationship detection with language priors." in Computer Vision-FCCV 2016: 14th European Conference, Amsterdam, The Netherlands, October l 11-14, 2016, Proceedings, Part I 14. Springer. 2016. pp. 852-869.
[30] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei, "Scene graph generation by iterative message passing." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5410-5419.
[31] X. Liang, L. Lee, and E. P. Xing, "Deep variation-structuree reinforcement learning for visual relationship and attribute detection," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017. pp. 848-857.
[32] 7. Cui, C. Xu, W. Zheng, and J. Yang. "Context-deeendeet diffusion network for visual relationship detection," " in Proceedings of the 26th ACM international conference on Multinedia, 2018, pp. 1475-1482.
[33] J. Yang, Lu, S. Lee, Batra, and Parikh, "Graph r-enn for seene graph generation," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 670-685.
[34] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi, "Neural motifs: Scene graph parsing with global context," in Proceedings of the IEFE conference on computer vision and pottern recognition, 2018, pp. 5831-5840.
[35] H. Zhang, X. Lan, X. Zhou, Z. Tian, Zhang, and Zheng, "Visual manipulation relationship net- work for autonomous robotics," in 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids). IEFE, 2018, pp. 118-125.
[36] G. Zuo, J. Tong, H. Liu, W. Chen, and Li, "Graph- visual manipulation relationship rcasoning network for robotic grasping," Frontiers in Neurorobotics, vol. 15, p. 719731, 2021.
[37] C. Tang, J. Yu, W. Chen, and H. Zhang, "Relationship oriented affordance learning through manipulation graph construction," arXiv preprint arXiv:2 10.22/1011444.. /4137, 2021.
[38] D. Ekpo, M. Levy, S. Suri, C. Huynh, and A. Shrivastava, "Verigraph: Scene graphs for execution verifiable robot planning." arXiv preprint arXiv:2411.1044 10446, 2024.
[39] P. Rabino and T. Tommasi, "A modern take on visual relationship reasoning for grasp planning." IEEE Robotics and Automation Letters, 2025.
[40] C. R. Qi. H. Su, K. Mo, and L. J. Guibas, "Pointnet: Deep learning on point sets for 3d classification and segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652-660.
12

---

[41] X. Yu. Y. Rao, Z. Wang. J. Lu, and J. Zhou, "Adapointr: Diverse point cloud completion with adaptive geometry-aware transformers." IEEE Transactions Puttern Analysis and Machine Intelligence, vol. 45, no. 12, pp. 14114-14 130, 2023.
[42] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, Song. K. Dang, P. Wang. S. Wang. J. Tang. H. Zhong, Y. Zhu, Yang, Z. Li, J. Wan, P. Wang. W. Ding, Fu. Y. Xu, J. Ye, X. Zhang, Xie, Z. Cheng, H. Zhang, Z. Yang. H. Xu, and J. Lin, "Qwen2.5-v technical report," arXiv preprint arXiv:2502.1392. 2025.
[43] E. Coumans and Bai, "Pybullet, a python module for physies simulation for games, robotics and machine learning." lttt::/pyybuller.ongg 2016-2021)
[44] B. Calli, A. Singh, A. Walsman, Srinivasa, Abbeel, and A. M. Dollar, "The yeb object and model set: Towards common benchmarks for manipulation research," in 2015 International Conference on Advanced Robotics (ICAR), 2015, pp. 510-517.
[45] Loshchilov and F. Hutter, "Decoupled weight decay regularization," in International Conference on Learning Reprexentations, 2019.
[46] "Sgdr: Stochastic gradient descent with warm restarts," in International Conference on Learning Representations, 2022.
[47] F. Yu, D. Wang. E. Shelhamer, and T. Darrell, "Deep layer aggregation," in Proceedings of the IEEE conference compuder vision and puttern recognition. 2018, pp. 2403-2412.
[48] X. Zhou, V. Koltun, and P. Krähenbühl, "Tracking objects as points," in European conference on computer vision. Springer, 2020, pp. 474 490.
[49] Anthropic, "Claude 3.7 sonnet and claude code," http:///wwwwattng.ccccoo///////s 2025.
Feb.
[50] G. DeepMind, "Gemini 2.5 flash: Best for fast performance on complex tasks," https://deepminndgoolll. technologie//geminnlllaahl Apr. 2025.
[51] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej. S. Perrin, T. Matejovicova, A. Ramé, M. Rivière et al., "Gemma 3 technical report," arXiv preprint arXiv:2503. 2tt.::2200...786,, 2025.
[52] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Yc, L. Gu, Y. Duan, H. Tian, W. Su, J. Shao et al., "Interv13:: Exploring advanced training and test-time recipes for open-source multimodal models," arXiv preprint TrXiv:2504110499 2025.
[53] Ultralytics, "ulralyties/ulrallyiss Ultralytics yolo," hhtp:///ghhaa..cmmmmlllacccccaau
2025.
[54] M. Kolodiazhnyi, A. Vorontsova, M. Skripkin, D. Rukhovich, and A. Komushin, "Unidet3d: Multi-dataset indoor 3d object detection," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 39, no. 4, 2025, pp. 4365-4373.
[55] Openrouter, "The unified interface for llms." hnttps///openroutrraai Feb. 2025.
[56] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehcad, A. C. Berg, W.-Y. Lo et al., "Segment anything," in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015-4026.
[57] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from rghd images," in ECCV, 2012.
[58] M. A. Fischler and R. C. Bolles, "Random sample consensus: paradigm for model fitting with applications image analysis and automated cartography," Communications of the ACM, vol. 24, no. pp. 381-395, 1981.
[59] H. Zhu, Y. Wang. D. Huang, W. Ye, W. Ouyang, and T. He, "Point cloud matters: Rethinking the impact of different observation spaces on robot learning," " Advances Neural Information Processing Systems, vol. 37, pp. 77 799-77 830, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have stated the challenge, our method and contribution in both abstract and introduc- tion.
Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitation in Sec.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not include theoretical results in our paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: We discussed the training implementation details in Sec. 4.1.2 and dataset construc- tion details Sec, 4.1.1
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide reproduciblc code along with detailed instructions, ensuring that the main experimental results can be faithfully reproduced.
6. Experimental sefting/details
Question: Does the paper specify training and test details (e.g.. data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specify the training implementation details Sec. 4.1.2 and dataset construction details in [.I.I
Experiment statistical significance
Question: Does the paper report enror bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: do experiments times and calculate mean and standard deviation, as demonstrated Tab,
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We have specified the experiments compute resources details in Sec. 4.1.2
9. Code ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Code Ethics
Answer:
[Yes]
14

---

Justification:
We have read and adhered to the NeurlPS Code of Ethics in conducting our research.
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the first paragraph in Sec. have discussed the broader impacts of work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: There are no such risks associated with our work.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Sec. and Sec. g we clearly specify the assets used in our work, including datasets and models. We provide proper citations the original papers and ensure that the licenses and terms of use explicitly mentioned and respected.
13. New assets
Question: Are new assets introduced paper well documented and the documentation provided alongside assets?
Answer: [Yes]
Justification: We provide our code as an asset and include comprehensive documentation alongside covering details such as training procedures, licensing, limitations, and any necessary consent ohtained from individuals whose assets are used.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does involve crowdsourcing experiments research with human subjects.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: Our work does not involve human subjects.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original. or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing. formatting purposes does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use VLM to generate ordering label for robotic manipulation, and also use VLMs for experiment comparision in Sec.
15

---

A Dataset Analysis
This section details the object categories featured in our dataset. In the simulated environments, incorporated five categories derived from the YCB dataset [44] namely CrackerBox, GelatinBox, PottedMeatCan, SugarBox, PuddingBox. our real-world dataset, we employed the two broader categories of box and bag. Objects classified as boxes generally possess a firm shape, facilitating perception. Conversely, bag objects are prone to deformation, thereby posing greater challenge to accurate spatial layout perception.
15.0%
30, 30.7%, (84.476]
200%
60, (1206,20)
Box
ES11
aa) Simluation Dataset
(h) Real-World Dataset
Figure 6: Visualization of the dataset category distribution pie chart, with object counts shown in
parentheses.
B Data Annotation Pipeline
This section details methodology for generating spatial layout perception labels within real-world envi- ronments. To claborate, for a given RGB image, we initially employ Segment Anything [56] produce instance-level segmentation masks. These masks are subsequently integrated with corresponding RGB-D data to create instance-level point clouds. We employ the depth inpainting method [57] to reduce noise during the data annotation stage. Following this, the RANSAC algorithm (58 is utilized to extract the dominant planar surface from these point clouds.
C Additional Experimental Results
C.1 Comparison with Baselines under Multiple Difficulty Mode
In this section, we present additional experimental result to demonstrate the effectiveness of our OrderMind model architecture. Tab. 6 Tab. 7nd Tab. |rare manipulation ordering results under casy, moderate and hard difficulty mode separately. Succ. Num., Num., RC, OD and SR refer number of successfully picked-up objects, number of failed pick-up attempts, residual count, object disturbance and success rate respectively. Privilege indicates models are accessible objects' ground truth information.
These tables demonstrate and hard difficulty levels. It is also superior in performance and faster than the VLM method with privilege information. Notably, OrderMind's performance advantage becomes more pronounced as the difficulty increases. Importantly, fail num' indicates the number of failed attempts. Thus, inaccurate detection results lead the robotic end-effector repeatedly try to pick false-positive objects, resulting attempts. So that experiments relying on UniDet3D (54) usually have a higher 'fail num' metric. A lower residual count, lower object disturbance, and higher success rate demonstrate OrderMind's 's strong understanding of scene stability and its ability to perform object manipulation quickly and accurately.
our OrderMind method achieves the best performance across easy, moderate,
higher num failure
C.2 Perception Result
The manipulation ordering task demands robust understanding of spatial layout. To evaluate this spatial comprehension, utilize detection tasks. Tab, presents the detection
Average Precision (mAP) under easy, moderate, and hard difficulty settings. While (59) suggests that point cloud-based methods generally achieve better performance robotic tasks, our OrderMind method's superior mAP compared to such methods demonstrates that its end-to-end architecture mutually enhances both spatial layout perception and ordering capabilities.
C.3 Generalization
To evaluate generalization beyond the YCB dataset, additional experiments were conducted in novel and more challenging environment. The setup replaces the flat tabletop with deep-sided metal tray, introducing
16

---

Table Manipulation performance under easy setting in the simulation environment. Succ. Num., Fail Num., RC, OD and SR refer to number of successfully picked-up up objects, number of failed pick-up attempts, residual count, object disturbance and success rate, respectively. Privilege indicates models are accessible to objects' ground truth information.
Suce. Num.t
Fail Num.
Framework
Privilege
RC1
OD4
SR(%))
GPT-4o 19
19.2
2.1
4.8
2.3
90.3
Claude-3.7-Soune
49
23.3
2.1
0.7
2.0
92.1
Gemini-2.5
550
/
23.3
2.1
0.7
1.6
92.4
Gemma-3
51
22.8
3.4
1.2
2.2
88.3
Intern [2
22.6
3.3
1.4
Qwen2.5-VI 42
23.1
2.1
0.9
1.8
92.5
UniDet3D [54] CContidene
20.9
26.4
3.1
2.5
49.4
UniDee3D+Distance
X
21.7
36.1
2.3
2.4
40.3
UniDe:3DD+GPT-40
X
16.6
22.8
7.2
2.0
42.4
UniDes)D-Clamde:37--0mm
X
19.3
29.6
4.7
2.8
39.9
UniDer3DD+Geminn-22
20.8
37.5
3.2
3.0
45.5
UniDer3D+Gemmaa3
20.9
43.8
2.8
UniDer3D+IntemML3
21.2
36.0
2,8
2.9
38.3
UniDec3D+Qwen2.5-VV
X
20.5
39.0
3.5
3.4
38.8
OrderMind-Mini (Ours)
23.0
1.5
1.0
1.6
94.2
OrderMind (Ours)
X
23.6
0.9
0.4
0.7
96.5
Table Manipulation performance under moderate setting the simulation environment. Suce. Num., Fail Num., RC, OD and SR refer to number of successfully picked-up objects, number of failed pick-up attempts, residual count, object disturbance and success rate, respectively. Privilege indicates models are accessible to objects' ground truth information.
Suce. Num.t
Fail Num
Framework
Privilege
RCJ
OD]
SR(
GPT-4o
27.7
8.1
8.3
5.0
77.9
Claude-3.7 (ile-3.--Sonee
33.8
6.6
2.2
4.5
84,0
Gemini-2.5
33.8
6.6
2.2
4.8
84.6
Gemma-3
33.6
7.2
2.4
4.9
83.2
Intern VI.3
33.2
7.8
2.8
Qwen2.5-VI.
33.3
7.6
2.7
4.5
82.1
UniDer3D+Contidence
31.3
29.4
4.5
4.8
53.0
UniDer3D+Distance
X
31.1
62.5
4.9
5.5
34.8
UniDet3D+GPTT-4
X
24.0
37.6
12.0
3.9
39.6
UmiDesDD-Claule337--5o
X
29.5
69.9
6.5
6.1
36.0
UniDer3D++Gemin-22.5
30.4
37.6
5.6
5.5
46.4
UniDer3D+Geemma-3
30.5
43.3
5.5
S.3
UniDee3D+HntemVL3
28.9
40.1
7.1
5.8
42,9
UniDec3D+Qwen2.55-V
X
28.3
59.1
7.7
7.1
34.5
OrderMind-Mini (Ours)
X
32.7
3.9
3.3
3.6
89.6
OrderMind (Ours) X
35.0
1.8
1.0
1.4
95.3
distinct visual textures, backgrounds, and physical constraints for manipulation. A completely new set of five objects with diverse geometries and categories was used, including handled laundry bottle, thin correction- fluid pen, cube-shaped tea box, cylindrical conditioner bottle, and rectangular book. Each object was randomly scaled random factor between 0.8 and 1.2 to further increase variability.
Although UniDet3D has fewer parameter counts, is limited 0.8 FPS due its point cloud-based architecture. We evaluate two-stage frameworks where UniDet3D results are passed to VLMs for ordering. which achieves only 0.1 FPS and cannot meet the real-time demand. Meanwhile, although a standalone 2D detector such YOLO can operate at 37.0 FPS, our robotic manipulation tasks require 3D oriented bounding boxes for effective performance. To establish complete pipeline, added post- processing step that converts YOLO's
17

---

Table Manipulation performance under hard setting in the simulation environment. Suec. Num., Fail Num., RC, OD and SR refer to number of successfully picked-up up objects, number of failed pick-up attempts, residual count, object disturbance and success rate, respectively. Privilege indicates models are accessible to objects' ground truth information.
Suce. Num.t
Fail Num.
Framework
Privilege
RC1
OD4
SR(S))
GPT-4o
43.2
17.7
16,8
12.0
71.4
Mlaude-3.--Somtt
54.7
16.0
5.3
10.4
77.9
Gemini-2.5
/
54.2
17.4
5.8
12.4
78.5
Gemma-3
52.5
22.8
7.5
11.2
70.6
Intern VI.3
47.9
19.8
12.1
13.9
70.4
Qwen2.5-VI.
52.3
23.2
7.7
11.4
70.4
UniDer3D+Contidence
50.3
64.8
9.7
8.4
46.1
UniDe(3D+Distance
X
45.5
141.9
14.5
12.8
24.9
UniDet3DDGGPT-40
X
36.4
73.9
23.6
9.2
33.4
UmiDesSDD<Cladde:377-5nnn
X
41.4
96.2
18.6
14.2
30.7
UniDer3DD+Gemini-22
43.5
120.7
16.5
13.1
26.6
UniDer3D+Gemmaa-
44.7
82.7
15.3
13.5
UniDe(3D+IntemWL3
43.2
62.8
16.8
11.4
43.1
UniDet3D+Qwen2.5-VV
X
40.1
103.6
19.9
13.2
28.3
OrderMind-Mini (Ours)
54.6
5.9
5.4
7.2
90.4
OrderMind (Ours)
X
56.7
2.9
3.3
4.3
95.4
Table 9: Detection Result under easy, moderate and hard difficulty modes in simulation environment.
Model
Modality
mAPer
mAAmod
mAAShard
UniDet3D
Point Cloud
46.3
36.7
33.6
OrderMind (Ours)
RGB-D
92.2
88.6
78.3
detections and masks into oriented bounding boxes. This involved extracting object point clouds from depth images according to the 2D predictions, followed by fitting 3D oriented bounding boxes using an optimized RANSAC algorithm. As result, the full YOLO-based pipeline runs at 11.9 FPS, In comparison, our proposed OrderMind-min model achieves significantly higher speed of 21.3 FPS, making it much more efficient solution for the actual manipulation task.
Tab. T0 compares method against the same baselines. Across all difficulty levels, the proposed approach consistently achieves higher success rates (SR) lower residual counts (RC), mirroring trend observed the main experiments. This consistency indicates that the VL.Mgennrrted sequential plans remain coherent and effective even in novel environments with unseen objects. A slight increase in Object Disturbance (OD) under the hard setting reflects minor contacts during grasping, yet tasks still completed successfully without destabilizing the scene, this unlike the baseline methods, which often fail such perturbations.
These results demonstrate
proposed VLM-based label generation process generalizes beyond original setup. producing reliable and interpretable manipulation sequenees across diverse scenes and object types.
D Additional Ablation Study
In this section, we present an ablation study to determine the optimal weight for the loss Corder, with results summarized in Table [T] We experimented with weights of 3, 5, and 10. The data indicates that weight of achieves the best performance, yielding RC of 1.0, OD of 1.4, and SR of 95.3%. Other weights resulted comparatively inferior metrics. Thus, a weight of 5 selected the optimal value for Corder.
18

---

Table 10: Generalization performance of different frameworks across varied environments and object configurations. RC, OD and SR refer to residual count, object disturbance and success rate respectively. "Privileged" indicates methods with access to ground-truth object attributes.
Easy
Moderate
Hard
Framework
Privilege
FPS
RCI
OD
SRT
RC1
OD]
SR
RCJ
ODL
SRT
GT+SPH
/
0.9
0.7
89.5
1.7
1.5
86.8
1.8
4.4
90.9
GT + Gemini-2.5
/
0.2
2.5
99.2
2.5
4.6
91.2
7.2
7.2
88.0
0.2
YOLO11-seg+S00
X
8.9
1.9
59.0
10.3
4.1
66.9
17.5
8.5
69.0
11.9
| Geeminnnn
X
9.2
1.9
60.8
13.8
4,4
65.4
21.6
8.7
67.5
11.9
YOLO! 1-dct+SPH
X
7.7
7.7
65.2
11.1
3.9
66.7
18.1
8.1
69.0
11.9
X
7.8
2.0
64.3
12.6
4.6
67.4
17.4
7.6
67.6
11.9
Ours
X
4.2
1.8
85,3
5.3
4.0
83.2
7.4
11.2
83.7
21.3
Table 11: Ablation study on weight selection for loss Lorder. RC, OD and SR refer to residual count, object disturbance and success rate, respectively.
Weight
RC |
OD
SR(%)T
3.7
3,9
89.6
1.0
1.4
95.3
7
2.6
3.5
91.3
10
2.8
3.6
92.2
E
Order Label Generation Pipeline with Vision-Language Model
To mitigate potential inconsistencies "LM-generated labels, we aggregate multiple rankings per seene using a Plackett- Luce model, which infers stable global ordering from noisy pairwise comparisons. This ensemble step improves label reliability during training. We demonstrate the prompt given VLM follows:
Listing I: Prompt for VLM
Image Prompt: RGB image at resolution 1048*1024 with object segment masks and ids, as demonstrated in 77
robotic to manipulate objects from a table. You need to manipulate objects quickly and accurately while making sure the success rate is high The arm has a suction cup and its starting position is [1408, 1024], shich is the bottom-right corner of the image. After manipulating each object, you need to return manipulating the next one. You must follow the rules caterrii_foo_mniiulatttti
Text Prompt: Based this image, imagine you
used
the starting position before
Criteria has the highest priority, and the priorities of the others decrease accordingly. You can be flexible needed, but the ultimate goal is to quickly and accurately collect
the objects. You must collect ! The object information template is in 'objects terpcttttte id is the ID of each object. The output should be in JSON format as shown in 'output_format'. where objects
the objects where
In 'ranking', write the final order,
the manipulating order, from first to last. All objects must be included and sorted correctly. Think step by step, show the correct manipulating order. Strictly follow my output format and instruction. Please analyze the input data thoroughly and carefully. Ensure the ranking is derived from serious and deep evaluation of available information.
sorted according
for www.maanillaiiion... "criteria_foo.aaapllaai"""" {
"criteria_1": "Manipulate objects whose <is_isolated is true and <aize> is large.",
"criteria_2"; "Manipulate objects whose <is_local_highen true and srolative.poiiiiontoooro torbott is small.", "criteria_3"; "Manipulate objects whose trotation_degres is small."
19

---

},
"objects_ templatee: "id": "<id>" object id "is_isolated"; "<is_lsolated>> # true / false true false
"is_local_highes "cis_ocal.highstt"", "size": "<size>", small large "box_area": "<boz_area>", small / large
"rotation_degree" pootaatio_ddegreeeee # small / large
pntecive_positio__tee # float number },
ccelaativo..osttionttooooo
{
"ranking": "<id> -> <id> -> <id> -> -> <id>"
F Details on Object Attributes
In this paper, object attributes are divided into intrinsic attributes and extrinsic attributes. Intrinsic attributes describe object's extent and category. Extrinsic attributes describe the object's pose in the world coordinate system, which includes we use different heads in the detector to predict the size information, the center point in the camera coordinate system and the 3D orientation in the object coordinate system 6. Subsequently, we use the object-to-camee to-cmrra homogeneous transformation matrix Toc E S3)3), and the camera-te to world homogencous transformation matrix Twe SE(3) to convert center point and 3D orientation into the world coordinate system. The specific process is as follows:
center point p [a. y, =|" and
orientation
[rull, pitch, yauj" First,
Two
Dc
0=,
Two Teo
0u 1
CHEEZ-I
OREEIT
D Domino
ADL
375
14
IGS
128
Domino
124
HHE MREEIT
19A
CHHEEZT
324
12
Domino
IN
TEAR
204
IF2
138
SPA
JEO
Figure 7: Example of an image prompt provided to the VLM, containing segment masks and object IDs marked on each object.
20


Reviews:
[{"review_id": "nAM7ZhIzNu", "reviewer": "Reviewer_Erw7", "summary": "The authors propose a neural network-based approach for the manipulation task of ordered object picking. The use an image backbone on RGB-D images to predict object bounding boxes together with attributes. On top of the bounding boxes and attributes, a neural network is trained to predict picking priority score, which is then used to create an execution plan. The approach is validated both in simulation and real-world experiments, and shows impressive performance in picking success rate, significantly outperforming VLM baselines.", "questions": "- What is the architecture of the image backbone? I could not find details neither in the paper nor in supplementary.\n- does OrderMind create a single plan and execute it in simulation and real-world experiments, or replan every few frames? in case of the latter, are plans stable for rearrangement? is there index shuffle during replan?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- a solution to an important real-world robotic problem is proposed\n- the solution is relatively simple and does not depend on heavy VLM compute at inference time\n- experimental validation consists of tests both in simulation in real world and shows high success rate. I find fascinating how state-of-the-art VLM are able to achieve reasonable performance on an unseen robotics task, yet cannot outperform a task dedicated solution, reinforcing the \"jack of all trades, master of none\" claim for LLM.\n- the system runs at 6-21 FPS which is additionally impressive.", "was_revised": true, "final_justification": "The authors responded and provided details on image backbone architecture and replanning. These are important points which should be added to the main text. After reading the rebuttal and other reviewers comments I stand by \"accept\". Solid work with potentially high impact.", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**Q1: Details on the image backbone architecture.**\n\nWe thank the reviewer for pointing out this omission. We will clarify the architecture of our image backbone in the revised version of the paper. \n\nTo directly answer the question, we use the DLA-34 network [1] as our image backbone. We selected DLA-34 for its strong balance of computational efficiency and feature extraction capability, which is crucial for real-time robotics applications. The primary role of the backbone is to transform the input image into a rich geometric representation of the scene. Specifically, it is trained to produce a heatmap of object centers [2], regress object 3D dimensions, and predict their 3D orientations using a robust classification-then-regression method [3]. Our image backbone provides a rich, geometric understanding of the environment that is essential for the subsequent ordering task.\n\nWe note that our framework is not a simple pipeline where detection is followed by a separate ordering step. Instead, the ordering module is tightly integrated with the perception backbone. It operates directly on the deep feature maps from DLA-34. By using an attention mechanism on these rich features, our ordering module can reason about the complex spatial and contextual relationships between objects. This integrated design is key to our method's ability to determine a feasible manipulation order and distinguishes it from approaches that rely only on post-processed detection results.\n\nWhile DLA-34 provides a strong and efficient baseline, we also note that the architectural principles of our method are general, and we anticipate that its performance could be further enhanced by leveraging more advanced backbones.\n\n---\n\n**Q2: Replanning strategy and plan stability.**\n\nWe are grateful to the reviewer for this insightful question about our system's planning strategy and robustness. \nTo directly answer the first part of the question, in the experiments reported in our main paper, OrderMind replans after each manipulation action is completed. We chose this strategy to maximize robustness against unexpected perturbations common in real-world interactions. For instance, manipulating one object can slightly shift others and alter the scene's state. Since our method is real-time, replanning after each action allows the system to always base its next move on the most current environmental state without incurring significant computational overhead.\n\nTo explore the stability of our generated plans, we conducted additional simulation studies to evaluate the system's performance with less frequent replanning. Specifically, we evaluated performance at replanning intervals of 1, 5, and 10 manipulations. We compared our unified model with the two-stage detection-then-sort method in terms of ID shuffle and task success rate. We adapted our Spatial Priors, which were described in Sec. 3.3 in our main paper, into a heuristic policy and name it as Spatial Priors Heuristic (SPH). \n\nTo quantify the degree of index shuffling between consecutive manipulation plans, we employed the Levenshtein Distance (LD). LD is a classic metric that measures how dissimilar two sequences are by counting the minimum required edits (insertion, deletion, or substitution). For instance, since swapping two elements requires two distinct edits so that $LD([A, B, C], [A, C, B]) = 2$. This characteristic makes the metric highly suitable for assessing structural changes within a sequence. We present the results in the table below, comparing the success rate and Levenshtein Distance on the moderate difficulty scenes.\n\n**Table 1.** Levenshtein Distance (LD) and success rate (SR) under different replanning intervals. \"Privileged\" indicates methods with access to ground-truth object intrinsic and extrinsic attributes.\n\n| Replanning Interval |    Framework            | Privilege | LD ↓ | SR ↑ |\n| :------------------ | :------------------- | :-------: | :--: | :--: |\n| **1**         | GT + SPH             |     √     | 1.5  | 96.3 |\n|                     | YOLO11-seg + SPH     |     ×     | 3.6  | 77.5 |\n|                     | YOLO11-det + SPH     |     ×     | 1.8  | 79.8 |\n|                     | **OrderMind (Ours)** |     ×     | 1.7  | 89.6 |\n| **5**         | GT + SPH             |     √     | 5.2  | 85.5 |\n|                     | YOLO11-seg + SPH     |     ×     | 6.6  | 75.3 |\n|                     | YOLO11-det + SPH     |     ×     | 6.5  | 72.2 |\n|                     | **OrderMind (Ours)** |     ×     | 3.5  | 82.3 |\n| **10**        | GT + SPH             |     √     | 7.5  | 85.5 |\n|                     | YOLO11-seg + SPH     |     ×     | 9.3  | 70.6 |\n|                     | YOLO11-det + SPH     |     ×     | 9.3  | 69.0 |\n|                     | **OrderMind (Ours)** |     ×     | 4.7  | 80.5 |\n\nWhile the heuristic-based baselines without privileged information may result in higher plan instability and a drop in success rate, our method maintains more stable plans and a higher success rate. After several manipulation steps without replanning, heuristic methods exhibit larger ID reshuffling, as reflected by higher Levenshtein distances between consecutive plans. Across all tested replanning intervals, our approach consistently outperforms the two-stage detection-then-sort method, achieving both a higher success rate and lower Levenshtein distances.\n\nWe attribute this to our unified design of jointly learning spatial representation and manipulation ordering. Heuristic methods are often greedy, i.e., they focus on the best immediate action based on local object stacking relationships, but they struggle to foresee the long-term consequences of an action. In contrast, our method distills the knowledge from large vision-language models and integrates rich visual and geometric context to generate a more globally coherent and far-sighted plan. This makes our plans inherently more robust and stable over longer execution horizons, leading to less index shuffling and a more gradual degradation in performance in few replan settings.\n\n**Reference**\n\n[1] Yu F, Wang D, Shelhamer E, et al. Deep layer aggregation. CVPR2018\n\n[2] Zhou X, Koltun V, Krähenbühl P. Tracking objects as points. ECCV2020\n\n[3] Guha E K, Natarajan S, Möllenhoff T, et al. Conformal Prediction via Regression-as-Classification. ICLR2024"}, {"review_id": "XUqx2yAKc2", "reviewer": "Reviewer_7BAs", "summary": "The agent should be able to look at a scene, understand 3D relationships between objects, and then take a series of manipulation actions to pick up unobstructed objects efficiently.", "questions": "1. I apologize to the authors if I've missed something critical but, its easy and fast to predict bounding boxes and masks, so it should be easy to simply rank unoccluded objects and then sort by proximity. Particularly because we are only working with overhead views.  Why is robot pose necessary here or even a bespoke architecture?\n\n2. The fact that in hard scenarios the results only drop to 76% seems to validate my concern that the task is not that difficult to code up? Is this due to the myriad number of ways to succeed? Is there really a good reason to choose one ordering over another? \n\n3. The real world examples appear incredibly simple.", "limitations": "See questions -- more generally I'm unclear on what the constraints are on the system, what the failures look like, what would be necessary to address them, etc.\n\nOne angle I could see is that there's an efficiency concern which limits the use of large VLMs but then I would still expect to see a few simple mask/bbox/yolo/... style baselines that can run at very high frequencies.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The application domain is straight forward to understand and the task is of real utility. There are a number of points where I'm confused by the experimental setup and goals. \n\nWhat I understand, is that a 2D image is mapped to 3D with biases and preferences like picking up two objects near each other is prioritized and the evaluation is done over varying amounts of clutter until models begin to fail. \nWhat I don't understand is why a heuristic policy (or simple pipeline) would ever fail.  Additional motivation, analysis, and insights would be greatly appreciated.", "was_revised": true, "final_justification": "The results presented in the rebuttal are crucial to the paper's claims and yet, still incomplete.  I recognize that more comparisons, particularly ones with visuals, cannot be included in the rebuttal but they should have been included in the original manuscript. As I noted, I can come up with a bunch of other simpler baselines, or reasons these particular comparisons don't match their system, but the paper should identify these and justify. each reply begins to give a little of what's necessary, but I still think the authors devised an approach and moved forward with publication without thinking deeply about the space or implications and are only now starting to perform analysis of useful baselines.", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**W1: Analysis on heuristic policy's failure case.**\n\nWe sincerely thank the reviewer for raising this excellent question. It addresses the core motivation for our work, and we appreciate the opportunity to elaborate on why heuristic policies falter in complex, realistic scenarios.\n\nThe main weakness of simple heuristics, such as always choosing an unoccluded object, is that they ignore complex physical interactions like stability and support. These approaches treat objects as independent and overlook crucial consequences of actions. For example, in a scenario with stacked objects, a heuristic might pick a tall, unoccluded object, but if that object supports others, removing it could destabilize the whole pile and cause a collapse. This shows how a locally optimal choice from a simple rule can have disastrous global effects.\n\nOur method aims to address this challenge by distilling the commonsense physical reasoning of a large Vision-Language Model (VLM) into a compact and efficient policy [1]. We employ the VLM as an expert teacher to generate spatial-aware optimal manipulation action labels for a given scene. A smaller, faster model is then trained on these labels to approximate the VLM's underlying logic about physical stability and object interdependencies, rather than memorizing simple geometric heuristics. This enables our manipulation policy to anticipate and prevent potential issues like the cascading collapses discussed earlier, making it more robust. We believe this distillation of reasoning is the key to overcoming the fundamental limitations of heuristic approaches, and we will clarify this motivation in the revision.\n\n---\n\n**Q1: On the necessity of our model over a simple pipeline.**\n\nWe thank the reviewer for this sharp and practical question. A simple pipeline approach is indeed an intuitive baseline, and we appreciate this opportunity to clarify why a more integrated model like ours is necessary for achieving robust and efficient manipulation.\n\nFirst, addressing the detection component, while predicting 2D boxes is fast, robotic grasping fundamentally requires precise 3D orientation. As demonstrated in **Tab. 1 in response to W1 for reviewer RBro**, a pipeline that starts with 2D detections would need an additional step to estimate the 3D pose from 2D masks and depth data. Our model streamlines this by directly and efficiently predicting the necessary 3D information in one go. Second, a heuristic policy that simply sorts unoccluded objects by proximity is insufficient for two main reasons. As we demonstrated in our response to the **Q2 for reviewer Erw7**, it ignores complex physical stability issues, such a pipeline is inherently short-sighted. As demonstrated in the **Tab.1 for reviewer Erw7**, if model inference is run only once for every five manipulation attempts, our model’s performance remains high because it has learned to plan ahead. The heuristic pipeline’s success rate, however, collapses, as it lacks any foresight beyond the immediate next step.\n\nThe robot's pose is included to help the model consider motion-related risks beyond simple proximity. Using the robot's fixed XYZ base, we calculate its position relative to each object, allowing the model to balance motion cost with physical outcomes. For instance, the model can prefer a slightly farther, stable object over a closer, unstable one if it reduces the risk of destabilizing the pile.\n\nIn summary, we do not simply model the manipulation ordering problem as a decomposable task of \"detect-then-sort\". It demands jointly reasoning about object geometry, physical stability, and robot-object mutual constraints.\n\n---\n\n**Q2: Regarding task difficulty and the significance of ordering with multiple solutions.**\n\nRegarding the 76% success rate in hard scenarios: In our experiments, the Easy scenario features relatively simple stacking and minimal overlap between objects, resulting in a higher success rate. In the Moderate scenario, increased complexity in stacking and occlusion causes the success rate to drop to 78.5%. Although the number of objects further increases in the Hard scenario, its stacking relationships are similar to those in the Moderate scenario, with the main challenge still coming from complex stacking. Thanks to the model's real-time inference capability, we can effectively handle various disturbances in the scene. As a result, even with a much larger number of objects, the success rate only slightly drops from 78.5% to 76.6%, demonstrating the model's robustness and adaptability to task complexity. This high success rate is not due to the simplicity of the task itself, but rather to the model’s strong ability to handle such complex scenarios.\n\nRegarding the selection of grasping order: While there may not always be one optimal order, the multiple successful sequences possible actually highlight our model's flexibility. Even when several objects share high priority, our model finds rational grasping orders. As shown in Table 1 in the main paper, our method outperforms heuristic and VLM-based policies by maximizing success rates, minimizing disruption, and efficiently reducing remaining objects. Thus, even without a \"best\" order, the model ensures reliable, efficient operation.\n\n---\n\n**Q3: The real world examples appear incredibly simple.**\n\nThe scenes in Figure 5 in the main paper are indeed from a controlled laboratory setting, which we created to clearly and intuitively demonstrate our model's core effectiveness. We have successfully deployed and validated our model in a real-world factory environment, where the conditions are substantially more cluttered and complex than even the 'hard' scenarios shown in Figure 4. To better showcase our method's performance in these practical conditions, we will replace Figure 5 in the final version of the paper with more complex, anonymized images from this real factory deployment. Furthermore, we will update our supplementary video to include anonymized footage from these challenging, real-world factory environments.\n\n---\n\n**Q4: The constraints and the failures of our system.**\n\nOur system faces key constraints: limited computational resources (15 TFLOPS on the edge device), a 30 ms time requirement, and strict safety rules— all inference must be done locally without cloud processing. Reviewing 30 minutes of robotic arm operation, we found four main failure sources. Perception errors were primary contributor, with 21% from inaccurate 3D rotation and another 15% from object center point misidentification. For challenges with deformable objects, an inability to find suitable suction surfaces caused 21% of the failures. Furthermore, incorrect manipulation ordering led to object disturbances, accounting for 39%. Other factors, like poor synergy between the robotic arm and the camera, accounted for 4%.\n\nWe intend to replace our current CNN-based backbone with a transformer-based one to enhance the model's understanding of object pose. We will also explore reinforcement learning and similar methods to optimize robotic arm motion paths, thereby reducing disturbances to other objects. Furthermore, we will refine the interaction logic between the camera and the robotic arm to improve their overall synergy.\n\n---\n\n**Q5: Comparison with fast heuristic baselines.**\n\nWe sincerely thank the reviewer for this constructive suggestion. We have implemented and evaluated two high-speed baselines built upon YOLO11 model, one using its detection outputs (YOLO11-det) and another using its segmentation masks (YOLO11-seg), as demonstrated in the **Tab. 1 in response to W1 for reviewer RBro**. \n\nFirst, a standalone 2D detector like YOLO can achieve 37.0 FPS on its own, However, our robotic manipulation task fundamentally requires 3D oriented bounding boxes to execute manipulation. Therefore, a complete baseline pipeline must include a subsequent step to convert these 2D outputs into 3D information. For this, we extracted object point clouds from the depth image using the 2D predictions and then fitted 3D oriented boxes with a highly optimized RANSAC algorithm. The final end-to-end speed of this YOLO-based pipeline is 11.9 FPS. This is slower than our proposed OrderMind-mini, which operates at 21.3 FPS. For the actual task at hand, our model provides a much faster solution. \n\nSecond, our method also demonstrates substantially better task performance. The YOLO-based baselines exhibit a lower manipulation success rate and leave a larger number of objects in the scene. This performance gap stems from two fundamental issues. The first is the error propagation in a sequential pipeline. The 2D-to-3D fitting process inevitably introduces inaccuracies in the resulting 3D bounding boxes, leading to grasp failures. The second and more critical issue is that such a baseline lacks a deep understanding of spatial relationships. It can only rely on simple heuristics for ordering, which is insufficient for resolving complex occlusions and dependencies.\n\nThis comparison highlights the advantage of our work. Instead of approaching the problem as a simple detection-then-sorting pipeline, our method leverages VLM-guided supervision to learn spatial awareness and ordering in an end-to-end fashion. Our model does not just detect objects. It comprehends the relationships between them to infer a valid and efficient manipulation order. This capability proves crucial for succeeding in long-horizon and orderly manipulation tasks where simpler heuristic-based approaches falter.\n\n**Reference**\n\n[1] Zhou W, Tao M, Zhao C, et al. Physvlm: Enabling visual language models to understand robotic physical reachability. CVPR2025"}, {"review_id": "oXGB0rXRhG", "reviewer": "Reviewer_84U8", "summary": "This paper introduces OrderMind, a spatial-aware framework for learning object manipulation order in cluttered scenes. The idea is to directly learn manipulation priority from RGB-D inputs using a combination of a k-NN spatial graph (for object-object and object-robot relationships) and a temporal ordering module that ranks objects by learned priority scores. Labels for supervision are generated using a vision-language model (VLM), but enhanced with spatial priors (like object isolation or stackability). The authors also built a large-scale benchmark (160k+ samples) and show both strong sim and real-world performance compared to heuristics, VLM pipelines, and two-stage baselines.", "questions": "Can you share examples of the prompts used for VLM ordering supervision? Was any spatial descriptor, e.g., relative pose or bounding box info, passed into the prompt?\nThe VLM-generated labels are critical to training. How consistent are these labels across different scenes and object types? Have you tested generalization to new objects or unseen environments beyond YCB?\nWhat kind of errors does the model tend to make in hard real-world scenes (e.g., with deformable or occluded bags)? Qualitative failure examples would be valuable.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The problem that author explores is real and under-explored by other works. OrderMind speedup on slow multi-stage pipelines by jointly encoding spatial reasoning and prioritization in one model. Use of k-NN spatial graphs and attention-based temporal priority modeling is a well-motivated and effective architectural choice. No manual annotation needed. Proposed a new benchmark. Both qualitative and quantitative results show consistent improvements across difficulty levels and across simulation/real-world settings.\n\n**Weaknesses**: Spatial priors help, but the generated order labels may still reflect unintended biases or fail in edge cases. Some analysis on this would help.\nThe authors miss details on VLM prompting: What exactly is the prompt format for order generation? Is there chain-of-thought prompting or just spatial descriptor input? Since the VLM is a core component for supervision, this should be shown.\nIn addition, the author should also discuss the label noise come from VLM. What happens when VLM labels are wrong or inconsistent? The model likely learns around it, but it'd be good to quantify how robust it is.", "was_revised": true, "final_justification": "The authors' rebuttal comprehensively addresses my concerns with concrete details: full VLM prompt examples, new experiments validating label consistency/generalization in novel environments/objects, noise robustness ablation, and failure mode quantification from real video. Minor unresolved: Deeper analysis of VLM biases in rare edge cases (e.g., human validation subset) and expanded real-world testing on deformables/occlusions could further improve, but these are low-weight given the core strengths.Overall, the rebuttal elevates quality/reproducibility, outweighing remaining issues.", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**W1 Q1: Prompt format and info for VLM ordering supervision.**\n\nOur prompt format has two components: a pre-processed image prompt and a meticulously designed text prompt. This format integrates both spatial descriptors and chain-of-thought (CoT) prompting.\n\nThe image prompt is an RGB image at a resolution of 1024×1408. We annotate each object in the image with its segment mask and ID. In cluttered scenes, VLMs struggle to interpret spatial and stacking relationships from the raw image. To solve this, we overlay the segment masks and object IDs directly onto the image. This allows the VLMs to associate visual objects with textual descriptors and better capture relevant spatial information.\n\nOur text prompt integrates a task description, object-specific information, and grasping rules, utilizing both spatial descriptors and CoT prompting. First, we provide a textual description of the task scene and the robot's grasping process to help the VLM comprehend the goal. Instead of directly inputting large amounts of numerical data, we convert object information into spatial descriptors, making the spatial context easier for the VLM to understand. \n\nBased on this object information, we establish grasping rules with priorities. Since scenes with many objects can satisfy multiple rules simultaneously, we introduce CoT prompting. This guides the VLM to flexibly combine the rules with their priorities and to think step-by-step. To accelerate the speed of label generation, we do not require the VLM to output its detailed inference process.\n\nDue to the space limits, we will show our prompt example in the discussion stage.\n\n---\n\n**Q2：Consistency of VLM-generated labels across scenes and objects.**\n\nWe sincerely thank the reviewer for this insightful question. The consistency of VLM-generated labels is indeed critical to the generalizability of our method and we agree it is an important aspect to validate.\n\nWe conducted a new set of experiments in a challenging environment. First, we designed a new simulation environment that significantly departs from the one used in our paper. We replaced the flat tabletop with *a deep-sided metal tray*. This change introduces not only new visual textures and backgrounds but also different physical constraints for manipulation. Second, we used a completely new set of five objects with diverse shapes and categories including *a laundry bottle with an integrated handle*, *a thin correction fluid pen*, *a cube-shaped tea box*, *a cylindrical hair conditioner bottle*, and *a rectangular book*. These objects are distinct from the relatively uniform box-like objects used in our original experiments. To further increase the complexity, we applied random scaling between 0.8 and 1.2 to the size of these new objects. The table below compares our method against the same baselines.\n\n**Table 1.** Comparison of different frameworks. RC, OD and SR refer to residual count, object disturbance and success rate respectively. \"Privileged\" indicates methods with access to ground-truth object intrinsic and extrinsic attributes.\n|Framework| Privilege|Easy RC ↓|Easy OD ↓|Easy SR ↑|Moderate RC ↓|Moderate OD ↓|Moderate SR ↑|Hard RC ↓|Hard OD ↓|Hard SR ↑|FPS|\n| :-------------------- | :-------: | :-------: | :-------: | :-------: | :-----------: | :-----------: | :-----------: | :-------: | :-------: | :-------: | :--: |\n|GT+SPH|√|0.9|0.7|89.5|1.7|1.5|86.8|1.8|4.4|90.9|/|\n|GT+Gemini-2.5|√|0.2|2.5|99.2|2.5|4.6|91.2|7.2|7.2|88.0|0.2|\n|YOLO11-seg+SPH|×|8.9|1.9|59.0|10.3|4.1|66.9|17.5|8.5|69.0|11.9|\n|YOLO11-seg+Gemini-2.5|×|9.2|1.9|60.8|13.8|4.4|65.4|21.6|8.7|67.5|11.9|\n|YOLO11-det+SPH|×|7.7|7.7|65.2|11.1|3.9|66.7|18.1|8.1|69.0|11.9|\n|YOLO11-det+Gemini-2.5|×|7.8|2.0|64.3|12.6|4.6|67.4|17.4|7.6|67.6|11.9|\n|**Ours**|×|4.2|1.8|85.3|5.3|4.0|83.2|7.4|11.2|83.7|21.3|\n\nThe results show that our method consistently achieves a higher success rate and a lower residual count  across all difficulty levels. This performance trend aligns with our findings in Table 1 of the paper. This consistency suggests that the VLM was able to generate high-quality and coherent sequential plans for this novel and more complex scenario. Notably, our method exhibits a slightly higher Object Disturbance in the hard mode. However, this is coupled with a much lower RC and a higher SR. This indicates that while our policy may cause minor contact with cluttered objects during a grasp, it successfully completes the task without knocking them out of the robot workspace. In contrast, the baseline methods often fail entirely after such disturbances. This behavior highlights the robustness of the policy learned from the VLM-generated labels.\n\nIn summary, these new experiments provide evidence that our VLM-based label generation process is not confined to a specific setup. It consistently produces effective and logical sequential labels for varied scenes and object types, confirming the generalizability and robustness of our approach.\n\n---\n\n**W2: Robustness to label noise from VLMs.**\n\nWe thank the reviewer for this excellent question regarding the potential for label noise from VLMs and our model's robustness.\n\nIn our experiments in the main paper, we designed our label generation process specifically to address the possibility of noisy or inconsistent VLM outputs to some extent. Specifically, our method first generates multiple rankings for one scene by many turns of VLM labeling. We then ensemble these rankings using a Plackett-Luce model, which allows us to infer a more stable and reliable global ranking from various potentially noisy inputs. This step is our primary mechanism for improving the quality of the training labels and mitigating the impact of occasional VLM errors.\n\nWhile this ensemble approach helps to reduce noise at the source, we believe that a direct quantification of the model's robustness is a valuable addition. We started with our refined VLM-generated labels and then intentionally introduced controlled levels of noise by randomly swapping a certain percentage of pairwise orders. We ran this experiment on the 15,000 samples from the new scene mentioned in our response to **Q2**. The results are presented in the table below.\n\n**Table 2**. Ablation on model robustness to label noise. \"Privileged\" indicates methods with access to ground-truth object  intrinsic and extrinsic attributes.\n| Noise  Ratio | Privilege | SR-Easy↑ | SR-Mod↑ | SR-Hard↑ |\n| ------------ | :---------: | :--------: | :-------: | :--------: |\n| 0  | × | 85.35 | 83.19 | 83.66|\n| 10%| ×| 85.10 | 82.71 | 79.13|\n| 20%| × | 80.08  | 79.00 | 76.30 |\n| 50%| × | 78.88  | 76.91 | 73.81 |\n| 70%| × | 75.99 | 70.36 | 67.31 |\n| GT + random| √  | 75.50| 75.58| 68.88|\n\nGT+random denotes the success rate when using a random manipulation policy given the ground-truth intrinsic and extrinsic attributes of the object. As the table indicates, the performance degrades slightly with moderate noise. Under the extreme 70% noise condition, the performance degrades more substantially. We hypothesize that this occurs because, at sufficiently high noise ratios, our model is unable to extract meaningful information and thus essentially degenerates into a random policy. In scenarios of moderate difficulty, the success rate of our model falls due to errors introduced by perception noise. Thanks to both our label ensemble strategy and the unified learning of perception and ordering, our method is robust to VLM label noise.\n\n---\n\n**Q3: Generalization to new objects and unseen environments.**\n\nYes, we have tested the generalization ability of our method to new objects and unseen environments beyond the YCB dataset.\n\nThe experiments presented in our main paper were specifically designed to evaluate performance in unseen environments. To clarify our experimental setup, the model was trained exclusively on data collected from a single dynamic conveyor belt in a factory. We then deployed and tested this model in two different and unseen settings. The first was a static scene within our laboratory for demonstrating, and the second was a different dynamic conveyor belt with various types of objects in the factory. These test environments featured notable variations from the training scene, including different lighting conditions, backgrounds, and degrees of motion blur. We believe this setup provides a direct evaluation of our model's ability to generalize to new environments and new objects.\n\nFurthermore, as described in our response to **Q2**, we set up a completely new scenario with a collection of novel objects that have diverse shapes. Our method continued to achieve a high success rate in manipulation tasks, and the object residual count and disturbance metrics remained low. These results are consistent with the results in our main paper.\n\n---\n\n**Q4：Failure case in real-world scenes.**\n\nWe manually analyzed a 30-minute video of our robotic arm's operation to categorize and quantify failures. Inaccurate object perception, challenges with deformable objects, and incorrect manipulation ordering emerged as the main issues.\n\nSpecifically, regarding perception errors, inaccurate 3D rotation prediction contributed to 21% of the failures, while incorrect object center point identification accounted for another 15%. For challenges with deformable objects, an inability to find suitable suction surfaces caused another 21%. Furthermore, incorrect manipulation ordering led to object disturbances, accounting for 39%. The remaining 4% were attributed to other factors, such as poor synergy between the robotic arm and the camera. For instance, the robotic arm occasionally blocked the camera's view, which adversely affected the perception process. In our revised version, we will incorporate qualitative failure cases and we will focus on resolving these issues in our future work."}, {"review_id": "Mfbx9SEnP0", "reviewer": "Reviewer_RBro", "summary": "In this paper, the authors address the problem of the order of manipulation, which is crucial in tasks such as picking up postal parcels. The robot must determine which item to handle first and which to handle next.", "questions": "1. The attributes are very important for representing the object. How are these attributes obtained? I think the authors should provide more details on this process.\n2. What are the extrinsic attributes? What do the arrows with different colors represent, and how are they obtained?\n3. In Fig. 2, do you directly combine all the embeddings together? How are the circle and rectangle combined? I believe there are several fusion methods; could the authors clarify which add is used here?\n4. How is fusion implemented in the extrinsic module?", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The topic is useful, and the performance is good. The method considers both spatial and temporal contexts, which should be valuable for this problem. However, when examining the framework, it seems that the authors treat the problem primarily as a computer vision task, similar to how appreciation order is used to detect objects in an image. I wonder if this problem has been considered from the perspective of object detection. If Yes. Whether this method could be applied to the context in this paper to achieve even better results.", "was_revised": true, "final_justification": "Thank you for your feedback. Although the method the authors have employed builds upon existing methods to tackle this issue, the technical novelty does not seem to be particularly high. Nonetheless, I believe the paper still carries some significance. However, the concerns raised by Reviewer 7B should be taken into consideration.", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**W1: On framing the task as an object detection pipeline.**\n\nWe sincerely thank the reviewer for this insightful question. We have implemented and evaluated two computer vision baselines built upon the YOLO11 model [1]. We use YOLO11-det as a detection baseline and YOLO11-seg as the segmentation baseline. To ensure a fair comparison, we adapted our Spatial Priors into a heuristic policy, which was described in Sec. 3.3 in our main paper, into a heuristic policy and name it as Spatial Priors Heuristic (SPH). As an effective heuristic policy, SPH can provide the detection model with a better sorting result.\n\nWhile a standalone 2D detector such as YOLO can operate at 37.0 FPS, our robotic manipulation tasks require 3D oriented bounding boxes for effective performance. To establish a complete pipeline, we added a post-processing step that converts YOLO’s 2D detections and 2D masks into 3D oriented bounding boxes. This involved extracting object point clouds from depth images according to the 2D predictions, followed by fitting 3D oriented bounding boxes using an optimized RANSAC algorithm. As a result, the full YOLO-based pipeline runs at 11.9 FPS. In comparison, our proposed OrderMind-mini model achieves a significantly higher speed of 21.3 FPS, making it a much more efficient solution for the actual manipulation task.\n\n**Table 1**. Success Rate (SR) of our model vs. simple heuristic baselines across difficulty modes. \"Privileged\" indicates methods with access to ground-truth object intrinsic and extrinsic attributes.\n| Method                                 | Privilege | SR-Easy ↑ | SR-Moderate ↑ | SR-Hard ↑ | FPS  |\n| -------------------------------------- | :-------: | :--------: | :------------: | :--------: | :--: |\n| GT + SPH        |     √     |    96.3    |      96.3      |    89.8    |  /   |\n| YOLO11-seg + SPH |     ×     |    75.1    |      77.5      |    76.4    | 11.9 |\n| YOLO11-det + SPH |     ×     |    75.5    |      79.8      |    74.9    | 11.9 |\n| **OrderMind (Ours)**           |     ×     |    94.2    |      89.6      |    90.4    | 21.3 |\n\nFrom the Table 1, our proposed model consistently outperforms the detection-based and segmentation-based baseline across all difficulty mode. The primary reason for this performance gap lies in the difference between a decoupled pipeline and our integrated end-to-end approach. The heuristic detect-then-sort method treats perception and ordering as two disconnected steps. The sorting heuristic operates only on the geometric outputs like bounding boxes, losing the rich visual and contextual information from the original image that is necessary for reasoning about complex spatial relationships.\n\nIn contrast, our model learns to perceive objects and infer their manipulation order simultaneously. It does not simply detect objects but learns to understand the intricate spatial arrangements and dependencies between them, guided by the VLM's knowledge. This integrated approach allows our model to handle challenging scenarios that require long-horizon planning and spatial understanding, which are difficult for a simple heuristic to capture. By unifying object understanding and order prediction into a single inference pass, our model achieves more accurate and robust manipulation ordering, effectively maintaining spatial consistency across diverse and cluttered scenes.\n\n---\n\n**Q1: Details on Attribute Acquisition.**\n\nIn this paper, object attributes are divided into intrinsic attributes and extrinsic attributes. Intrinsic attributes describe object's extent and classification. Extrinsic attributes describe the object's pose in the world coordinate system, which includes the center point $p_{w}=[x,y,z]^T$ and the 3D orientation $\\theta_{w}=[roll,pitch,yaw]^T$. First, we use different heads in the detector to predict the size information, the center point in the camera coordinate system $p_{c}$, and the 3D orientation in the object coordinate system $\\theta_{o}$. Subsequently, we use the object-to-camera homogeneous transformation matrix $T_{oc} \\in SE(3)$, and the camera-to-world homogeneous transformation matrix $T_{wc} \\in SE(3)$ to convert center point and 3D orientation into the world coordinate system. The specific process is as follows:\n$$\n\\begin{bmatrix} p_{\\text{w}} \\\\\\\\ 1 \\end{bmatrix} = T_{wc} \\begin{bmatrix} p_{\\text{c}} \\\\\\\\ 1 \\end{bmatrix} , \\begin{bmatrix} \\theta_{\\text{w}} \\\\\\\\ 1 \\end{bmatrix} = T_{wc}\\cdot T_{co} \\begin{bmatrix} \\theta_{\\text{o}} \\\\\\\\ 1 \\end{bmatrix}.\n$$\n\n---\n\n**Q2: Details on extrinsic attributes.**\n\nWe sincerely thank the reviewer for their meticulous observation. We deeply apologize for a labeling error in Figure 2. The labels for intrinsic attributes and extrinsic attributes at the right of the image backbone were inadvertently swapped and we will revise it in the final version.\n\nExtrinsic attributes refer to an object's 6D pose within the world coordinate system. This pose comprises the object's center point and its 3D orientation. These attributes vary as the object's position changes within the world coordinate system.\nArrows of different colors represent the object's orientation angles in the world frame. The RGB colors correspond to the x, y, and z axes, respectively. To determine the 3D orientation, our detector head first predicts the object's 3D orientation $\\theta_{o}=[roll,pitch,yaw]^T$, relative to the camera coordinate system. Next, we use the object-to-camera homogeneous transformation matrix $T_{oc} \\in SE(3)$, and the camera-to-world homogeneous transformation matrix $T_{wc} \\in SE(3)$ to convert 3D orientation into the world coordinate system as the following equation:\n$$\n\\begin{bmatrix} \\theta_{\\text{w}} \\\\\\\\ 1 \\end{bmatrix} = T_{wc}\\cdot T_{co} \\begin{bmatrix} \\theta_{\\text{o}} \\\\\\\\ 1 \\end{bmatrix}.\n$$\n\n---\n\n**Q3: Clarification on the embedding fusion method in Figure 2.**\n\nWe would like to clarify that we combine all the embeddings using element-wise addition. The embeddings represented by rectangles in Figure 2 correspond to the intrinsic and extrinsic embeddings of the objects. The circles refer to the object order queries. All these embeddings are projected into the same feature dimension before being combined. More specifically, the intrinsic embedding is calculated by encoding the 3D extent of the object using a sinusoidal positional encoding, while the extrinsic embedding is derived from the robot-object's relative position using the same encoding method. Both are mapped to the same feature dimension as the object order query. For the visual embedding, the object RoI queries from the image backbone are passed through an MLP to match the object order query's feature dimension.\n\nRegarding the fusion methods, we have tried a few simple alternatives, but we observed that the final results were similar. For reasons of simplicity and computational efficiency, we ultimately chose element-wise addition as our fusion strategy. In the future, we will explore more advanced or complicated fusion approaches to see if they further improve performance.\n\n---\n\n**Q4: Details on the implementation of fusion in the extrinsic module.**\n\nWe use the spatial graph from object-centric point clouds as introduced in Section 3.3 in our main paper. The fusion process consists of two main steps. First, for each object represented by its center point $p_i$ and feature $f_i$, we identify its spatial neighbors using a k-Nearest Neighbors strategy. This allows us to capture the local geometric relationships and context for every object. Next, for each object, we perform feature fusion by concatenating the feature $f_i$ with the difference $f_j - f_i$ from each neighboring object $p_j$ in the k-nearest set $N_k(i)$. This concatenated feature is then processed by a linear layer, and finally, we aggregate over all neighbors using a max pooling operator $M$, as described in as follows:\n$$\nFusion(f_i) = M(Linear(Concate(f_i, f_j-f_i)),\\forall p_j \\in N_k(i).\n$$\n\nThis fusion mechanism is motivated by the need to encode the physical interaction and alignment of objects, which can represent how objects affect each other's manipulability in the scene. By constructing these spatially-aware object embeddings, our approach improves the model's ability to perceive the local scene structure. These fused features are subsequently combined with visual representations to better support downstream ranking tasks.\n\n**Reference**\n\n[1] Glenn Jocher and Jing Qiu. Ultralytics YOLO11. 2024"}]

Author rebuttal:

Meta-review: This paper focuses on an interesting and specific manipulation task: object ordering in cluttered environments. This research holds practical significance for robotic manipulation.

Extensive experiments conducted in both simulated and real-world environments demonstrate that the proposed method significantly outperforms prior approaches in terms of both effectiveness and efficiency.

One reviewer votes a borderline reject, primarily raising concerns about the necessity of the proposed method when compared to a simple pipeline. While a simple pipeline is indeed feasible in straightforward scenarios, it suffers from a low performance ceiling and cannot be easily scaled to complex environments. Moreover, the proposed method does demonstrate superior performance to LM-based methods.

The AC recommends accepting this submission.
Decision: Accept (poster)