AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
Authors: Xinyi Wang
Xun Yang
Yanlong Xu
Yuchen Wu
Zhen Li
Na Zhao
Venue: NeurIPS 2025 poster ()
Keywords: 3D Vision
Embodied Artificial Intelligence
Vision–Language Models
Primary area: applications

Abstract: Effective human-agent collaboration in physical environments requires understanding not only what to act upon, but also where the actionable elements are and how to interact with them. Existing approaches often operate at the object level or disjointedly handle fine-grained affordance reasoning, lacking coherent, instruction-driven grounding and reasoning. In this work, we introduce a new task: Fine-grained 3D Embodied Reasoning, which requires an agent to predict, for each referenced affordance element in a 3D scene, a structured triplet comprising its spatial location, motion type, and motion axis, based on a task instruction. To solve this task, we propose AffordBot, a novel framework that integrates Multimodal Large Language Models (MLLMs) with a tailored chain-of-thought (CoT) reasoning paradigm. To bridge the gap between 3D input and 2D-compatible MLLMs, we render surround-view images of the scene and project 3D element candidates into these views, forming a rich visual representation aligned with the scene geometry. Our CoT pipeline begins with an active perception stage, prompting the MLLM to select the most informative viewpoint based on the instruction, before proceeding with step-by-step reasoning to localize affordance elements and infer plausible interaction motions. Evaluated on the SceneFun3D dataset, AffordBot achieves state-of-the-art performance, demonstrating strong generalization and physically grounded reasoning with only 3D point cloud input and MLLMs. Our code is available at [https://github.com/hannahwxy/AffordBot](https://github.com/hannahwxy/AffordBot).

AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
Xinyi Wang Xun Yang!,
Yanlong Xu Yuchen Wu², Zhen Li¹, , Na Zhao 2
University of Science and Technology of China Chinese University of Hong Kong, Shenzhen
Singapore University of Technology and Design
Unplug Christmas tree lights
Open left part of door.
Prodicted Element: [omaski=]
Motion Typer trunsiation
Predicted Element: [<mastb4> Motion Type: rocation Motion vertical
horienntal
Emplag
Kary Bress
Open the bottom drower of the wooden cobinst with the finwer vase and the picture on top.
Tern on the using the right remote on top of the couch.
Prodicted
(omasiil 3x]
Preulited Blement (=moukii)) Motton Type: tr translation Modion Dir: vertical iswards
Modion Type: crondotion
Modion DDc. hortiontal outwards
Figure I: We propose fine-grained 3D embodied reasoning: given a 3D scene and a language task instruction, the agent must identify relevant affordance elements and predict a structured triplet for each: its 3D mask, motion type, and motion axis direction.
Abstract
Effective human-agent collaboration in physical environments requires understand- ing not only what to act upon, but also where the actionable elements and how interact with them. Existing approaches often operate at the object level or disjoint- edly handle fine-grained affordance reasoning, lacking coherent, instruction-driven grounding and reasoning. In this work, we introduce new task: Fine-grained 3D Embodied Reasoning, which requires an agent to predict, for each referenced affordance element in 3D scene, structured triplet comprising its spatial lo- cation, motion type, and motion axis, based on a task instruction. To solve this task, we propose AffordBot, a novel framework that integrates Multimodal Large Language Models (MLLMMs) with a tailored chain-of- chain-of-thought (CoT) reasoning paradigm. To bridge the gap between 3D input and 20-compati MI.LMs, we render surround- surooudd-view images of the scene and project 3D element candidates into these views, forming rich visual representation aligned with the scene geometry. Our CoT pipeline begins with an active perception stage, prompting the MLLM to select the most informative viewpoint based on the instruction, before proceeding with step-by-step reasoning to localize affordance elements and infer plausible interaction motions. Evaluated on the SceneFun3D dataset, AffordBot achieves sfate-of-the- performance, demonstrating strong generalization and physically grounded reasoning with only 3D point cloud input and MLLMs. Our code is available at htttp:://// //gi httpt/ggtttt..t/////ge com/ http//annnaaay/aaa wwwpxy/Afffforr AffordBot.
"This work was carried out during Xinyi's visit to the IMPL Lab at SUTD. Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
For intelligent agents to collaborate effeetively with humans and operate autonomously in complex 3D physical worlds, they must perceive and interact with their surroundings at fine- -une-graiddd actionable This requirement aligns with the concept of "affordance", originally introduced in ecological psychology 44. which describes how elements in the enviromment offer possibilities for action. For example, to execute an instruction like "unplug the Christmas free lights", an agent must recognize and attend the fine details of the plug and reason about the interaction they afford, rather than merely identifying the larger object, such as the lights. Meeting this challenge requires agents to understand not only what elements in a scene afford interaction, but also where they are located and how to manipulate them. Such fine- grained embodied understanding is essential for grounded task execution in real- world environments [ 8 9
level I 2 3
While recent developments in multimodal large language models (MLLMs) 14 15 16
18
19 20 [1] and 3D scene understanding 22
23
24
26 27] have advanced object-centric 3D) 33 stop at high-level object recognition and
25
perception, existing approaches (28) 29
30
31
32
spatial grounding. However, they often overlook finerggraine structures required to infer how parts of objects afford specific interactions. SceneFun3D [44] makes a notable step forward by introducing benchmarks for fine-grained affordance grounding and motion estimation. However, it treats these subtasks in isolation and assumes instruction-agnosse motion prediction, requiring agents to infer motion parameters for all functional parts regardless of task context, limiting its applicability in coruuction-coditiinnn scenarios.
To address these limitations, propose a unified and cetptucttiooconnittiinnn task: Fine-grained 3D Embodied Reasoning, which jointly performs 3D affordance grounding and motion estimation based natural language instruction. Specifically, the task is formulated as a structured triplet prediction problem: for each referenced affordance element, the agent predicts a triplet comprising affordance mask, motion type, and motion axis direction. This formulation explicitly couples spatial grounding and interaction reasoning under natural language guidance, forming a coherent inference pipeline tailored for instruction-condiiioned embodied tasks.
As a solution, we introduce AffordBot, a novel framework that integrates 3D geometric information with the reasoning capabilities of MLLMs. Unlike prior work [34 35| 36
37 38) that relies on vidco-based inputs, which incur high computational overhead by processing redundant visual frames and often suffer from viewpoint limitations, AffordBot operates directly on 3D point clouds. However, MLLMs are inherently designed for 2D input and general-purye purpose reasoning, presenting a significant challenge when applying them to 3D spatial tasks that require physical grounding.
To bridge the modality gap between 3D input and 2D-native MLLMs, AffordBot begins by con- structing rich multimodal representation of the 3D scene. Specifically, we render surround-viww images from the 3D point cloud and project structured 3D affordance candidates onto these images, establishing explicit 3D-to-2D correspondences. This enables dense and spatially aligned visual context to be provided to the MLLM, without relying on redundant video streams.
On top of this foundation, we develop a task-specific chain-of tf-tuught (CoT) reasoning paradigm that systematically guides the MLLM through physically grounded, step-by-step logical inference. The process begins with an active perception phase, which we specifically designed to empower the MLLM effectively interpret the task instruction and autonomously select the most informative viewpoint. This initial "observe" step improves reasoning focus by reducing input redundancy and emphasizing task-relevant visual cues. Subsequently, the selected viewpoint anchors the model's reasoning process. Then the MI..M is guided through two distinct reasoning stages: afforlance grounding, where it localizes the target part in the scene, and interaction inference. where it predicts the motion type and axis direction based on the scene context and instruction. By conditioning every step on spatial input and task intent, our CoT paradigm enables physically plausible and semantically aligned reasoning, enhancing the agent's 's embodied intelligence.
We make three key contributions: (1) We introduce a new task formulation for fine-grainer task- driven embodied reasoning, 3D affordance grounding and motion estimation as structured triplet prediction from natural language. (2) We present AffordBot, a novel framework that integrates 3D perception and MI.. reasoning via holistic multimodal representation construction and tailored of-thught process. (3) We achieve state-of-the-ar results on SceneFun3ID, validating the effectiveness of our approach in physically grounded, instruction-conditione 3D reasoning.
2

---

Input
Multimodal Representation (Sec. 3.2)
CoT Reasoning (Sec. 3.3)
View Synthesis
3D-2D Association
Instruction
Multimodal Language Models
Surround View Imagus
Visual joput
Boplogehe mree lights
Mask Modio Type Divection
Descriptor Generation
Task Instruction
Geo-Sem Descriptor
3D Elements
Output Triplet
Figure 2: AffordBot Overview. Our method first constructs a holistic multimodal representation designed to bridge 3D scenes with 2D-native MLLMs. This process involves view synthesis, extrac- tion of geometric-semantic descriptors, and their association. Then, our designed Chain-of-Thoughh (CoT)) paradigm guides the MLLM ultimately predict a structured triplet for the task.
2 Related Work
Affordance Understanding. Understanding affordances, the action possibilities offered by the is crucial for robot interaction 3D scenes. SceneFun3D 34] introduced challenging task and dataset for grounding referred affordance elements based on task descriptions within complex 3D indoor environments, unlike earlier works focusing on simpler settings [S] 6
environment [5 39 40 41
44 45
9
10
12 13 Fun3DU (37] further leveraged VLMs [46] and universal
segmentation model 47
parse instructions and localize the target elements in video frames.
3D Motion Estimation. 3D motion enables agents to predict and comprehend how objects move
and can be manipulated 48
57 Earlier research efforts often focused on estimating the articulated motion and mobility of individual interactable objects with These approaches typically rely analyzing the geometric structure of those individual objects to infer their motion properties. In contrast, SceneFun3D (4 broadened this by formulating a scene-level motion estimation task across all affordance elements, providing a dataset for comprehensive evaluation.
49
50
51
52
53
54
55
56
predefined structures, such hinged parts 48 40
50
MLLMs for 3D Understanding. Multimodal large language models (MLLMs) 14
L3 16
17
18 [9] are being applied to 3D understanding through two main approaches: developing native
3D 3D-aware models (30 58, 59 31 60 for direct processing of spatial data, and adapting existing by transforming 3D data into 2D representations. These efforts highlight MLLMs' potential for enhancing 3D visual understanding and semantic reasoning. Building upon these, we leverage the MILM empowered by the tailored chain-of th-tugght paradigm for fine-graine enaiina 3D embodied reasoning, jointly tackling affordance grounding and motion estimation tasks.
2D VLMs 61
62
63
Methodology
We introduce task termed Fine-grained 3D Embodied Reasoning, which aims equip embodied agents with the ability to interpret natural linguistic instructions and reason about actionable elements in complex 3D environments. Given 3D scene represented as point cloud and natural language task instruction T describing a human-intender window door"), the agent is required predict a of structured triplets {MM,, t, a )= where each triplet corresponds to referenced affordance element in the scene. Note that N=1 when only one unique element is referenced in the instruction. M, indicates a 3D instance mask identifying the spatial region of the element involved in the interaction; t; T denotes the motion type (e.g., "Translation")): a;
interaction (e.g., "open the left part of the
denotes the motion axis (e.g., "Horizontal outwards") representing the axis along which the motion occurs. The task requires joint perception and reasoning over geometry, semantics, and language intent, and presents challenges in grounding ambiguous task references, understanding object affordances, and predicting physically plausible interaction cues in 3D space.
3

---

(a) Lack of Context.
(b) Incomplete target coverage.
Figure 3: Illustrations of video-based method limitations: (a) Instructions like "Unplug the Christmas tree lights" or "Adjust the room's temperature using the radiator dial next to the curtain" require anchors (c.g., Christmas tree, curtain) that are missing from the limited video frame. (b) Target objects or parts (e.g., cabinet, door handle) in instructions like "Open the bottom drawer of the wooden cabinet..." or "Open the left part of the window door" are partially visible within the frame.
3.1 Bot Overview
To address the ine-grained 3D embodicd reasoning task, we present AffordBot, a framework that leverages MLLMs to enable instruction-conitionnn reasoning over 3D point cloud scenes. An overview of the framework is shown in Fig. 2 AffordBot integrates 3D perception with vision- language reasoning through two key components: 1) holistic multimodal representation (Sec(3.2 that bridges the modality gap between 3D input and the 2D-native input format MILMs, and 2) chain-of-thought reasoning paradigm (Sec. 3.3 that enables interpretable and accurate prediction.
For the first component, we begin by generating a set surround-view images to effectively capture the 3D scene. Following this, we extract geometric-semantic descriptors from the 3D scene and project them onto the rendered views by an adaptive labeling strategy, establishing robust 3D-to-2D association. This representation effectively eliminates traditional video processing bottlenecks while preserving comprehensive information for downstream reasoning, as illustrated in green in Fig. 2
Based on this constructed representation and the given instruction, the MLLM engages in a tailored chain-of-it oftthhughtthhtttthh process to actively perceive and select the most informative view, localize the target element, and infer its required motion. By decomposing the task into sequence of interpretable steps, our method enables the MLLM to perform robust and physically grounded inference for complex embodied tasks, depicted in blue in Fig.
3.2 Holistic Multimodal Representation
In this section, we construct the holistic multimodal representation foundational for 2D MILM reasoning. We design an enriched visual synthesis approach using dynamic surround-view generation to overcome limitations of traditional video data. Next, we describe the extraction and representation of 3D geometry and semantics via ecometry-semantiee descriptors. Finally, we establish the 3D-2D associations by projecting 3D information onto the generated 2D views with adaptive labeling.
Enriched Visual Synthesis. Bridging the gap between 3D input and 2D MLLMs is nontrivial, primarily because it requires establishing accurate and robust associations between 3D structures and their corresponding 2D visual representations.
However, these methods face fundamental limitations: due the limited field of view, is often difficult to simultaneously capture the target and its associated anchors within the same frame, as shown in Fig. 3 Furthermore, the process of extracting key information from large number of frames is both time-consuming and bottlenecks the final accuracy.
Existing methods typically rely on video sequences collected from datasets 37 34 59].
To overcome the limitations of static video frames, we propose dynamic surround- view generation strategy. Inspired by human visual exploration of unfamiliar environments, our robot performs a 360 horizontal panoramic scan centered the scene's central viewpoint. This scan produces set of N candidate views V (V....., Vv), where the i-th view V; is captured at a rotation angle 0: ( 17
Compared to relying on traditional video data, this method provides more comprehensive field of view, thereby capturing more scene information. This effectively alleviates the problem of

---

missing information caused by limited field of view and incomplete coverage typical of traditional video data obtained through random sampling. Furthermore, by scanning each 3D scene to acquire a corresponding set of high-quality views, we eliminate the overhead of performing keyframe extraction or detection for each task instruction, thereby completely removing the time processing overhead and accuracy bottleneck associated with analyzing redundant video frames. This ability enables agents to efficiently acquire comprehensive, high-quality visual context.
Soometry-Semancee Descriptors. For the input 3D scene P. our method employs instance segmen- tation extract affordance elements, and encodes their geometric and semantic features for downstream reasoning.
During training, optimize segmentation quality by combining Dice loss to encourage region-level level alignment and cross-entropy loss for accurate point-wise classification. The overall training objective is defined as:
Liotal A1 -iice + X2 LCE,
(I)
where Like denotes the Dice loss and LCE denotes the cross- cros--etropyy loss. The weights a and A2 balance the trade-off between the region-level and point-level supervision.
To handle the challenge of small element segmentation, we implement the coarse-to-fine curriculum strategy from (34] with progressive ground-truth mask dilation. At curriculum stage i, each ground-
truth mask Q is dilated within P according to the following:
Ss = 1
min | ull2 6c), = - y
Here dg is the initial dilation radius, 8 is the dilation factor, and is the step length for the dilation factor update.
Subsequently, for each predicted affordance element j. we construct its geometry descriptor Gj- which captures the element's spatial properties, formally defined Cj and Σ₁ denote the position and size, respectively. We also employ a semantic descriptor Sj. which, in conjunction with the geometric one, represents the affordance type for element j. In summary, these descriptors together form a compact yet visually unified representation of the 3D scene:
D(P) = ((C E IR3 Σ, E (,)))
where N is the total number of predicted affordance elements in the scene P. These descriptors, capturing both geometric and semantic information of the affordance elements, provide a structured representation of the scene that facilitates subsequent reasoning.
3D-2D Associations. Using the generated surround-view images V. we ground 3D affordance elements in these 2D views. For the predicted 3D elements with their descriptors D, we project their 3D geometry onto every view Vi2 V.
Specifically, compute the element's 2D bounding box projection based on its 3D position and dimensions, assigning each projected box both unique identifier linking it to the original 3D element j and its corresponding affordance type S, mapped the 2D region. This process is formalized
V = MaD (D(P),
(4)
where Man-220 denotes our projection operator. This process effectively transfers key 3D information onto the 2D images.
Furthermore, ensure legible MLLM input, we introduce adaptive-labeling refinement strategy that resolves label collisions. This involves pre-defining candidate anchor positions around each projected box. When annotating element, our pipeline iterates through these anchors, evaluating nonoverlap with existing elements, and selects the first suitable location. Such a lightweight spatial check effectively prevents label stacking, maintains clear object visibility, and provides an uncluttered canvas for subsequent MLLM reasoning.
Together, this collaborative representation enables downstream modules to access both the fine- grained geometry of affordance elements and their visual context within the scene, laying the groundwork for subsequent reasoning.
5

---

0. Multinodal Representation
1. Active View Selection
2. Affordance Grounding
3. Motion Estimation
Query
Task Prompt
Task Pompt
Yask Prompt
Opes the écctoo drawer of thee woo- den coblict with the flower de top.
Adenty she mout relevant vlow based on the equerys.
Select the part (Qts) best motch the equery,, Pt2 ler will, pract pull..
Spectt the motion ope and exis direction bared on the Selected 10s
Descriptor
AffordRRot
AffordRot
Afford
Port lit 1, Geometry: (1yx whi)) Semuntic affondence pinch pull
Splect &s This Imsge shows the woodee coblict with a flower vosr and the picture on top, and all reterned elemonts are clearyy votble
Predicted (20, 11 Part 10 and represents buttoos drooav of the cabinet hus picture frams and flower wase top curfore
Modin Type Translation, Motion Direction: Hortsonal outwerdss A drawer speus slding Bosur out. This S cransiscional movenest and its tovemmon tortzoncat To "ayee" the drower it moves outwards from the mate body the endmett
Observations
Figure 4: AffordBot's Chain-of-Thought Pipeline for Embodied Reasoning. This structured observe-then-infer process leverages multimodal inputs to perform: (1) Active View Selection to identify the most informative view, which may involve zooming in to better see the details of the images, followed by (2) Affordance Grounding localize target elements, and finally (3) Motion Estimation to infer the required action details.
3.3 Shain-of-thoughh Reasoning
This section presents our method for enabling MLLM to perform embodied reasoning. Specifically, we design a tailored chain-of-thought paradigm that follows a structured observe-then-infer pipeline, shown in Fig. Grounded in the real physical world, our pipeline leverages visual observations to guide the MLLM through a sequence of inference steps: first actively selecting the most informative viewpoint (observe), followed by identifying targets and their required interactions in context (infer).
Step 1: Active View Selection. The goal of this step is to select the most informative view from the generated surround- -veew images V. Unlike prior methods that rely on instruction parsing and heuristic filtering over pre-processed features, we leverage the multimodal reasoning capabilities of an MI.LM to guide view selection directly, enhancing both flexibility and accuracy.
Given the set of annotated views {V,,.... VN ). where each includes projected 3D elements with unique IDs and affordance types, along with the instruction T. the MLLM receives these as inputs. is then tasked with selecting the view in which all referenced elements are visible and their identifiers are clearly shown, providing the most relevant visual content. The model directs attention to the selected view Vieleste, which serves the semantically grounded visual input for subsequent reasoning.
Step 2: Affordance Grounding. This step aims to localize the specific affordance elements referenced by the linguistic instruction within the scene. Using the selected view Vssleesed from the previous step. the MLLM receives this annotated view, the original instruction T, and detailed descriptors of the 3D affordance elements, including their unique IDs and spatial attributes. The MLLM interprets the instruction and visual cues to identify the region that best matches the described task. This process yields unique IDs of the localized target elements, which serve critical intermediate decision for the subsequent motion inference stage.
Step 3: Motion Estimation. This final step focuses on inferring the motion details of the elements localized in Step 2. The MLLM is provided with the original instruction T. the image V and its localized information from the previous step. Using these inputs, the MLLM deduces the intended action for the target element, including both the motion type and the direction of the motion axis.
To enable compatibility between continuous direction vectors and MLLM outputs, we discretize the former into interpretable categories. These categories broadly distinguish motion directions as horizontal or vertical, with specific refinements for translational movements (e.g., inward/outward relative to the object's centroid) and rotational axes. This structured discretization ensures physical plausibility while remaining interpretable by the MI..M. The final output an affordance-motton tuple that integrates the discretized motion representation with the model's 's language-driven reasoning.

---

Table Quantitative comparisons our fine-grained embodied reasoning task. We report the quantitative results of affordance grounding and motion estimation task on SceneFun3D [34] dataset.
Task
Grounding
Motion
Method
Raw 2D Input
mloU
AP
AP50
AP25
+T
+TD
OpenMask3D
0.0
0.0
LERF 35
11.3
OpenMask3DDFF
17.5
OpenIns3D 66
X
0.0
0.0
0.0
0.0
Fun3DU 37
11.5
6.1
12.6
23.1
Fun3DU (+motion)
10.0
4.6
9.9
18.7
11.5
4.0
AffordBot
X
14.0
15.5
20.0
23.3
18.3
10.8
Experiments
This section presents a comprehensive experimental evaluation of our proposed fine- tineggranee embodied reasoning framework, validating its effectiveness in joint affordance grounding and motion estimation. In addition, we conduct an in-depth analysis to assess how key module optimizations impact the system's s overall accuracy and robustness.
4.1 Experimental Setup
Dataset. We conduct experiments on SceneFun3D [34), currently the only dataset that provides comprehensive annotations for fine-grained affordance grounding and motion estimation 3D indoor scenes. It comprises a total of 230 richly annotated scenes, including 200 scenes for training, 30 for validation. Each scene provides dense point clouds annotated with element-level affordance masks, motion types, and motion axis directions. To facilitate instruction-oriented embodied reasoning for our task, we curate the annotation with task task-specific annotation triples.
Evaluation Metrics. We adopt standard evaluation metrics (34] to assess performance on 3D affordance grounding and motion estimation. Specifically, report mean Intersection-over-Uniie (mloU), mean average precision (mAP), and average precision at loU thresholds (AP, AP25, AP50), between predicted and ground-truth masks. To incorporate motion parameter accuracy, adapt the AP25 metric as proposed in [34]5]] 51 [50), extending it with additional constraints on motion type and direction. Specifically, we further constrain mask prediction based whether the model correctly predicts the motion type (+T), and both the motion type and motion axis direction (+TD).
Implementation Details. For
reasoning, employ Owen2.5-VL-722. [1] locally
deployed on four NVIDIA A800 GPUs. To construct the geometry descriptors and get segmented
elements masks, fine-tune Mask3D (65] from pretrained checkpoint ScanNer200 [67]. We
train for 1,000 epochs on an NVIDIA A800 with the learning rate of 0.0001, a batch size of 2, and 2cm voxelization preserve spatial detail.
4.2 Quantitative Results
The quantitative results for the fine-grained embodied reasoning on SceneFun3D dataset, as presented T demonstrate the effectiveness of our AffordBot approach. By encoding 3D scenes into structured representations and processing them with task instructions via the MILMM, AffordBot outperforms existing methods including OpenMask3D [36] 34), LERF [35). OpenIns3D (66), and Fun3DU (37), well as enhanced Fun3DU (+motion) bascline. As shown in the table, AffordBot reports higher scores in both affordance grounding and motion estimation.
Notably, the results of our reproduced Fun3DU (+motion) baseline (second to last row) highlight the impact of incorporating motion estimation branch into their original affordance grounding framework. For this baseline, we prompted Molmo [46] to infer motion parameters based on AffordBot across all reported metrics underscores the advantage of our approach in accurately identifying and understanding the potential motions associated with affordance elements in 3D scenes. Specifically, higher AP score, which is the average precision over IoU thresholds ranging from 0.5 to 0.95,
segmentation results of affordance elements. The significant outperformance

---

Table 2: Ablation on key components of our Afford- Bot. ALR denotes Adaptive Label Refinement, EVS denotes Enriched Visual Synthesis, and AVS denotes Active View Selection. Each variant incrementally in- corporates one module, and finally Ex4 corresponds to our AffordBot.
Table Ablation on viewpoint selection. 'BEV" projects bird-eye view; 'Video Frame' uniformly samples frames from dataset, while 'Query-Aligned" picks the query-matching view; 'Ours' renders sur- round views for MLLM selection.
ALR
EVS
AVS
AP
AP50
AP25
Method
AP
AP
AP25
Ex1
9.7
12.8
15.7
BEV
6.1
9.1
12.7
Ex2
9.7
13.0
16.1
Video Frame
9.4
11.4
15.6
Ex3
14.8
19.4
22.1
Query-Aligned
9.7
13.0
16.1
Ex4
/
15.5
20.0
23.3
Ours
15.5
20.0
23.3
indicates that AffordBot not only performs well in rough localization but also maintains high precision under stricter localization requirements (i.e., higher IoU thresholds). This is crucial for robotic manipulation tasks, where precise segmentation is essential for accurate grasping and manipulation. Furthermore, the results suggest the importance of grounding accuracy for subsequent tasks and indicate the enhanced spatial awareness provided by 3D-based motion reasoning.
4.3 Ablation Studies
To quantify the contribution of AffordBot, conduct ablation studies focusing solely the poordance-groundiing task, which is the critical prerequisite for motion estimation and downstream execution. This targeted approach is justified because the investigated modules (representation design and the MLLM-driven view-selection mechanism) operate entirely upstream of motion estimation. Once a target is accurately grounded, motion prediction relies solely on that grounded object and a fixed MLLM prompt. Consequently, any modifications to these upstream components propagate through the entire pipeline and are comprehensively reflected in grounding performance metrics.
Ablation on Key Components. Through systematic component wise analysis, we demonstrate how progressive module integration contributes the performance, as shown in Tab. 2 Baseline Ex adapted from [68]. initially employs the MLLM to parse instructions and identify target affordance types. It then renders all matching segmented elements from the scene's ' center view, annotating each with unique identifiers at their 2D centroids. Finally, the MILM processes these rendered views to localize the correct element. Adding Adaptive Label Refinement (ALR, Ex2) identifier labels to avoid occlusion, our method yields modest but consistent lift of +0.4% AP25- The major improvement comes from enriched visual synthesis (EVS, Ex3). The model gains much richer context, pushing AP25 from 16. 5 to 22. 2.11% (+6.0%). This significant improvement demonstrates that global, information-denss. observation, single frame used in (68). Finally, Ex4 employs the active view selector (AVS). This seeus-theniinfe pipeline both trims redundant visual information and exploits the best evidence, raising AP25 to 23.3% and achieving the highest overall accuracy.
provided by EVS, is much more valuable than the
Ablation Viewpoint selection. To probe how viewpoint choice affects subsequent inference, evaluate this in Tab. 3 While Bird's-Eye View (BEV) representations provide scene overview, they prove ineffective for our task as affordance elements typically require fine-grained appearance details due to their small size. Sampling images directly from the video stream (i.e. Video Frame baseline) also yields limited effectiveness, outperforming the previous results. Query-aligned baseline retrieves a single pre-tagged frame whose affordance class matches the query.
Our Dynamic strategy takes a different route: it first synthesises dense 360 sweep of surround views, then asks the MLLM to select the frame that best matches the instruction. This active "observe- then-infer" routine supplies rich global context while keeping the final input compact, boosting AP to 23.3%, an absolute gain of 7.6% over the query- method, as shown in Tab.
Probing the Primary Bottleneck of AffordBot. To investigate the primary bottlenecks constraining upstream segmentation and downstream active view selection performance, we progressively replace Mask3D's 's predicted masks with ground-truth masks (GT proposals) and provide an ideal front-view perspective (GT proposals + views), as summarized in Table 4 The first row shows the baseline configuration (Mask3D proposals), which corresponds to AffordBot. Replacing the predicted
8

---

Table Probing the bottleneck of our method. "Mask3D proposals' refers to our Affordbot, which uses predicted proposals. 'GT proposals' denotes the use of ground-truth masks, while 'GT proposals views' additionally adopt ground- truth views.
Table 5: Comparison different MLLMs. Deployable models (LLaVA-v1.6-34BB Qwen2.5-VL-722) and commercial GPT APIs show consistent trends, with larger models yiclding stronger performance.
Method
AP
AP
AP25
Method
AP
AP50
AP25
LLaVA-v1.6-344
10.6
14.2
16.9
Mask3D proposals
15.5
20.0
23.3
Owen2.5-VL-72B
15.5
20.0
23.3
GT proposals
35.7
39.4
45.4
GPT-4o
16.5
22.1
28.9
GT proposals views
38.3
42.3
47.4
GPT-ol
24.8
30.3
33.4
Table 6: Performance variation across affordance types. 'Segment" measures upstream segmenta- tion accuracy, while 'Reason' reports final grounding.
Type
rotate
key_press
press
tip_push
book pull
pinch pull
hook turn
foot_ foot_pussh push
plug in
unplug
Segment
0.0
11.3
5.3
27.5
27.1
67.8
100.0
11.1 8.3
15.3
Reason
2.5
30.4
5.1
18.0
23.5
45.1
100.0
16.7
Table 7: Performance variation with different numbers of target elements. We compare tasks involving single ground-truth element ('Unique') versus multiple elements ('Multiple')
Method
mIoU
AP
APso
AP25
+T
+TD
Unique
13.2
13.8
18.2
21.4
16.5
9.9
Multiple
19.1
27.2
32.1
35.8
30.2
17.0
Overall
14.0
15.5
20.0
23.3
18.3
10.8
masks with GT proposals results in 22.1% boost in AP25, highlighting that instance segmentation factor. With perfect segmentation, adding the optimal viewpoint further improves performance by +2.0% AP25, suggesting that while active perception still be optimized, it is not the dominant bottleneck.
noise is the primary factor.
Comparison of Different MILMs. We replace the default Swven2.5-VL-722 with several representa- tive alternatives (L.LVVA-vl.6) -.3344, GPT-40, and GPT-ol), as reported in Tab. While the commonly used Qwen achieves 23.3% AP25. adopting the more advanced GPT-ol, which features superior reasoning and visual understanding, further boosts performance to 33.4% AP25. This demonstrates that leveraging stronger MLLMs can unlock even greater potential within our framework.
Performance Variation Analysis. We conduct a detailed performance analysis to uncover variations across different affordance types and target element counts. Tab, blighlligs significant disparities in APso across different affordance types, partly reflecting dataset class imbalance and strong dependence on initial segmentation quality. Performance ranges widely (e.g., 100% for "foot_ "apppppss"" vs. 0% for "rotate")) "), limiting grounding accuracy. The MILM improves performance for some categories using linguistic cues (e.g., "key_press"" but challenges remain in aligning visual and language cues for others (e.g., rii_push" degradation).
Further analysis of task subsets (Tab. reveals better performance for "Multiple" target elements than "Unique" ones. This difference is primarily due Mask3D struggling with the small, weakly- textured objects typical of "Unique" instances, resulting in noisier descriptors that hinder subsequent affordance grounding and motion estimation.
4.4 Qualitative Results
Fig. 5 provides detailed qualitative results our fine-grained reasoning task. AffordBot generally shows more accurate and consistent grounding of target elements, particularly in complex scenarios with multiple small targets. Overall, qualitative evidence further suggests that AffordBot achieves significantly improved affordance understanding compared to the prior SOTA method [37].

---

Input
Fun3DU
Ours
RGB
Figure 5: Qualitative Results. The figure showcases visual examples of AffordBot performing fine-grained grounding. The illustrated examples include: (1) "Turn on the TV using the remote control on the table." " (2) "Open the middle drawer the TV stand." (3) "Close the bedroom door."
(4) "Open the window above the radiator". Please zoom in digitally to view more details.
5 Conclusion
Fine-grained embodied reasoning in 3D worlds serves crucial bridge from perception to action, making pivotal for agents to perform sophisticated tasks. While prior work has explored affordance grounding and motion estimation in isolation, our work unifies these tasks under structured reasoning framework, AffordBot, bridging perception and action through instruction awwae triplet prediction. By leveraging MILM with tailored chain-of-thought paradigm, method ensures physically grounded reasoning that advances from scene perception to affordance localization and motion synthesis. Through extensive experiments, AffordBot not only advances state-of of-the-art performance but also demonstrates the feasibility of efficient, task-coherent embodied reasoning, paving the way for more intuitive human-agent interaction in complex 3D spaces. In the future, will empower AffordBot with more advanced multimodal understanding ability 669
Acknowledgments
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant U22A2094, and also supported by the Ministry of Education, Singapore, under its MOE Academic Research Fund Tier 2 2000--22PP0122-00111 We also acknowledge the support of the Supercomputing Center USTC for providing advanced computing resources and NSFC with Grant No. 62573371.
10

---

References
[1] Mohammed Hassanin, Salman Khan, and Murat Tahtali. Visual affordance and function understanding: survey. ACM Computing Surveys (CSUR), 54(3):1-35, 2021.
[2] Paola Ardón, Eric Pairet, Katrin S Lohan, Subramanian Ramamoorthy, and Ronald Petrick. Building affordance relations for robotic agents-a review. arXiv preprint arXiv:2105.06700 2021.
[3] Dongpan Chen, Dehui Kong, Jinghua Li, Shaofan Wang, and Baocai Yin. A survey of visual affordance recognition based on deep learning. IEEE Transactions on Big Data, 9(6):1488 1476, 2023.
[4] James J Gibson. The theory of affordances:(1979). In The people, place, and space reader, pages 56-60. Routledge, 2014.
[5] Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A benchmark for visual object affordance understanding. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1778-1787, 2021.
[6] Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. 020-afford: Annotation-fis large-scale object-object affordance learning. In Conference on robot learning, pages 1666-1677. PMLR, 2022.
[7] Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Jiebo Luo, and Zheng- Jun Zha. Grounding 3d object affordance from 2d interactions in images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10905-10915, 2023.
[8] Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction exploration in 3d environments. Advances in Neural Information Processing Systems, 33:2005 2015, 2020.
[9] Chao Xu, Yixin Chen, He Wang, Song-Chin Zhu, Yixin Zhu, and Siyuan Huang. Partafford: Part-level affordance discovery from 3d objects. arXiv preprint XrX:: 13519, 2022.
[10] Ruihai Wu, Kai Cheng. Yan Zhao, Chuanruo Ning, Guanqi Zhan, and Hao Dong. Learning environment- aware affordance for 3d articulated object manipulation under occlusions. Advances in Neural Information Processing Systems, 36, 2024.
[11] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi. Mare Pollefeys. Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocahularies. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815-824, 2023.
[12] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. WhereZact: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813 6813-6823, 2021.
[13] Chuanruo Ning, Ruihai Wu, Haoran Lu, Kaichun Mo, and Hao Dong. WhereZexplore: Few-shot affordance learning for unseen novel categories articulated objects. Advances in Neural Information Processing Systems, 36, 2024.
[14] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 121911 2024.
[15] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li. Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv: 13923, 2025.
[16] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions tokens of context. arXiv preprint arXiv:2403.05530, 2024.
[17] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19744. PMLR, 2023.
[18] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen. Sen Xing. Muyan Zhong Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguisin tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185 24185-24198, 2024.

---

[19] Deyao Zhu, Jun Chen, Xiaoqian Shen. Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:330411559.. 2023.
[20] Zhen Zeng, Leijiang Gu, Xun Yang. Zhangling Duan, Zenglin Shi, and Meng Wang. Visual-oriented fine- lineggrined knowledge editing for multimodal large language models. In ICCV, 2025.
[21] Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. Finding and editing multi-moda neurons in pre-trained transformers. In Findings of the Association for Computational Linguisties: ACI. 2024, pages 1012-1037, 2024.
[22] Yucheng Han, Na Zhao, Weiling Chen, Keng Teck Ma, and Hanwang Zhang. Dual-perspective knowledge enrichment for semi-supervised 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2049-2057, 2024.
[23] Pengkun Jiao, Na Zhao, Jingjing Chen, and Yu-Gang Jiang. Unlocking textual and visual wisdom: Open- vocabulary 3d object detection enhanced by comprehensive guidance from text and image. In European Conference on Computer Vision, pages 376-392. Springer, 2024.
[24] Linfeng Li and Na Zhao. End-to-end sumi-supervised 3d instance segmentation with peteacher. In 2024 IEEE International Conference on Robotics and Automation (ICRA). pages 5352-5358, IEEE, 2024.
[25] Jiangyi Wang and Na Zhao. Uncertainty meets diversity: A comprehensive active learning framework for indoor 3d object detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20329-20333 2025.
[26] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Few-shot 3d point cloud semantic segmentation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 8873-8882, 2021.
[27] Chaofan Luo, Donglin Di, Xun Yang. Yongjia Ma, Zhou Xue, Chen Wei, and Yebin Liu. Trame: Trajectory- anchored multi-view editing for text-guided 3d gaussian splatting manipulation. IEEE Transactions on Multimedia, 2025.
[28] Dave Zhenyu Chen, Angel X Chang, and Matthias NieBner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-2211 2221.. Springer. 2020.
[29] Xinyi Wang, Na Zhao, Zhiyuan Han, Dan Guo, and Xun Yang. Augrefer: Advancing 3d visual grounding via eross-modal augmentation and spatial relation-based referring. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8006-8014, 2025.
[30] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023.
[31] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun I Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 2023.
[32] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428 20442,6438822 2024.
[33] Kuan-Chih Huang, Xiangtai Li, Lu Qi, Shuicheng Yan, and Ming-Hsuan Yang. Reason3d: Searching and reasoning 3d segmentation via large language model. In International Conference on 3D Vision 2025, 2025.
[34] Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. Scenefun3d: Fine-grained functionality and affordance understanding in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14531- 14542, 2024.
[35] Justin Kerr, Chung Min Kim, Ken Goldberg, Angioo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729 19739, 2023.
12

---

[36] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. arXiv preprini arXiv:2306.1363 2023.
[37] Jaime Corsetti, Francesco Giuliari, Alice Fasoli, Davide Boscaini, and Fabio Poiesi. Functionality understanding and segmentation in 3d scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2025.
[38] Zhihao Yuan, Shuyi Jiang. Chun-Mei Feng, Yaolun Zhang, Shuguang Cui, Zhen Li, and Na Zhao. Scene-rl: Video-grounded large language models for 3d scene reasoning without 3d annotations. arXiv preprint arXiv:2506. 17545, 2025.
[39] Thanh-Toan Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In 2018 IEEE international conference on robotics and automation (ICRA), pages 5882 5889, IEEE, 2018
[40] Yicong Li. Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language- guided affordance segmentation 3d object. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14251-14260, 2024.
[41] Timo Luddecke and Florentin Worgotter. Learning to segment affordances. Proceedings of the IEFE International Conference on Computer Vision Workshops, pages 769-776, 2017.
[42] Xue Zhao, Yang Cao, and Yu Kang. Object affordance detection with relationship-awar network. Neural Computing and Applications, 32(18);14321-14333 2020.
[43] Gen Li, Deqing Sun, Laura Sevilla-Larg and Varun Jampani. One-shot open affordance learning with foundation models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3086-3096, 2024.
[44] Edmond Tong, Anthony Opipari, Stanley Lewis, Zhen Zeng, and Odest Chadwicke Jenkins. Oval-prompt: Open-vocabulary affordance localization for robot manipulation through Ilm affordance-grounding arXiv preprint arX4::2404. 1000, 2024.
[45] Claudia Cuttano, Gabriele Rosi, Gabriele Trivigno, and Giuseppe Averta. What does clip know about peeling banana? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2238-2247, 2024.
[46] Matt Deitke, Christopher Clark, Sangho Lec, Rohun Tripathi, Yue Yang, Jac Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409. /7146, 2024.
[47] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEF/CVF international conference on computer vision, pages 4015-4026, 2023.
[48] Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang. Opd: Single-view 3d openable part detection. In European Conference on Computer Vision, pages 410-426. Springer, 2022.
[49] Cheng Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In 2023 IEER International Conference Robotics and Automation (ICRA), pages 3933-3939. IRFF, 2023.
[50] Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel Chang. Opdmulti: Openable part detection for multiple objects. 2024 International Conference on 3D Vision (3DV), pages 169-178. IEEE, 2024.
[51] Yongsen Mao, Yiming Zhang. Hanxiao Jiang, Angel Chang, and Manolis Savva. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. Advances in neural information processing systems, 35:9058-900 2022.
[52] Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decomposition for man man-made articulated objects. In European Conference on Computer Vision, pages 558-575. Springer, 2022.
[53] Xiaogang Wang. Bin Zhou, Yahao Shi. Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876 8884, 2019.
13

---

[54] Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song. Category-leve articulated object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 3706-3715, 2020.
[55] Xianghao Xu, Yifan Ruan, Srinath Sridhar, and Daniel Ritchie. Unsupervised kinematic motion detection for part-segmented 3d shape collections. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[56] Liu Liu, Jianming Du, Hao Wu, Xun Yang. Zhenguang Liu, Richang Hong, and Meng Wang. Category- level articulated object 9d pose estimation via reinforcement learning. In Proceedings of the 31 ACM International Conference on Multimedia. pages 728-736, 2023.
[57] Liu Liu, Anran Huang. Qi Wu, Dan Guo, Xun Yang, and Meng Wang. Kpa-tracker: Towards robust and real-time category- level articulated object 6d pose tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pages 3684-3692, 2024.
[58] Yilun Chen, Shuai Yang. Haifeng Huang. Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405. 2024.
[59] Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.014 004428, 2025.
[60] Donglin Di, Jiahui Yang, Chaofan Luo, Zhou Xue, Wei Chen, Xun Yang, and Yue Gao. Hyper-3dg; Text- to-3d gaussian generation via hypergraph. International Journal of Computer Vision, 13365):2886-2909 2025.
[61] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403. 1401, 2024.
[62] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. Advances in Neural Information Processing Systems. 36:37193-37229, 2023.
[63] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang. Xuesong Niu. Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded seene understanding. In European Conference on Computer Vision, pages 289-310. Springer, 2024.
[64] Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging scene and large language models with ohject identifiers. In The Tiihtt- Annual Conference on Neural Information Processing Systems, 2024.
[65] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9216-8223. IEEE, 2023.
[66] Zhening Huang. Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3:: Snap and lookup for 3d open-vocabulary instance segmentation. European Conference on Computer Vision, 2024.
[67] David Rozenberszki, Or Litany, and Angela Dai. Languaggc-groudder indoor 3d semantic segmentation in the wild. In Proceedings the European Conference on Computer Vision (ECCV), 2022.
[68] Rong Li, Shijie Li. Lingdong Kong, Xulei Yang, and Junwei Liang. Seeground: See and ground for zero-shot open- vocabulayyy 3d visual grounding. arXiv preprint wrXiv:2412.04385 2024.
[69] Zhiyuan Han. Beier Zhu, Yanlong Xu. Peipei Song, and Xun Yang. Benchmarking and bridging emotion confficts for multimodal emotion reasoning. In ACM Multimedia, 2025.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
14

---

You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation.
While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provvdedd proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We prove our claim through quantitative and qualitative experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
15

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
There is no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Yes, we will release the inference code on GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed
16

---

reproducible, they instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: No, We are committed to reproducible research and plan to publicly release our source code upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
www.sammmsisiooooooooiiooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ https//ccpubbcccuuudddeooo http://dddd//eeSuubmmmmm wwttimissioooooliiioc
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, see the Experiments section and Appendix for details.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No, error bars are not available.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///herrrg...ccoaaaa//hhhhp
Answer: [Yes]
Justification: Yes, the research the paper conform, in every respect, with the NeurIPS Code of Ethics.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, all existing assets used in this paper are properly credited with citations. And the license of assets are listed in Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not crowdsourcing conduct research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: the study described in this paper, no potential risks were identified for participants. The research design was carefully crafted to ensure the safety and well-being all individuals involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes)
Justification: Yes, see the Method and Experiment section for details.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettts://neur// wwteeiippsccccoooeeeee /C//ooeerenccee2225 /LLM) for what should or should be described.
21


Reviews:
[{"review_id": "a20ip8PSI9", "reviewer": "Reviewer_DXFg", "summary": "This paper proposes a new task termed fine-grained 3D embodied reasoning, which jointly predicts 3D affordance masks, motion types, and motion axes in a 3D scene, conditioned on natural language instructions. To tackle this task, the authors introduce AffordBot, a framework that leverages surround-view synthesis and 3D-to-2D projection to transform point cloud input into a holistic multi-modal representation. A tailored chain-of-thought (CoT) paradigm then enables MLLM (e.g., Qwen2.5-VL-72B) to actively select a view and reason step-by-step over the scene. The approach achieves state-of-the-art performance on the SceneFun3D benchmark, demonstrating strong affordance grounding and motion reasoning capabilities.", "questions": "- How is the information from the geo-sem descriptor injected into the selected 2D views? Directly mark the segments and semantics on 2D images? Or represent in textual formats such as json?\n- Why is AffordBot marked “without 2D input” in Table 1? Though starting with 3D point clouds, AffordBot takes as input 2D views and perceives the scene with a 2D visual encoder. Hence, I think AffordBot utilizes 2D input.\n- Typo: Line 307 “agentS”", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "#### Strength\n- This paper proposes a new task that unifies affordance understanding and interaction estimation. The motivation is sound and the authors formulate this task well, representing the output with three joint parts: affordance grounding, motion type and motion axis.\n- Well-designed multi-modal representation. The surround-view synthesis and adaptive 3D-to-2D projection pipeline successfully transforms 3D point cloud data into an MLLM-compatible format, avoiding the limitations of traditional video-based input and enabling richer spatial context.\n- Extensive experiments and analyses. The authors conduct in-depth analyses such as on how different components contribute to the final performance, which shows the significance of enriched visual synthesis.\n\n#### Weaknesses\n- The motion axis is represented by a few text templates specifying the motion direction, which limits the expressiveness. And the prediction regarding such coarse templates can likely be handled simply by LLM’s capability of commonsense reasoning.\n- The framework of AffordBot could confront cumulative error from the upstream segmentation module, i.e., the whole performance can be bottlenecked by the geo-sem descriptor. As shown in Table 4, the segmentation accuracy is quite low on some categories, which further consolidates the concern.\n- For the proposed fine-grained 3D embodied reasoning task, I think the relationship to “embodied” is somewhat weak. It appears that the task does not require embodied awareness and can be handled by a non-embodied generic model. And for the proposed framework AffordBot, the only embodied-related part is enriched visual synthesis that generates views based on a specific anchor.", "was_revised": true, "final_justification": "Technically solid paper. I lean towards accept.", "rebuttal": "Thank you for your valuable comments. We are pleased that the reviewer recognizes our contribution, particularly its clear motivation, well-designed method, and thorough experimental analysis. Please see our responses below.\n\n**W1: Expressiveness of the motion-axis templates.**\n\nThe discrete motion-axis templates adopted in our framework serve as a directly executable semantic interface that covers common situations in the real physical world. Specifically, the majority of affordance elements (e.g., \"drawer\", \"plug\") are aligned with primary axes. Thus, discretizing motion directions into this set of semantic categories is not merely a simplification, but a design choice grounded in the spatial regularity of real-world environments. Moreover, the predicted motion in our framework functions as a high-level semantic signal—analogous to instructions like \"pull outward horizontally\". This abstraction enables our high-level system to focus on semantic reasoning, while the execution details will be handled by the downstream low-level action system. As reviewer #m6AC noted, this design provides a practical and interpretable bridge between multimodal LLMs and robotic control, and has also been validated by prior work such as [1]. We appreciate the reviewer’s insight and plan to extend the current design toward finer-grained primitives or continuous vectors in future work.\n\n**W2: Influence of segmentation quality.**\n\nThank you for the insightful comment. We agree that upstream segmentation quality affects overall performance. To assess this, we conducted an oracle study using ground-truth masks, which improved AP50 from 20.0 to 39.4, highlighting potential gains from better segmentation. Despite current imperfections, our framework still significantly outperforms the previous SOTA Fun3DU. We plan to explore MLLM-based refinement to further enhance segmentation and will include this analysis in the revised version.\n\n| Upstream Method   | mAP  | AP50 | AP25 |\n|-------------------|------|------|------|\n| Mask3D proposals  | 15.5 | 20.0 | 23.3 |\n| GT proposals      | 35.7 | 39.4 | 45.4 |\n\n**W3: Embodied relevance.**\n\nWe understand the reviewer's concerns about the degree of \"embodiment\". We would like to clarify that, by definition, our task inherently requires direct interaction with the real environment, which we believe constitutes a meaningful form of embodiment. Whether in terms of task conditions or application scenarios, the model must perceive and adjust decisions in real time within the physical world, represented by the 3D point cloud of the scene. This key distinction sets our approach apart from typical non-embodied visual question answering. Our task serves as the high-level semantic reasoning component in the robotic execution chain, with outputs directly interfacing with low-level action controllers to form a complete task pipeline. Although some sub-steps seem to be replaceable by non-embodied methods (e.g., OpenMask3D [2], Fun3DU [3]), as shown in Table 1 of the paper, their performance is significantly lower than ours. Our framework exhibits embodiment in several concrete ways. First, the surround-view synthesis and active view selection simulate the perception process of an embodied agent observing a real scene. Second, the Chain-of-Thought reasoning and prompt design operate on multi-modal inputs grounded in 3D space, enabling high-level semantic decisions based on spatial cues. Third, the model outputs target locations, motion types, and axes as interpretable signals that guide downstream action execution. Together, these components form an observation–reasoning–action chain tailored for embodied tasks, rather than passive visual understanding.\n\n**Q1: Explanations of geo-sem descriptor injection.**\n\nAs illustrated in Section 3.2 (Main Paper) and Figure 4 (Supplementary), we first project each 3D element’s bounding box onto the rendered 2D view, producing a corresponding 2D region bounding box. Each 2D box is assigned a unique ID, ensuring a one-to-one correspondence between 3D elements and their 2D projections. We then convert the geometric (e.g., center coordinates, box size) and semantic (e.g., affordance type) information of each element into natural language and append it to the MLLM prompt. Instead of using a separate JSON file, we integrate all geo-sem descriptors as inline text alongside the image, which enables the language model to jointly reason over visual, geometric, and semantic information in a unified manner.\n\n**Q2: Clarifications of input modality.**\n\nWe would like to clarify that our intent of \"without 2D input\" refers specifically to not relying on any **raw 2D input** provided by the dataset. AffordBot operates solely on 3D point cloud and generates all 2D views internally via online surround-view rendering. These rendered images serve as intermediate representations, not external inputs. To avoid confusion, we will revise the column header in Table 1 to read \"2D raw input\" for clarity.\n\n**Q3: Minor issues.**\n\nThanks for your feedback, and we will correct this in the revised version.\n\n**References:**\n\n[1] Zawalski M, Chen W, Pertsch K, et al. Robotic control via embodied chain-of-thought reasoning[J]. arXiv preprint arXiv:2407.08693, 2024.\n\n[2] Takmaz A, Fedele E, Sumner R W, et al. Openmask3d: Open-vocabulary 3d instance segmentation[J]. arXiv preprint arXiv:2306.13631, 2023.\n\n[3] Corsetti J, Giuliari F, Fasoli A, et al. Functionality understanding and segmentation in 3D scenes[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24550-24559."}, {"review_id": "8DwLd5C4bZ", "reviewer": "Reviewer_qb6v", "summary": "This paper introduces AffordBot, a method that uses 2D MLLM for embodied reasoning in 3D scenes. It utilizes view synthesis and Geometry-Semantic Descriptors for scene representation, then propose to use MLLM for step-by-step affordance grounding by first select the most informative view and then ground the action label and axis. The resulting pipeline demonstrates improved performance on SceneFun3D dataset in terms of grounding accuracy and motion prediction accuracy. Extensitve ablation studies show the effectiveness of the proposed method.", "questions": "1. Have the authors tried to understand how the model would choose between views when the task can be applied to multiple instances?\n\n2. In the abstract, the authors state: we introduce a new task: Fine-grained 3D Embodied Reasoning. How is this task different from SceneFun3D?\n\n3. Have the authors tried to understand the effect of changing the number of views?", "limitations": "See Questions and Weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. AffordBot proposes a scene representation and a reasoning chain that effectively help with better affordance and motion grounding in the 3D scene.\n\n2. Extensive experiments and ablation studies show the effectiveness of the proposed method.\n\n3. Analyses of the experimental results are comprehensive.\n\nWeaknesses:\n\n1. The proposed view synthesis method seems to be less effective when the structure of the house is complex, for examle, taking a 360 views at the center of house 0 in ScanNet will result in no observations of the restroom and some cabinets in the kitchen.\n\n2. The proposed active view selection chose the most informative view for the downstream grounding, which could cause the issue of ambiguity when there're multiple objects of the same category.\n\n3. To avoid the problems raised in Figure 3, the number of frames generated by the view synthesis stage would be huge to get a good coverage and enough context for the downstream tasks, which could be inefficient.", "was_revised": true, "final_justification": "The author has mainly addressed my concerns and solved my confusion about the task setting w.r.t SceneFun3D. I'll increase my scroe to 5", "rebuttal": "Thank you for your valuable comments. We are pleased that the reviewer recognizes our method's effectiveness, thorough experiments, and comprehensive analyses. Please see our responses below.\n\n**W1: View‑coverage considerations in complex environments.**\n\nThank you for this valuable insight. In our current experiments, we use the SceneFun3D dataset, where annotated affordance elements are densely distributed in the main room. Under this distribution, our surround‑view synthesis strategy can cover almost all labeled elements. We agree that in environments with complex layouts and multiple sub‑rooms, such as those found in ScanNet, a single panoramic sweep may fail to capture occluded areas like restrooms. While our current implementation demonstrates an initial step toward the fine-grained embodied reasoning task, we plan to extend this in future work by developing more comprehensive observation strategies, including incremental observation, to better handle complex scenes.\n\n**W2&Q1: Multi‑instance scenarios.**\n\nWe thank the reviewer for the thoughtful insights. Indeed, in scenarios where multiple affordance instances match the instruction, the language input often includes referential cues (such as \"open the top drawer of the right cabinet\") to help disambiguate. In our current pipeline, we let the MLLM itself select the most informative view, based on the tailored prompt and the multimodal representation. This design allows the model to leverage spatial references in language to resolve ambiguity among identical objects. During the subsequent reasoning step, the MLLM is required to locate all instances that satisfy the instruction, which may include one or multiple valid instances. Even without explicit disambiguation modules, our method demonstrates stronger understanding than previous work.\n\n**W3&Q3: Analysis of view coverage and quantity.**\n\nWe respectfully disagree. The images shown in Fig. 3 are original RGB frames from the dataset, not our synthesized views. Precisely to address the limited context in such original observations, our surround-view synthesis module (Figure 2, middle) generates comprehensive coverage using only 8 synthesized views, significantly reducing the number of frames required while mitigating the visibility issues shown in Figure 3. To understand the impact of view quantity, we test 2, 4, 8, and 16 views. As shown in the table below, even 4 views offer strong coverage and performance, while 8 views provide complete 360° scanning with peak accuracy. However, increasing to 16 views leads to performance degradation due to redundant overlaps and the limitations of our MLLM (Qwen2.5-VL-72 B), which supports at most 9 images—forcing us to tile views and reduce resolution. Given this circumstance, we adopt 8 views as the best trade-off between efficiency and accuracy and will include this analysis in the revised version.\n\n| No. of Views | mAP  | AP50 | AP25 |\n|--------------|------|------|------|\n|      2       | 11.5 | 14.7 | 18.5 |\n|      4       | 14.5 | 19.0 | 22.6 |\n|      8       | 15.5 | 20.0 | 23.3 |\n|      16      | 10.1 | 13.9 | 17.6 |\n\n**Q2: Difference from SceneFun3D tasks.**\n\nAs outlined in Section 1, SceneFun3D provides multiple independent tasks (affordance segmentation, affordance grounding, and motion estimation) with separate benchmarks. However, this pipeline processes subtasks in isolation and predicts motions for all functional parts independently of user instructions, limiting its applicability in real-life scenarios. To overcome these, our task unifies affordance-mask localization, motion type, and motion axis into a structured triplet prediction, enabling joint spatial grounding and interaction reasoning in a single stage. Besides, our task estimates motion parameters only for instruction-relevant affordances, while SceneFun3D operates in an instruction-agnostic manner. These differences in task formulation and output granularity demonstrate that our benchmark extends beyond SceneFun3D and better aligns with real-world embodied task demands."}, {"review_id": "mVQFs38nUh", "reviewer": "Reviewer_m6AC", "summary": "This paper teaches an agent to jointly perform 3D affordance grounding and motion estimation based on a natural language instruction. For a given 3D scene and command, the agent must predict a structured triplet (3D mask, motion type, motion axis) for each relevant interactable part. To address this task, the authors propose AffordBot, a novel framework that leverages Multimodal Large Language Models (MLLMs). A key aspect of the framework is its method for bridging the 3D-to-2D modality gap by rendering surround-view images of the scene and projecting 3D element candidates onto them. AffordBot then employs a tailored Chain-of-Thought (CoT) reasoning paradigm, which notably begins with an \"active perception\" stage where the MLLM selects the most informative viewpoint before proceeding to localize the target element and infer its motion parameters. Experiments on the SceneFun3D dataset show that AffordBot achieves state-of-the-art performance.", "questions": "1.  The absolute performance, while state-of-the-art, remains relatively low (e.g., an overall AP of 15.5 and AP50 of 20.0). This suggests the task is extremely challenging. Could the authors provide a more in-depth analysis of the primary bottlenecks? Based on your experiments, what are the main sources of error? Is it the upstream segmentation quality (as hinted in Table 4), the MLLM's spatial reasoning capabilities, ambiguities in the language instructions, or the inherent difficulty of the dataset itself? A detailed error analysis would be highly valuable for future work in this area.\n\n2.  The active view selection mechanism is a core contribution. Could you elaborate on its failure modes? Are there cases where the MLLM selects a suboptimal or incorrect view, and if so, how does the system's performance degrade? An analysis of the view selector's standalone accuracy and its downstream impact on the grounding and motion estimation tasks would provide deeper insight into its robustness.\n\n3.  The experiments are conducted using the Qwen2.5-VL-72B model. How dependent are the framework and the CoT paradigm on this specific MLLM? Have you experimented with other MLLMs (e.g., LLaVA, OpenAI o-series)? It would be helpful to understand whether the success is primarily due to the power of this specific model or the generality of the AffordBot framework itself.\n\n4. As far as I see, the advantage that a vision-language model brings in this task is the possibility to conduct long-horizon reasoning in order to finish a certain goal. Have you tried the capability of AffordBot in such settings like [1]?\n\n[1] PreAfford: Universal Affordance-Based Pre-Grasping for Diverse Objects and Environments", "limitations": "Please see the weakness section.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1/ Effective Framework Design: The AffordBot framework is well-conceived and effectively tackles key challenges. The method for creating a rich multimodal representation from a 3D point cloud—by synthesizing surround-views and projecting 3D geometric-semantic descriptors—is a smart solution to make 3D scenes accessible to 2D-native MLLMs without relying on computationally expensive video processing.\n\n2/ Thorough Experimental Validation: The paper provides a comprehensive set of experiments that convincingly validate the proposed approach. The overall results demonstrate a clear state-of-the-art performance. The component-wise ablation studies are particularly strong, systematically quantifying the contribution of each module (ALR, EVS, AVS) and providing clear evidence for the authors' design choices.\n\nWeaknesses\n\n1/ Heavy Reliance on Upstream Segmentation: The performance of the entire AffordBot pipeline is critically dependent on the quality of the initial 3D instance segmentation from the Mask3D model. The authors' own analysis in Table 4 shows that poor segmentation accuracy for certain affordance types directly leads to low final grounding accuracy. This indicates that errors from the upstream module can propagate and are not recoverable, making the system brittle. The framework is not end-to-end, which may limit its generalization ability in real-world scenarios with imperfect perception.\n\n2/ Impact of Motion Discretization: The method discretizes continuous motion axis directions into a set of text-based categories to be compatible with the MLLM's output format. While this is a practical solution, it is a simplification that inherently loses precision. The paper does not discuss the potential impact of this information loss or analyze the error introduced by this categorization scheme.", "was_revised": true, "final_justification": "See my comments below.", "rebuttal": "Thank you for your valuable comments. We are pleased that the reviewer recognizes our method as novel, effective, and thoroughly evaluated. Please see our responses below.\n\n**W1: Reliance on upstream segmentation.**\n\nThank you for the valuable insights. We agree that the quality of upstream segmentation impacts overall performance. Nonetheless, our two‑stage design already achieves the strongest results. Specifically, AffordBot attains 20.0 AP50 on SceneFun3D, which significantly outperforms the previous SOTA Fun3DU (12.6 AP50). While an end-to-end approach would be elegant theoretically, such alternatives require large-scale training. Given the current dataset scale (230 scenes and 2K instructions), our solution remains the optimal choice. The data scaling and the exploration of end-to-end architectures should be left as future work. We will incorporate these analyses in our revision.\n\n**W2: Impact of motion discretization.**\n\nWe thank the reviewer for the thoughtful feedback. In practice, the vast majority of motion-related objects in SceneFun3D—such as drawers, doors, sockets, and switches—are intentionally aligned along the primary axes, reflecting their real-world physical constraints. Therefore, discretizing the motion axis into a small number of semantic categories is not merely a simplification, but a design choice grounded in the spatial regularity of common indoor affordances. Moreover, the predicted motion axis in our framework serves as a high-level reasoning output in the execution system—analogous to a planning signal that instructs \"pull outwards horizontally.\" This allows subsequent low-level modules or agents to refine and execute the motion with more precision. In this sense, the discretization operates as an abstraction appropriate for reasoning, not for direct actuation. To quantify potential information loss, we analyzed the angular deviation between ground-truth motion axes and their nearest canonical direction. We found that over 58% of all instances fall within 10° of the nearest primary axis, and over 96% fall within 30°. This reflects that most motion directions in our dataset naturally concentrate around the canonical axes we designed.\n\n**Q1: In-depth analysis of bottlenecks.**\n\nThank you for your inspiring suggestion! We agree that the overall affordance grounding performance is still relatively low. This reflects the large difficulty of the SceneFun3D dataset and our task. It elevates 3D affordance reasoning to the scale of entire scenes, requiring a system to parse millions of points in indoor environments guided by natural-language instructions to precisely localize small affordance elements such as \"knob\" or \"handle\", which obviously showcase the challenging nature of the benchmark. Under such conditions, our approach still improves the previous state-of-the-art AP50 score from 12.6 to 20.0, nearly doubling it, demonstrating that our AffordBot delivers meaningful progress even in such a demanding setting.\nTo dissect the primary bottlenecks, we progressively replaced Mask3D’s predicted masks with ground-truth (GT) masks and further provided an ideal front-view perspective. The results reveal that correcting segmentation alone boosts AP50 by nearly 20 percentage points, indicating that instance segmentation error is currently the primary limiting factor. With perfect segmentation, further providing an optimal viewpoint yields an additional improvement of 2.9 pp, suggesting that active perception still has room for optimization but is not the major bottleneck. Notably, when given both perfect masks and viewpoints, AP25 only reaches 47.4, exposing an additional challenge in cross-modal geometric reasoning—MLLMs may be confused about target localization in some multi-instance or complex spatial relation scenarios. We will add these in-depth analyses in the revised paper, hoping that our method and its thorough dissection will further inspire future work in the community.\n\n|         Setting         | mAP  | AP50 | AP25 |\n|-------------------------|------|------|------|\n| Mask3D proposals        | 15.5 | 20.0 | 23.3 |\n| GT proposals            | 35.7 | 39.4 | 45.4 |\n| GT proposals & GT views | 38.3 | 42.3 | 47.4 |\n\n**Q2: In-depth analysis of Active View Selection.**\n\nThank you again for recognising the contribution of our Active View Selection (AVS) mechanism. AVS leverages MLLM to select the most informative viewpoint based on the observation, which enables the model to perform better. As reported in Table 3, replacing AVS with a simple fixed viewpoint obtained by uniformly sampling video frames lowers AP50 by 8.6 pp, underscoring AVS’s core contribution. Motivated by your insights, we analyzed the failure cases of AVS and found two main patterns: (i) the target part is occluded by objects further ahead, making it absent or blurry in the image; (ii) several instances of the same category and appearance coexist in the scene as distractors.\nFurthermore, to quantify AVS accuracy, we adopt a strict criterion: a selection is considered successful only if every ground‑truth target element is visible and unobstructed. Under this, AVS achieves a success rate of 56%, which reflects that although our current strategy design has significantly improved compared to previous methods, there is still room for improvement. We will include this analysis in the revised version and, for future work, plan to explore more robust strategies such as multi‑candidate view generation or post-refinement.\n\n**Q3: Additional experiments with different MLLMs.**\n\nThe success of our AffordBot stems from its integrated pipeline, which comprises multimodal scene representation, active view selection, and chain‑of‑thought prompting. It can be easily integrated into different MLLMs and fully utilize their strong reasoning capabilities. To verify this, we replaced Qwen‑2.5‑VL‑72B with several representative alternatives: LLaVA‑v1.6‑34B and the commercial GPT series. These results demonstrate that AffordBot is not tied to a particular model and maintains strong performance across MLLMs. Moreover, the results suggest that the better the MLLM’s vision-language capability, the higher the overall grounding accuracy. We will supplement these findings in the revised paper.\n\n|   MLLM    | mAP  | AP50 | AP25 |\n|----------------|------|------|------|\n| LLaVA-v1.6-34B | 10.6 | 14.2 | 16.9 |\n| Qwen2.5‑VL‑72B | 15.5 | 20.0 | 23.3 |\n|     GPT-4o     | 16.5 | 22.1 | 28.9 |\n|     GPT-o1     | 24.8 | 30.3 | 33.4 |\n\n**Q4: MLLM capabilities and extended settings.**\n\nWe fully agree with the reviewer that one of the most exciting promises of an MLLM‑driven pipeline is its capacity for truly long‑horizon reasoning. In our paper, however, we focus on the high-level reasoning as aligned with the existing benchmark. Extending AffordBot to a setting akin to PreAfford, where the agent plans and executes a chain of affordance‑conditioned actions that progressively reshape the environment, would be an appealing next step. Implementing this requires not only multi‑turn CoT prompting but also a closed‑loop simulation that updates the scene after each predicted action, so that subsequent reasoning can take the new state into account. Because such an end‑to‑end simulator integration and experimental campaign lie beyond the time available for the rebuttal period, we plan to explore this in future work."}, {"review_id": "Wr0fX9iStk", "reviewer": "Reviewer_HoJn", "summary": "The paper proposes AffordBot, a framework for fine-grained 3D embodied reasoning. Given 3D point clouds and task instructions, it predicts affordance masks, motion types, and directions for target elements. The approach projects 3D data into surround-view images, uses MLLMs with chain-of-thought reasoning for active view selection, affordance grounding and motion estimation, and demonstrates state-of-the-art results on SceneFun3D.", "questions": "1. The current framework relies on 2D MLLMs after projecting 3D point clouds into 2D views. Have the authors considered leveraging recent 3D-aware MLLMs or directly incorporating point cloud features into the reasoning pipeline? Could the method benefit from native 3D spatial reasoning rather than relying on 2D projections?\n2. The approach appears highly dependent on the initial segmentation quality from Mask3D. Could the authors provide more analysis or ablations on how segmentation noise affects downstream reasoning performance? How robust is AffordBot when upstream segmentation contains errors or uncertainties?\n3. The evaluation is limited to SceneFun3D. Could the authors test on additional datasets or real-world scans to verify the generalization ability of the framework? How would the method handle out-of-distribution instructions or affordances not seen in SceneFun3D?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses the problem of fine-grained embodied reasoning in 3D scenes, combining affordance grounding and motion estimation in a unified task.\n2. The paper demonstrates superior performance over existing baselines on the SceneFun3D dataset, with comprehensive quantitative, ablation, and qualitative analyses.\n3. The paper is well-organized and easy to follow.\n\nWeaknesses:\n1. The proposed approach avoids the core challenges of 3D reasoning by projecting 3D point clouds into 2D surround views and relying on 2D MLLMs for reasoning. While effective, this design overlooks the need for deeper spatial understanding and physical reasoning in 3D space, which are central to embodied reasoning. Moreover, recent advances have explored directly incorporating point cloud inputs into MLLMs or 3D-aware vision-language models, but this direction is not sufficiently explored.\n2. As indicated in Table 4 and 5, the method’s performance is heavily dependent on the initial Mask3D segmentation quality. The approach may struggle with small or occluded objects where segmentation is less reliable, leading to potential failures in downstream reasoning steps.\n3. The evaluation focuses solely on SceneFun3D, which is a single-domain dataset. The generalization ability to more diverse real-world environments, or more complex instructions remains untested. Additional benchmarks would validate the method’s generalization applicability.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal and clarifications. Most of my concerns have been addressed. After considering the response and the opinions of the other reviewers, I have decided to raise my original score to 4.", "rebuttal": "Thank you for your valuable comments. We appreciate your positive assessment of our method’s performance and analysis. We hope the responses below address your concerns.\n\n**W1.1: Clarification on 3D-aware reasoning.**\n\nWe sincerely appreciate your recognition of our method's effectiveness. However, we respectfully disagree with the concern that our design overlooks deeper 3D spatial or physical reasoning. On the contrary, it preserves 3D spatial information by explicitly associating each element in the 2D rendered views with its precise 3D coordinates. These associations are incorporated into the MLLM input as reasoning-aware cues—each image is overlaid with bounding boxes and paired with textual prompts that enumerate the corresponding 3D attributes. As a result, the model operates not on raw 2D pixels alone, but on spatially grounded visual–textual inputs, enabling effective 3D-aware reasoning throughout the chain-of-thought process.\n\n**W1.2&Q1: Comparison to native 3D MLLMs.** \n\nThanks for your valuable comments. The cutting‑edge 2D MLLMs are currently an order of magnitude larger and more capable than 3D‑aware models in both vision-language understanding and long-horizon reasoning, as evidenced by our superior performance.  While leveraging 3D-aware MLLMs or directly incorporating point cloud features is promising, such directions currently face several limitations. First, our task requires fine-grained reasoning on small affordances (e.g., knobs, handles) embedded in cluttered 3D scenes, which is particularly challenging. However, most existing 3D MLLMs operate at a coarse object-level granularity and struggle to support our fine-grained task. Moreover, incorporating point cloud features into MLLMs would require architectural modifications and retraining. These scene-level architectures typically require large-scale training data, for instance, 3D-LLM [1] utilizes 300K 3D-text pairs, whereas our current dataset SceneFun3D (230 scenes and 4K instructions) remains quite limited. \n\n\n**W2&Q2: Dependence on segmentation quality.**\n\nWe appreciate the reviewer’s feedback regarding the dependence on Mask3D segmentation. As acknowledged in Section 4.2 (Lines 290–292), our framework does rely on upstream segmentation to propose candidate regions. However, the subsequent multimodal representation and chain-of-thought reasoning provide a level of robustness that can mitigate some of these errors. This is evident in Table 4, where categories such as \"rotate\" and \"key_press\"—despite lower segmentation quality—still yield strong downstream performance. Moreover, compared to the prior state-of-the-art Fun3DU, which also adopts a multi-stage architecture, our method achieves substantially higher AP50 performance, suggesting stronger tolerance to segmentation noise. \nWe also agree that small or occluded affordance elements pose an even greater challenge for segmentation. We plan to explore hybrid segmentation strategies that combine 2D instance cues with 3D geometry, or leverage strong 2D segmentation priors as cross-modal constraints to better localize hard-to-see instances. \nTo better quantify the impact of segmentation quality, we conducted two analyses:\n\n(i) Performance upper bound: We replaced the predicted instance masks with ground-truth segmentation. As shown below, AP50 improves from 20.0 to 39.4, indicating significant headroom and confirming segmentation as a key bottleneck.\n\n|   Upstream Method   | mAP  | AP50 | AP25 |\n|---------------------|------|------|------|\n| Predicted proposals | 15.5 | 20.0 | 23.3 |\n|   GT proposals      | 35.7 | 39.4 | 45.4 |\n\n(ii) Robustness under noise:\nWe simulate segmentation errors by randomly deleting a percentage of predicted masks. Interestingly, the downstream performance remains stable up to 20% noise. The performance only begins to decline more noticeably when deletion reaches 30%:\n\n| Noise Ratio | mAP  | AP50 | AP25 |\n|-------------|------|------|------|\n|     0%      | 15.5 | 20.0 | 23.3 |\n|     10%     | 15.4 | 20.0 | 23.8 |\n|     20%     | 16.4 | 19.8 | 22.6 |\n|     30%     | 13.3 | 17.5 | 21.9 |\n|     40%     | 11.7 | 15.1 | 18.0 |\n\nThese results reflect the strength of the reasoning module, which is capable of partially recovering from upstream failures. This finding outlines a promising direction for future work: employ more robust segmentation strategies or leverage the multimodal reasoning capabilities of subsequent MLLMs to iteratively refine segmentation outputs. We will incorporate these findings into the revised version.\n\n\n**W3&Q3: Additional evaluation of generalization ability.**\n\nWe appreciate the reviewer’s suggestion regarding the investigation of generalization capability. However, to the best of our knowledge, SceneFun3D is currently the only available scene-level dataset for our task. Preparing new 3D datasets with compatible labels requires extensive manual annotation and is unfortunately infeasible within the rebuttal period. To address the generalization concern, we conducted a test using GPT-rephrased SceneFun3D instructions with more diverse syntax and complex phrasing to evaluate the model’s robustness to out-of-distribution language inputs. Our method maintains comparable performance across original and rephrased instructions, with only a minor drop. In contrast, the baseline Fun3DU suffers a significant performance drop. These results suggest that AffordBot generalizes well to more diverse and complex instruction formulations, validating its robustness to out-of-domain language.\n\n|       Method       | mAP  | AP50 | AP25 |\n| ------------------ | ---- | ---- | ---- |\n| Fun3DU (original)  | 6.1  | 12.6 | 23.1 |\n| Fun3DU (rephrased) | 3.7  | 8.1  | 15.1 |\n| Ours (original)    | 15.5 | 20.0 | 23.3 |\n| Ours (rephrased)   | 13.7 | 18.5 | 22.6 |\n\n**References:**\n\n[1] Hong Y, Zhen H, Chen P, et al. 3d-llm: Injecting the 3d world into large language models[J]. Advances in Neural Information Processing Systems, 2023, 36: 20482-20494."}]

Author rebuttal:

Meta-review: This paper introduces a new task, Fine-grained 3D Embodied Reasoning, which requires predicting a structured triplet (spatial location, motion type, motion axis) for affordance elements in a 3D scene based on a task instruction. Despite the noted limitations, this paper makes a clear contribution by defining a new and important task and providing a strong, well-validated baseline solution. The concerns about segmentation dependency and 2D projection are important for the discussion and future work, but do not diminish the paper's novelty, technical soundness, or significance for the NeurIPS community. All four reviewers hold positive ratings (Accept, Accept, Weak Accept, Weak Accept).
Decision: Accept (poster)