3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds
Authors: Hengshuo Chu
Xiang Deng
Qi Lv
Xiaoyang Chen
Yinchuan Li
Jianye HAO
Liqiang Nie
Venue: ICLR 2025 Poster (2025)
Keywords: 3D.+Large Language Model.+Robot.+Affordance.+Perception
Primary area: applications to robotics, autonomy, planning

Abstract: 3D Affordance detection is a challenging problem with broad applications on various robotic tasks. 
Existing methods typically formulate the detection paradigm as a label-based semantic segmentation task.
This paradigm relies on predefined labels and lacks the ability to comprehend complex natural language, resulting in limited generalization in open-world scene.
To address these limitations, we reformulate the traditional affordance detection paradigm into \textit{Instruction Reasoning Affordance Segmentation} (IRAS) task. 
This task is designed to output a affordance mask region given a query reasoning text, which avoids fixed categories of input labels.
We accordingly propose the \textit{3D-AffordanceLLM} (3D-ADLLM), a framework designed for reasoning affordance detection in 3D open-scene.
Specifically, 3D-ADLLM introduces large language models (LLMs) to 3D affordance perception with a custom-designed decoder for generating affordance masks, thus achieving open-world reasoning affordance detection.
In addition, given the scarcity of 3D affordance datasets for training large models, we seek to extract knowledge from general segmentation data and transfer it to affordance detection.
Thus, we propose a multi-stage training strategy that begins with a novel pre-training task, i.e., \textit{Referring Object Part Segmentation}~(ROPS).
This stage is designed to equip the model with general recognition and segmentation capabilities at the object-part level.
Then followed by fine-tuning with the IRAS task, 3D-ADLLM obtains the reasoning ability for affordance detection. 
In summary, 3D-ADLLM leverages the rich world knowledge and human-object interaction reasoning ability of LLMs, achieving approximately an 8\% improvement in mIoU on open-vocabulary affordance detection tasks.

Published as a conference paper at ICLR 2025
3D-AFFORDANCELLMM: HARNESSING LARGE LAN- GUAGE MODELS FOR OOON-OOOOAAAEEEO VOCABULARY AFFOR DANCE DETECTION IN 3D WORLDS
Hengshuo Chu', Xiang Dengi', Qi
,iaii,QiiVV Lv*, Xiaoyang Chen', Chen',Yinchuan Yinchuan Li², Jianye Hao", Liqiang Niet*
Harbin Institute of Technology (Shenzhen), HHuawei Noah's Ark Lab
ABSTRACT
3D Affordance detection is a challenging problem with broad applications on var- ious robotic tasks. Existing methods typically formulate the detection paradigm label-based semantic segmentation task. This paradigm relies on predefined labels and lacks the ability to comprehend complex natural language, resulting in limited generalization in open-world scene. To address these limitations, we reformulate the traditional affordance detection paradigm into Instruction Rea- soning Affordance Segmentation (IRAS) task. This task is designed to output a affordance mask region given query reasoning text, which avoids fixed cat- egories of input labels. We accordingly propose the 3D-AAfordlanee...M (3D- ADLLM), a framework designed for reasoning affordance detection in 3D open- scene. Specifically, 3D-ADLLM introduces large language models (LLMs) to 3D affordance perception with custom-designes decoder for generating affordance masks, thus achieving open-world reasoning affordance detection. In addition, given the scarcity of 3D affordance datasets for training large models, we seek to extract knowledge from general segmentation data and transfer affordance de- tection. Thus, we propose multi-stage training strategy that begins with novel pre-training task, i.e., Referring Object Part Segmentation (ROPS). This stage is designed to equip the model with general recognition and segmentation capabil- ities at the object-part level. Then followed by fine-tuning with the IRAS task, 3D-ADLLM obtains the reasoning ability for affordance detection. In summary, 3D-ADLLM leverages the rich world knowledge and human-object interaction reasoning ability of LLMs, achieving approximately an 8% improvement in mloU on open-vocabulary affordance detection tasks.
INTRODUCTION
Robots increasingly integrating into various aspects of our daily life (Matheson et al. 2019). As we progress toward developing the next gemeration of more advanced robotic agents, it is essential enable robots comprehend natural language instructions within context and to perceive task- relevant information in their surroundings. This skill is particularly vital for seamless interactions in unstructured environments, such as homes, where adaptability to diverse situations crucial. Specifically, the robots need to not only identifyy the objects in the environments but also locate the specific regions of each object that are suitable for interaction: affordance.
The concept of affordance was introduced by ecological psychologist James Gibson (Gibson [966]
and has since played a significant role in various robotic applications, including object recogni-
tion (Hong et al. 2023a Hou et al., 2021). action anticipation (Roy & Fernando 2021 agent activity recognition (Chen et al. 2023), and object functionality understanding (Li al. 2023)
Affordance describes potential interactions between robots and their environment, such using knife's blade for cutting tasks. Detecting affordances is challenging due to object diversity and com- plexity (Min et al. 2016). Traditionally, 2D images and CNNs are used (Nguyen et al. 2016 Do et al. 2018 Cacheco-Ortega & Mayol-Cuervar. (222)KKizeessk et al. 2012). but 2D information
1 CCorresponding authors

---

Published as a conference paper at ICLR 2025
lacks the depth necessary for precise manipulation, necessitating 3D transformations (Deng al. 2021).
With advanced depth cameras, 3D point clouds have become widely used modality in robotic ap- plications (Liu et al. 2019). Unlike conventional images, 3D point clouds offer robots direct and detailed 3D information about surrounding objects and environments. Hence, the 3D affordance detection has been deemed as a critical step in bridging perception and manipulation in the physical world for an embodied agent, thus has shown substantial impact on practical applications such as robotic manipulation (Geng el al., 2023; Moldovan el al. 2012). While current approaches are lim- ited by fixed label sets (Deng et al., 2021 Mo et al., 2022), reducing flexibility and generalization in dynamic settings. To overcome the fixed label set problem in affordance detection, Nguyen et al. (Nguyen et al., 2023) have incorporated text encoder to enable models to handle certain lev- els of open-vocabulary detection, but these algorithms still rely on a classification based training paradigm. As a result, they lack the ability for rapid and continuous learning when presented with new affordance label data. Furthermore, current affordance detection methods also heavily rely on the predefined labels and lack the ability to understand and reason over long contextual text. Addi- tionally, the scarcity of 3D affordance datasets (Deng al. 2021 Nguyen et al. 2023) constrains the effective training of large-scale scale models.
Towards these issues, redefine the 3D affordance detection Instruction Reasoning Affor- dance Segmentation (IRAS) task and accordingly propose 3DD-AfforlanceLL.MM (3D-ADLLM), The IRAS task is designed to output an affordance mask region in response to complex, reasoning- query text, overcoming the limitations of fixed affordance labels and the difficulty of understand- ing complex instructions. Our 3D-ADLLM framework introduces large language models (LLMs) to 3D affordance perception with a specifically designed decoder for generating affordance masks, thus achieving open-world reasoning affordance detection. Specifically, we introduce an additional token, <AFF>, into the original LLM vocabulary. When the <AFF> token is generated, its hid- den embedding further decoded into the corresponding segmentation mask. By representing the segmentation mask an embedding, 3D-ADLLMM not only gains segmentation capability but also benefits from end-to-end training. However, due to the scarcity of 3D affordance datasets for training large models, propose multi-stage training strategy to extract knowledge from general segmen- tation data and transfer to affordance detection. This process involves pre-traue pre-training on PartNet (Mo) et al. 2019) with Referring Object Part Segmentation (ROPS) tasks to acquire the object-part level general recognition and segmentation knowledge. Subsequently, we fine-tune the model with the IRAS task to achieve context-aware reasoning ability and robust performance in open-set zero-shot affordance detection.
Our main contributions are summarized as follows:
Different from the existing affordance detection methods that rely on fixed sets of labels, we address this limitation by introducing a new detection paradigm based on the Instruc- tion Reasoning Affordance Segmentation (IRAS) task. By reforming the label-based se- mantic segmentation task in the traditional affordance detection paradigm into natural language-driven reasoning affordance segmentation task, our model enables more flexible and context-aware reasoning, facilitating effective zero-shot learning capabilities.
To address the IRAS tasks driven by semantic complex natural language, we consequently propose the 3D AffordanceL.MM (3D-ADLLM) model, combining a large language model (LLM) with carefully designed Affordance Decoder. Our 3D-ADLLM framework can understand semantically-rich, long-context instructions and leverages the LLM's world knowledge for superior ooca-vocyy affordance detection.
Due to the scarcity of 3D affordance datasets for training large models, we propose a multi- stage training strategy to transfer general segmentation knowledge into affordance detec- tion. First, the model is equipped with general recognition and segmentation knowledge through novel pretraining task, i.e., the Referring Object Part Segmentation (ROPS). Sub- sequently, the model is fine-tuned with the IRAS task to handle context-aware reasoning and affordance region prediction.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Affordance Detection.
Originating from the 2D domain, initial work in affordance detection pri- marily focused on identifying objects with affordances eDo al. (018)). Building on this foundation, later studies (Lu et al. [2022] introduced linguistic descriptions to improve detection, but they contin- ued to emphasize object-level affordances, lacking fine-grained analysis. Addressing this problem,
subsequent research (Chen et al. 2023 Li ct al. 2023; Luo et al.) 2022; Nagarajan et al. [019 Mi
et al. 2020) has focused on detecting specific affordance parts, establishing new benchmark for precision in the field. With the advancement of embodied AI, the scope of affordance learning has expanded into 3D domain. 3D AffordanceNet (Deng et al. (021)) introduces the first benchmark dataset for learning affordance from object point clouds. IAGNet (Yang et al. 2023) propose a setting for learning 3D affordance parts guided by image queries. Recently, some work (Nguyen) itt al. 2023) also explores the open-vocabulay affordance detection in point clouds. However, these methods primarily focus on linking object geometric features with fixed affordance labels, over- looking the semantic aspect. This limitation makes it challenging to understand natural language instructions and hampers the ability to generalize affordance detection to unseen scenarios. In con- trast, the proposed 3D-ADLLM overcomes the limitations of fixed label sets and enhance the ability to comprehend semantic complex description. Specifically, we shift the detection paradigm from label-based semantic segmentation into Instruction Reasoning Affordance Segmentation (IRAS).
3D Large Multi-Modal Models. 3D object-level LMMs (Yu et al., 2022, Xue et al. 2023 Zhou (et al. 2023) have successfully bridged the gap between 3D vision and text by leveraging large-scale scale 3D object datasets like (Deitke et al.] [202]] Vishwanath et al. 2009). ShapeLLM (Qi et al. 2024) further advances the embodied interaction and referring expression grounding through its novel and powerful point encoder. However, despite these advances, such models still face challenges in interpreting complex spatial relationships within 3D scenes. For scene-level LMMs, models like Chat-3D (Wang et al. (023) and LL3DA (Chen et al. 2024) enable interaction with scene objects using pre-selection mechanisms. Building on this foundation, Chat-3D v2 (Huang et al. 2023) enhances referencing and grounding accuracy by incorporating object identifiers. while 3D- LLM (Hong et al.] 2023b) improves scene comprehension by integrating positional embeddings and location tokens. Unlike previous works that primarily focus on 3D grounding and understanding, our method introduces a specialized token, <AFF>, which enables L.LMs to directly detect affordances and generate affordance masks within 3D open- scene.
METHOD
3.1 PARADIGM REFORMULATION
Affordance detection aims to identify specific regions of objects that are suitable for interaction. It has been deemed as critical step in bridging perception and manipulation in the physical world for embodied agents. As illustrated in Fig. (a). the traditional paradigm uses a shared point back- bone (Qi et al.] 2017 Zhao et al.] [20TT] Wang al,, 2019) to extract point-wise features, and gen- erates masks with a predefined type semantic segmentation head. Alternatively, they leverage a text encoder like CLIP (Radford et al. 202T) to associate point-wise features with text embeddings of affordance labels using cosine simi similarity, achieving limited open-vocabulary detection on the phrase level. This paradigm relies on predefined labels and has a limited ability to understand complex natural language, which restricts its generalization in 3D open-world scene.
To address these limitations, we introduce new paradigm formulated as an Instruction Reasoning Affordance Segmentation (IRAS) task as depicted in Fig. (b). This paradigm is designed to estab- lish robust connection between language context and object affordance, avoiding the overreliance on auxiliary affordance label prediction. This approach facilitates a significant improvement in our ability to understand and interact with the physical world.
IRAS Definition. Given a query reasoning instruction Qa and an object point cloud P. E B with N points, the goal of IRAS is to predict binary mask of Ma E IRN that delineates the functional regions pertinent to the query, affordance regions:
Fatdel (Qay Po) + Ma
3

---

Published as a conference paper at ICLR 2025
Observation
Label-based
(b) IRAS
Affordance region
I'm feeling thireyy could you pass me something can use to drink water?
Predetined Label
PointNet Head
pour
Handle
3D AffordanceLLM
grasp
00000000000
Affordance Decoder
contain
The comparison of the affordance detection paradigm based on our IRAS or traditional label-based segmentation tasks. (a) shows that label-based paradigm can only detect the fixed set of affordance regions through the predefined label and seg-head; (b) demonstrates the IRAS based paradigm forges a link between semantic complex instruction and object affordance, enabling open- world reasoning affordance detection.
Figure 1:
3.2 AD-AITORRAANCLLMM
To the traditional methods that rely on fixed label sets and are limited to short-text detection, IRAS demands robust language comprehension and reasoning to associate the potential affordance in in- put query with 3D objects areas. Thus, we incorporate large language models (LLMs) into 3D affordance perception. L.Ms, trained trillions of tokens, excel in understanding and reasoning about instructions and possess extensive world knowledge. For instance, when asked where to in- teract with mug to grasp it, LLMs suggests using the handle for firm grip to avoid spilling. This demonstrates LLMs' world knowledge and the capability in understanding human-object interac- tions. To harness this capability for 3D affordance perception, we introduce the 3D AffordanceLLMM Model, aiming to improve affordance detection in previously unseen contexts.
Our framework, 3D AffordanceLLM, as illustrated Fig. primarily consists of two main com- ponents: (1) a point cloud multimodal model which is trained to accept point cloud and text inputs and generate response, including a special token, <AFF> >; (2) an Affordance Decoder (AFD), which extracts hidden layer features from these <AFF> tokens and combines them with segmentation point features to generate affordance masks.
Instruction:
Text Embedding
Please locate the on this mug that are ideal for grasping it?
Crass Attertionn
har!
LLM
Ctwry
MLP
Point Encoder
Projector
LoRA
Sel Altertoo
Figure 2: The Pipeline 3D-ADLLM. Given the input point cloud and query reasoning instruction, the point cloud multimodal model is trained with lora to predict special token <AFF>. Finally, the special token and dense point features from frn is fed into our designed affordance decoder to generate the final affordance mask.
3.2. MODEL ARCHITECTURE
As is shown in Fig. point cloud encoder a projector fproj= a point backbone frB, an affordance decoder fAPD and pre-trained large language model (L.LM) backbone film-
3D AffordanceLLM consists of the following modules: pre-trained
4

---

Published as a conference paper at ICLR 2025
Point Encoder. The point cloud encoder fpe, takes point cloud P Peloud R input, where represents the number of points and denotes the feature dimension of each point. The output of the encoder is sequence of point features X (2), ; Im) E R xe where m is the number of point features and is the feature dimension. Similarly, the point backbone fPB, also processes input point cloud P Pelndd Ra extracting the dense point cloud features X' = (2 22, -........ ......... snn))
Raxe', specifically tailored for segmentation tasks. These features are subsequently fed into the Affordance Decoder.
LLM Projector. The projector Sproj is ML.P layer that maps the point features X point tokens
Y = (i), 001.3/2..... 3/...... Fmm E Rmx"", where c" is the dimension of the point tokens, matching the dimension of the text tokens.
The LLM backbone film is a decoder-only Transformer model (Vaswani et al. 2017), which processes a sequence of tokens comprising text and point tokens. This mixed token sequence is denoted as Z (21, :2, 24) R where is the total number of tokens. Leveraging a self-attention mechanism, the LLM backbone captures contextual relation- ships between different token types, enabling it to generate responses based on both text and point cloud inputs. Formally, the output of the LLM backbone fulm is sequence of predicted tokens 2 (21, .q.....
Large Language Model.
. 2x) E Ieke"", The prediction of the i-th token, 2, is conditioned on all previous tokens, Zei = (Z1,...; 11,1), which can be expressed mathematically as:
2 = fum (z))).
Each 2 is passed through a final linear layer followed by softmax operation, which maps the hidden states to probability distribution over the vocabulary. This layer is denoted fvocab R - IR where V is the size of the vocabulary. The final prediction & for the i-th token is the word in the vocabulary with the highest probability, expressed
2 arg max wEvocab
[vocat(22)) (2:)
Affordance Decoder. Building on the success of learnable query-based based methods in object segmen- tation, we introduce an Affordance Decoder Module (AFD) that leverages a set of learnalle output queries conditioned on input questions, termed affordance queries Tp to decode segmentation masks. A two-layer decoder updates both the point features and the question features via cross-attention. Then, the updated query tokens and point features are used to dynamically predict affordance masks.
3.2.2 EMBEDDING AS AFFORDANCE
Unlike conventional tasks such grounding, question answering, etc., within the realm of 3D large multi-modal models (LMMs), the IRAS task is depicted to generate a affordance segmentation mask directly given reasoning query. Most current 3D LLM (such 3D-LLM (Hong al. 2023a), ShapeLLM (Qi et al. 2024) support 3D scenes or objects and text as input, but they can only output text or bbox and cannot directly output fine- -reingr segmentation masks. Inspired by the LISA model (Lai et al. 2024), which directly outputs the segmentation mask in the 2D domain, we adopt similar idea 3D affordance detection. To achieve that, we propose the
aaaaddiin-----ffrdaaeec paradigm to inject new affordance segmentation capabilities into the 3D LMM. The pipeline of our method is illustrated in Fig. 2 new token, <AFF>, which signals request for an affordance output. Given complex reasoning instruction query Qui and point cloud input P Peloud, we feed them into the multimodal point clouds LLM Fan FaD-ADLLMS ADLLM, which outputs a text response Vixe: "Sure, it is <AFF>." This process can be formulated as:
Specifically, we expand the original LLM vocabulary by adding a
Text PaD-ADLLM (PALLMMPaaa Peloud, Qut)-
When the LLM intends to generate a binary affordance mask, the output Pixt would include a <AFF> token. We then extract the LLM last-layer embedding haя corresponding to the <AFF> token and apply an MLP projection layer Proj to obtain haft Simultancously, the point cloud backbone frB extracts the dense point clouds features f from the points input P cloud Finally, hat and f are fed to the decoder SAED to produce the final affordance mask M. The process can be formulated as
hal Proj(hair)
f = SPB (clodd) Matt = fAPD f)-
5

---

Published as a conference paper at ICLR 2025
(a) Pre-training Stage: Extracting General Segmentation Knowledge
Mask Decoder
Point Cloud
Point Backbone
Mask Regions
Text
Text Encoder
Projector
back soft
solissiitii seat soft surface
surface
frame Knowledge Transfer
(b) Specific Task Finetuning: Transferring Knowledge to Affordance Detection
Point Backbone
Point Cloud
Point Encoder
Affordance
Regions
ALLM Projector
Text
Lora
Affordance Decoder
Please ... Where is the Can you take seat. reposition ...?
support
Figure 3: Multi-stage training strategy. Illustration of transferring general segmentation knowledge to affordance detection. (a) depicts the process of extracting general segmentation knowledge, while (b) illustrates the framework for transferring this knowledge to affordance detection
3.3 MULTI-STAGE TRAINING
Existing 3D affordance datasets, such as 3D AffordanceNet datasets, OpenAD datasets in (Deng al., 2021] Nguyen et al. 2023), constrained in availability and dataset sizes. Thus, given the scarcity of 3D affordance datasets for training large models, we devise a multi-stage training strategy which extracts knowledge from general segmentation data and transfers dance detection. In addition, due to the varying scales of target affordance regions, we propose sample unbalanced loss factor to enhance the model's learning effectiveness and adaptability across different region scales.
IRAS affor-
3.3. EXTRACTING GENERAL SEGMENTATION KNOWLEDGE
Considering the limited amounts of affordance datasets for training large models, this stage aims to leverage general datasets equip the model with general recognition and segmentation capabilities at the object-part level. Thus, we introduce Referring Object Part Segmentation (ROPS) task to acquire the general knowledge.
ROPS Definition. Given a referring expression that denotes the name of the object's components Q and object point cloud P. Rox3 consisting of N points, the objective of ROPS is predict a binary mask for My E RN that corresponds to the query:
Foadet Pe) = Mp
In the pre-training phase, we employ the framework in Fig. 3 (a) to train the ROPS task on the PartNet dataset (Mo et al. 2019). As depicted in Fig. (a). the object point cloud is processed by trainable backbone to extract point features aPat.n The object part descriptions are encoded using a frozen text encoder to generate text features fq... which are then mapped via offline MLP layer to produce f' Ququa Finally, f' Q'q. and fPanus are passed into the Mask Decoder to generate the final part mask M Mpart, formulated
M Mpant MaskDecoder(f Qpen F
Training Objectives. Unlike (Mo et al., 2019), which uses multi-class head for prediction, stragety seeks to extract the knowledge relationship between referring object part description and the corresponding mask regions. Thus, solely employ Dice Loss and Binary CrossEntropy (BCE) loss to guide the segmentation mask prediction.
CLCBCE + A2CDICE
3.3.2 TRANSFERRING KNOWLEDGE TO AFFORDANCE DETECTION
Building upon the extensive segmentation knowledge acquired from the ROPS task, we transfer this knowledge to affordance detection by IRAS finetuning to enhance the model's generalization. We

---

Published as a conference paper at ICLR 2025
also propose sample unbalanced loss factor to address the learning strategies for affordance regions of different scales. Specifically, during IRAS fine-tuning: we use the pretrained checkpoint Wip. and Wix fMD to initialize the modules fPB and fAFD in our framework 3D-ADLLM as shown in Fig. We then use the Lora method to fine-tune a pre-trained L.M for affordance segmentation.
Training Objectives. The model is trained end-to-end using text generation loss L and segmen- tation mask loss maskk- Apecifically,.car encourage the LLMs generate the response including the <AFF> token and forces the features to map the same <AFF> placeholder. Then, Lmask diversi- fies the <AFF> features to contain the information for affordance prediction and guides affordance mask generation. The overall objective the weighted sum of these losses, determined by Aext and maakk
Aext Lixt + Amask Linask-
Specifically, Lixt is the auto-regressive cross-entropy loss for text generation, and mask is the mask loss for high-quality segmentation. To compute Lmask, we use a combination of per-pixel BCE loss and DICE Loss, with weights Λbee and Adice. Given the ground-truth targets Yext and M obtained from dataset, these losses are formulated as:
Lixt = CCCCEEER ttxt, Ytxt),
Luaak Abbe BCE(M, M) + Adice DICE,I M).
Sample Unbalanced Loss Factor. Due to the varying scales of target affordance regions, our model 3D-ADLLM naturally challenges model's adaptiveness at different scales. This variability will sults in an imbalance in the difficulty of learning samples between different affordance types during the training process. To mitigate the issue of sample imbalance across different affordance types during training, we apply weights to the mask losses for each class. The weighted loss is defined as: faaakk Di= with miskk The weight w; is calculated as:
max ....... Cma co)
1/4
Wi
where o is the number of ground truth points for class i, and co denotes background points.
4 EXPERIMENT
4.1 EXPERIMENT SETTING
Network Architecture. We use mhi-3.5-mini-instrutt (fim) (Abdin et al. 2024) as our base LLM.
For the point encoder (fpe). we adopt Point-BERT (Yu al. 2022), pre-trained with ULIP-2 (uue
et al. 2024) in the ModelNet dataset /Vishwanath et al. 2009). ). The projector layer (proj) between the point encoder fee and the LLM fimm is linear layer. Additionally, we utilize the Point Trans- former (Zhao et al. 2021 jas the backbone for our point segmentation model (/PR).
training data is made up of two types of task data: (1) Referring Object Part Segmentation Dataset: we build this dataset on PartNet (Mo et al.) 2019), which contains 573,585 part instances across 25,571 3D models and 24 object categories. For pre- training, we split it into single-part segmentation instances. (2) Instruction Reasoning Affordance Segmentation Dataset: we meticulously compile a question- poinnt affordance dataset with 42119 paired samples from 3D AffordanceNet dataset (Deng et al., (021), covering 23 classes and 36 affordance types. The Detailed data settings (full or partial view) and the visualized data analysis can be seen in Appendix Sect. A.3
Datasets. As is mentioned in Sec. 3.3
We compare our method with the following recent methods for zero-shot learn- ing in 3D point clouds: ZSLPC (Cheraghian al.] 2019). TZSLPC (Cheraghian et al., 2020 3DGenZ (Michele et al., 2021), OpenAD (Nguyen et al. 2023), IAGNet (Yang et al. 2023), LASO (Li et al. 2024) and ShapeLLM (Qi al. 2024). Detailed baseline model explanation for experiments can be found in Appendix Sect.[A.]
Baseline Models.
Evaluation metrics. We divide the IRAS dataset following the split of OpenAD and evaluate the close-set and open-set of IRAS. According Nguyen et al. (202)), we use three metrics to evaluate the results over all classes: mIoU (mean loU over all classes), Acc (overall accuracy over all
7

---

Published as a conference paper at ICLR 2025
Table I: Main results of 3D--DDLMM zero-short open vocabulary detection. The result is calcu- lated over all classes. The overall results all comparative methods, the best results are in bold. The method of ShapeLLMM is tested without finetuning.
Full-view Acc
Partial-view Acc
Method
mloU
mAce
mloUe
mAce
TZSLPC Cheraghian et al
2020
3.86
10.37
4.14
8.49
3DGenZ Michele el al.
6.46
18.33
6.03
15.86
ZSLPC Cheraghan
9.97
18.70
9.52
17.16
ShapeI.IMM LM (Qi et al.
2024
0.88
0.28
0.99
1.49
1.35
1.70
OpenAD-Point PointNet++ Nguyeen et al.
2023
13.53
3.97
16.40
11.29
2.41
13.88
OpenAD-DGCC (Nguyen et al. 2023
11.15
3.84
13,86
8,04
1.58
9.85
IAGNet Yang al. 2023
16.16
19.07
23.92
14.36
16.90
21.73
LASO (Li et al. 2024
22.41
15.90
30.22
20.06
8.80
26.84
Ours-Qwen Ours-Phi
24.43
23.90
35.45
26.25
29.5
41.57
30.43
29.36
47.78
27.25
27.87
39.04
points), and mAec (mean accuracy over all classes). However, unlike OpenAD, which includes the "none" category in the calculation of metrics, we only compute the 36 affordance types, excluding "none," it has little comparative significance. For comprehensive evaluation versus existing methods, we additionally assess each instance across the entire dataset. The specific evaluation metrics over all instances: mIoU' (mean IoU over all instance data), mAce (mean accuracy of points over all instance data), mPrec' (mean precision of points over all instance data), mRec' (mean recall of points over all instance data), mAP (mean average precision 50% intersection over union).
4.2 EXPERIMENT RESULTS
4.2.1 COMPARISON RESULTS
3D-ADLLM vs. Other Models. Table demonstrates that our 3D-ADLLM achieves superior per- formance across both full and partial view tasks, as well as on three evaluation metrics. Notably, 3D AffordanceLLM significantly outperforms the rumner-up model (LASO) in terms of mloU, with improvements of 8.02% and 7.19% on the full and partial view tasks, respectively. Compared to OpenAD, which predicts regions based on fixed set of affordance labels, method utilizes long- context understanding and reasoning for segmentation. In experiment results, our method surpasses OpenAD in terms of mloU 16.9% (full-view) and 15.96% (partial-view) separately across 18 affor- dance types. Additionally, for metrics over all instance, we surpass the sota model (LASO) 23.38% (full-view) and 24.93% (partial-view) in mAP The comparison results on close-set detection can be found in Appendix Sect. A.2
4.2.2 OUT-OF-DISTRIBUTTOO RESULTS
The test in out-of-distribution (ood) datasets is essential to assess the generalization capability of the model. Thus, constructed filtering out some combinations of affordance-objed that already existed in our IRAS dataset from the AffordPose dataset (Jian et al. 2023 Compared to existing datasets, this new dataset includes different types of affordances as well as unique affordance-object pairs, such as (twist, faucet), (lever, faucet), (press, dispenser), etc. As is shown in Table 3 our approach achieved the best zero-shot performance on this ood data.
test dataset consisting of approximately 559 entries by
4.3 ABLATION STUDY
Effeets of Different Components. To investigate the effectiveness of each component in 3D- ADLLM, conduct experiments with different variants of ADLLM.. In particular, compare 2 different implementations: (1) w/o PC removes the pre-trained weights frn and fAPD, directly training our 3D-ADLLM; (2) w/o UL removes the sample unbalanced factor. As is shown in Table
8

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot Open- detection results on over all instances.
Method
mloU
mAcc!
m mrrect
mRec
mAP 50
Partial-viev Full-view
OpenAD-Pointeett
3,46
74.59
11.84
5.84
0.02
OpenAD-DGCNN
3.79
74.42
11.13
6.67
0.04
LASO
20.47
71.47
37.95
34.93
2.42
3D-ADLLM (ours)
30.28
70.66
40.89
55.93
27.80
OpenAD-PointNett+
2.17
71.97
5.64
3.74
0,02
OpenAD-DGCNN
2.08
72.00
6.65
3,40
0.02
LASO
11.46
72.14
32.70
16.49
0.70
3D-ADLLM (ours)
28.72
68.28
41.71
47.73
25.63
Table 3: Zero-shot Open-vocabulary detection results on AffordPose data over all instances.
Method
mIoU'
mAcc'
mProc'
mRec
mAPi
OpenAD-PoimtNNet++
7.61
65.13
22.47
13.01
0.37
OpenAD-DGCNN
8.02
66.76
15,83
13.52
0.39
LASO
34.49
77.12
56.04
37.88
8.40
3D-ADLLM (ours)
36.33
74.79
55.46
46.80
36.33
the performance of 3D-ADLLM drops significantly without either of these components. Notably, the substantial performance degradation with about 6% occurs in mloU when the PC module is removed. UL is also critical for our framework. Once it is removed, the performance, there is noticeable reduction in the model's performance.
Effects of Different Backbones. As shown in Table we experimented with different LLM backbones to evaluate the effectiveness of our framework. Specifically, we chose Phi-3.5- mini-instruct (Abdin et al. (2024) and Qwen2-1.5B (Yang et al. 2024) as the LLM backbone.
In terms of experimental results, Phi outperforms Qwen in the full-view setting. However, in the partial-view setting, the performance of Phi shows no significant difference compared Qwen. Based these find- ings, 3D-ADLLM adopts Phi as the default LLM back- bone. In addition to testing different LLM backbones, we also explored different point encoders. Table 4 sum- marizes the performance ULIP2 (Xue al. (024) and Uni3D (Zhou et al. 2023) as point encoders, while ULIP2 obtained slightly better mean accuracy.
Table The efforts with different point encoder fpe in 3D-ADLLM.(Ful) View)
fpe
mloU
Acc
mAec
ULIP2 Uni3D
30.43
29.36
47.78
30.26
26.21
48.16
Effects of Different Learning Objectives. We define the Affordance Region Ratio (Arr) as Patt Pat/Peten Peloud. representing the proportion of affordance regions relative to the point cloud. In the IRAS task, the average Art is approximately 18%. However, for specific categories like "pull" and "listen," " around 5%, while for "wear" it reaches about 40%. Variations Arr across different predictions lead to class imbalances. Dice Loss, segmentation loss function, measures the similar- ity between predictions and ground truth. Unlike Binary Cross-Entropy Loss (BCE), which focuses on pixel-level differences, Dice Loss emphasizes global region similarity, making it more effective for handling data imbalance. As shown Table 5 the model utilizing Dice Loss achieves superior mloU metrics in both and unseen settings. Table demonstrates that while the exclusive appli- cation Dice Loss yields a marginal improvement on unseen data, it does not perform as well on seen data when compared the combined usage of Dice Loss and BCE Loss.

---

Published as a conference paper at ICLR 2025
Table 5: The comparison results regarding differ- ent settings of (oss((full-view)
Table Results 3D-ADLLM variants with
removing different craplonnnts.fulllvvveee
Openset-mloU
Closeset-mloUU
Model
mloU
Acc
mAce.
DICE & BCE
30.43
42.35
3D-ADLLM
30,43
29.36
47.78
DICE BCE
31.00
38.65
3D-ADLLM w/o PC
24.82
20.54
36.73
15.99
31.14
3D-ADILM w/o UL
25.35
26.84
40.69
4.4 QUALITATIVE RESULTS
As shown in Fig. model demonstrates the capacity accurately comprehend object affor- dance given the complex reasoning instruction, noteworthy that even when dealing with small affordance components, such the switch of faucet, model still exhibits decent ability. More- over, our 3D-ADLLM surpasses other models by employing a multi-stage training strategy that facilitates knowledge transfer and extraction of world knowledge from LLMs. For example, when identifying areas on a chair that can take seat Fig. 4 (e) or areas that can wrap around a cup Fig. (g), our model significantly outperforms other models.
OpenAD
(PointNet*+)
OpenAD (DGCNN)
LASO
Ours
3D-ADLLM
GT
Please find...grab..
Please help ..app.....
Where is...unlooc...
Can you...grab..
Please...take
Where is...grab..
Can you...raise.. (h)
.rr.p.......
(b)
(c)
(d)
(e)
Figure 4: The visualization results of our 3D-ADLLM compared with others.
5 CONCLUSION
In this work, we reformulate the traditional affordance detection paradigm into Instruction Rea- soning Affordance Segmentation (IRAS) task, enabling open-world affordance detection. Then, we propose the multi-stage learning strategy with a novel defined Referring Object Part Segmentation (ROPS) task to extract general segmentation knowledge to affordance detection. Finally, we accord- ingly proposed the 3D-AfffordanceLLMM (3D-ADLLM), firstly injecting LLM into 3D affordance perception, framework designed for query reasoning affordance segmentation in 3D open scenar- ios. Experimental results demonstrate the effectiveness of 3D-ADLMM, we hope our work can shed new light the direction of affordance detection in open-world scene in the future.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
We would like to thank the reviewers for their constructive comments. This study is supported by Shenzhen College Stability Support Plan (Grant No. XXWD202208111444280000
National Natural Science Foundation of China (Grant No. 62406092), National Natural Science Founda- tion of China (Grant No. U24B20175), Shenzhen Science and Technology Program (Grant No. KJZID20240903 100011022)))) Research on Efficient Exploration and Self-Evolution of APP Agents & Embodied Intelligent Cerebellum Control Model and Collaborative Feedback Training Project (Grant No. IC20240403047).
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Joya Chen, Difei Gao, Kevin Qinghong Lin, and Mike Zheng Shou. Affordance grounding from demonstration video to target image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-6808, 2023.
Sijin Chen, Xin Chen, Chi Zhang. Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26428 26448,, 2024.
Ali Cheraghian, Shafin Rahman, and Lars Petersson. Zero-shot learning of 3d point cloud objects. In 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1-6. IEEE, 2019.
Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learn- ing for 3d point cloud classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 923-933, 2020.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142- 13153, 2023.
Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: bench- mark for visual object affordance understanding. In proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1778-1787, 2021.
Thanh-Toan Do. Anh Nguyen, and lan Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In 2018 IEEE international conference on robotics and automa- tion (ICRA), pp- 5882-5889, IEEE, 2018
Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang. and Hao Dong. Rlafford: End-to-end affordance learning for robotic manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5880-5886, IEEE, 2023.
James Jerome Gibson. The Senses Considered as Perceptual Systems. Houghton Mifflin, Boston, USA, 1966.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023a.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 1-200022-2494 2023b.
11

---

Published as a conference paper at ICLR 2025
Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 495-504, 2021.
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint 231 2023.
Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: large-scale dataset of hand-object interactions with dfford hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14713-14724, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1579-95899 2024.
Gen Li, Varun Jampani, Deqing Sun, and Laura Sevilla-Lara. Locate: Localize and transfer object parts for weakly supervised affordance grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10922-10931, 2023.
Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language- guided affordance segmentation on 3d object. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14251-14260, 2024.
Weiping Liu, Jia Sun, Wanyi Li, Ting Hu, and Peng Wang. Deep learning on point clouds and its application: A survey. Sensors, 19(19)4488,, 2019.
Liangsheng Lu, Wei Zhai, Hongchen Luo, Yu Kang, and Yang Cao. Phrase-based affordance detec- tion via eyelic bilateral interaction. IEEE Transactions on Artificial Intelligence, 4(5): 1188-198, 2022.
Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning affordance grounding from exocentric images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2252-2261, 2022.
Eloise Matheson, Riccardo Minto, Emanuele GG Zampieri, Maurizio Faccio, and Giulio Rosati. Human-robot collaboration in manufacturing applications: A review. Robotics, 8(4): 2019.
Jinpeng Mi, Hongzhuo Liang, Nikolaos Katsakis, Song Tang, Qingdu Li, Changshui Zhang, and Jianwei Zhang. Intention natural language grounding via object affordance detection and intention semantic extraction. Frontiers in Neurorobotics, 14:26, 2020.
Bjöm Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In 2021 International Confer- ence on Vision (3DV), pp. 992-1002. IEEE, 2021.
Huaqing Min, Ronghua Luo, Jinhui Zhu, Sheng Bi, et al. Affordance research in developmental robotics: A survey. IEEE Transactions on Cognitive and Developmental Systems, 844)2237-255, 2016.
Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: large-scale benchmark for fine-graine fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. 020-afford: Annotation-fie large-scale object-object affordance learning. In Conference on robot learning, pp. 1666-1677. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Bogdan Moldovan, Plinio Moreno, Martijn Van Otterlo, José Santos- Victor, and Luc De Raedt. Learning relational affordance models for robots in multi-object manipulation tasks. In 2012 ieee International Conference Robotics and Automation, pp. 4373-4378. IEEE, 2012.
Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-objer object interac- tion hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8688-8697, 2019.
Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis. Detecting object affordances with convolutional neural networks. In 2016 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2765-2770. IEEE, 2016.
Toan Nguyen, Minh Nhat Vu, An Vuong, Dzung Nguyen, Thieu Vo. Ngan Le, and Anh Nguyen. open--occabulary affordance detection in 3d point clouds. In 2023 IEEE/RSJ International Con- ference Intelligent Robots and Systems (IROS), pp. 5692-5698. IEEE, 2023.
Abel Pacheco-Ortega and Walterio Mayol-Cuervas. One-shot learning for human affordance detec- tion. In European Conference on Computer Vision, pp. 758-766. Springer, 2022.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. arXiv preprint 4711 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Debaditya Roy and Basura Fernando. Action anticipation using pairwise human-object interactions and transformers. IEEE Transactions on Image Processing, 30:81 16-8129, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 30, 2017.
Kashi Venkatesh Vishwanath, Diwaker Gupta, Amin Vahdat, and Ken Yocum. Modelnet: Towards a datacenter emulation environment. In 2009 IEEE Ninth International Conference on Peer-to-Peer Computing, pp. 8 81-82. IEEE, 2009.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5): 1-12, 2019.
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08766 2023.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1179-1189, 2023.
Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27091-27101, 2024. 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.106) 2024.
13

---

Published as a conference paper at ICLR 2025
Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Grounding 3d object affordance from interactions in images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10905-10915) 20233.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19313-19322, 2022.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16259 16268, 2021.
Junsheng Zhou, Jinsheng Wang, Baorui Ma Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale, arXiv preprint urXiv:2310.06773, 2023.
A APPENDIX
A.1 BASELINE MODELS DETAILS
We compare our method with the following recent methods for zero-shot learning 3D point clouds: ZSLPC (Cheraghian et al. 2019), TZSLPC (Cheraghian et al., 2020), and 3DGenZ (Michele et al.) 202T]. For these baselines, we change their original text encoders with CLIP and retain the same settings in OpenAD (Nguyen et al. 2023). Furthermore, we incorporate two affordance detection works (IAGNet (Yang al. 2023) and LASO (Li et al., 2024)) to provide a more comprehensive comparison of our approach. For IAGNet (Yang al. (023), an affordance detection method that utilizes paired image-point cloud data as input. To tailor LAGNet (Yang et al. 2023) to our require- ments, scamlessly integrate language model in place of its original image backbone, while maintaining the rest its architecture unchanged. Shapel point cloud model that accepts point cloud and natural language inputs and possesses grounding capabilities. Consequently, we leverage its grounding abilities to perform zero-shot detection and calculate masks for comparison.
(Qi et al. 2024) is a large-scale
A.2 COMPARISON RESULTS ON CLOSE SET
In this work, we primarily focus enhancing affordance detection capabilities in open-world scene. However, our model still performs well in closed-set affordance detection tasks. As is shown in Fig. 7 Fig. & our 3D-ADLLM achieves optimal performance on nearly all metrics in both the over- all-classes and over-all-instances settings.
Table 7: Main results of 3D-ADLLM compared with other methods on close-set detection over all class.
Full-view Acc
Partial-view Acc
Method
m
mAce
m
mAcc
Point Transformer (hao et al.
2021
41.26
67.03
40.51
65.34
+ointNet++ QQ et al. 2017
41.26
68.14
41.10
66.74
DGCNN Wang et al.
2019
42.09
61.47
41.93
63.12
OpenAD-PPintNNe PoinLNet++ Nguyen
2023
40.17
38.61
66.83
40.44
38.92
65.84
OpenAD-DGNNN (N Nguven el al. 2023
41.17
35.71
59.17
39.87
35.15
59.27
IAGNet Yang al. 2023
40.04
35.12
53.05
41.24
34.68
52.58
LASO [L. al. 2024
41.31
35.02
53.96
40.11
35.21
52.68
3D-LDLMM
42.85
41.84
66.35
41.92
43.40
61.93
14

---

Published as a conference paper at ICLR 2025
Table 8: The performance of close set affordance detection over all instances.
Method
mloU
mAcc!
mPrre
m mRec
m.A 50
Partial-viewy Full-view
OpenAD-PointNeHt
28.34
64.11
33.91
61.45
5.12
OpenAD-DGCNN
26.98
65.94
34.38
54.76
4.77
LASO
44.43
83.80
62.73
60.25
21.13
3D-ADLLM (ours)
46.29
81.24
57.90
64.27
46.38
OpenAD-PointNert
29.50
63.26
35.21
61.34
6.77
OpenAD-DGCNN
17.07
67.11
27.96
30.15
1.87
LASO
43.35
82.31
60.27
59.57
20.85
3D-ADLLM (ours)
44.06
79.64
56.23
64.21
46.60
A.3 DATA ANALYSIS
A.3.1 THE DETAILED SETTINGS FULL VIEW PARTIAL VIEW.
Building the IRAS dataset based on the 3D AffordanceNet (OpenAD) Dataset (Nguyen al. 2023 We retained the settings for both full-view point cloud and partial-view point cloud.
Full-view:
Given an object as 3D point cloud without knowing the affordances supported by object, the full-shape affordance estimation task aims to estimate the supported affordance type and predict the point-wise probabilistic score of affordance.
Partial-view: in real-world application scenarios, represented as partial point cloud. Therefore, another important task we are concerned with is to estimate the affordance from partial point cloud.
only expect partial view of 3D shapes,
A.3.2 THE VISUALIZATION OF DATA ANALYSIS.
A.4 TRAINING DETAILS
constructed our IROS dataset based on the PartNet dataset and aim to acquire general segmenta- tion knowledge through the pre-training phase. To balance training costs with model performance, we selectively sampled a subset of the data based on categories to obtain general segmentation knowledge for objects.
15

---

Published as a conference paper at ICLR 2025
Object Affordance Distribution
Ooject
Bag
Eacc
Butte
Bost
8000
Thappay
Door
Earshone
Num Samples
8000
Fauoeet
Laptop
Microwaee
Mag
4000
Retegertto
Solssors
Storagel-amtun
2000
contain
display
gravp
layable
leten
move
operaabl
paurable
presa
pisiabble
alitable
alob
support
enter
wrap sramp
Affordance Type
Figure 5: The analysis of IRAS task.
Object Affordance Distribution
200
Object
bodee
disperaue
faucet
hancle-cof cotte
pot
90/so0rs
Num of Samples
25
handle--ren grasp
lnver
iross
support
Wraph grans
Affordance Type
Figure 6: The analysis of extensive test dataset in sec. 4.2.2
16

---

Published as a conference paper at ICLR 2025
Inxtaroes per Object
Object Farts Distribttonn
Ocgeca
Meryoaye
Natbere el Paatt
Cook
Larng
Truch
Scision
Chaot
Dhotdr
Mad
dipricse
Retiperaa dtn Shee
aloicee Centzoe
Objectt
Figure 7: The analysis of ROPS task.
17


Reviews:
[{"review_id": "OsIMn5jU7z", "reviewer": "Reviewer_uifM", "summary": "The paper introduces 3D-AffordanceLLM, a novel framework for open-vocabulary affordance detection in 3D point clouds that reformulates traditional affordance detection as an Instruction Reasoning Affordance Segmentation (IRAS) task. Moving beyond fixed label-based approaches, the framework combines Large Language Models (LLMs) with a custom affordance decoder to generate segmentation masks from natural language queries. To address the challenge of limited 3D affordance datasets, the authors propose a multi-stage training strategy that first extracts knowledge from general segmentation tasks through a novel Referring Object Part Segmentation (ROPS) pre-training task, followed by fine-tuning with IRAS. This approach leverages the rich world knowledge and reasoning capabilities of LLMs while equipping the model with general recognition and segmentation capabilities at the object-part level. The method demonstrates significant improvement, achieving approximately 8% increase in mIoU on open-vocabulary affordance detection tasks, showcasing its effectiveness in bridging the gap between natural language understanding and 3D affordance detection.\n.", "questions": "See my primary concerns at weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of affordance detection in 3D is important for robotics applications and human-robot interaction1\n- The technical approach is sound and demonstrates meaningful improvements over baselines\n- The paper is well-written and clearly structured", "weaknesses": "My primary concerns are:\n\n- The motivation for using LLMs is not sufficiently justified. The paper argues that traditional methods are limited by fixed label sets and short-text detection, but recent works like CLIP-based approaches have shown success in open-vocabulary detection [1].\n- The paper lacks proper discussion of its relationship to similar concurrent work, particularly AffordanceLLM [2]. The architectural similarities and differences should be explicitly addressed. The primary difference is the feature encoder (PointBERF for point cloud vs OWLViT for 2D images).\n\n[1] Gen Li, Deqing Sun, Laura Sevilla-Lara, Varun Jampani. One-Shot Open Affordance Learning with Foundation Models. CVPR 2024.\n[2] Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, Li Erran Li. AffordanceLLM: Grounding Affordance from Vision Language Models. CVPRW 2024.\n\nAdditional comments which do not need to be addressed\n\nL366 projcetor -> projector", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRzsB440rp", "reviewer": "Reviewer_oFkQ", "summary": "The paper focuses on point cloud affordance segmentation. It adapts the LISA model to the 3D affordance domain. Specifically, given a text prompt requesting an affordance and a 3D point cloud, it fine-tunes a Large Language Model (LLM) to predict a specialized affordance token, which is decoded with point cloud embeddings to predict the segmentation mask on the point cloud. It is first trained on the 3D part dataset PartNet (pointwise part label) and further on the annotated 3D AffordanceNet dataset (pointwise affordance label with questions). It appears to be evaluated on the OpenAD dataset and claims to outperform previous works.", "questions": "The paper doesn’t clearly state which dataset is used for evaluation. Inferred from L400, it may be 3D AffordanceNet (OpenAD annotated). Please make it clearer in the paper.\n\nMoreover, there is a clear discrepancy between the quantitative results in OpenAD and this paper. Please explain.\n\nEditing Suggestions:\n\nL109: transferr → transfer\n\nL313: Defination → Definition\n\nL366: typo: projector\n\nL370: Refering → Referring\n\nL534: enableing → enabling\n\nL708: no space in baselines\n\nI suggest the authors use an LLM to perform a grammar check.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It extends LISA to the 3D domain. It's interesting to find LLMs can also predict the next token for 3D tasks.\n\n2. It makes point queries more open-vocabulary. The paper supports asking for affordances in natural-language questions, making the input question more flexible.\n\n3. Dataset annotation effort. The authors annotate the 3D AffordanceNet dataset (point labels) with questions.", "weaknesses": "1. Not open-world. The method cannot generalize to unseen 3D object parts, as the current 3D part dataset PartNet covers a limited category of objects with few instances. The proposed method is 3D part data dependent, which is a significant bottleneck. It is too preliminary to claim it works in a 3D open-world scene. In contrast, existing works on open-world part segmentation of 3D point clouds, such as PartSLIP, address this issue.\n\n2. No detailed table on per-class evaluation results. The dataset is unbalanced, and per-class results should be reported.\n\n3. No qualitative results on real-world partial-view point cloud segmentation.\n\n4. No multi-affordance experiments. A single part or region on an object may have multiple reasonable affordances, which is crucial in an open-vocabulary scenario. Related experiments are missing here.\n\n5. Ablation: What if the first stage is also trained on 3D AffordanceNet?\n\n6. No robotics experiments. Affordance is always about interaction, which differentiates it from general part semantics. There are no real-world experiments to show if these better predictions lead to higher robotic manipulation success rates. Many papers introduce real-world robotic experiments to demonstrate learned affordance: SayCan, Where2Explore, OpenAD, SAGE, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l8xDQK7Bw2", "reviewer": "Reviewer_PzNX", "summary": "This paper proposes to solve open-world affordance prediction by integrating LLM into the pipeline. The pipeline start with pre-training the point cloud backbone and affordance mask decoder on part segmentation task conditioned on text. Then, the LLM is fine-tuned with affordance discovery on specific tasks. During inference, the LLM will take the complex text description and point cloud tokens encoded from a fixed encoder and output the affordance token <AFF>, conditioned on which the pre-trained affordance mask decoder will predict the affordance mask on the point cloud. Compared with zero-shot learning methods on point cloud, this framework outperforms in terms of accuracy and generalization ability.", "questions": "1. Could you please show more qualitative results on the affordance prediction to see whether it has tendency for predicting the whole part?\n2. The use of two point cloud networks need to be clarified since there are misalignment between two figures and descriptions in the paper.\n3. Could you explain more on the two \"Projector\" in Figure 3? Are they the same? It seems that they have different input and output domain.\n4. What does the text targets $y_{txt}$ in IRAS fine-tuning refer to?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Leveraging common sense from LLM for 3D affordance grounding is well-motivated, which could potentially help downstream open-vocabulary manipulation.\n2. The main challenging of using LLM for 3D affordance prediction is handling the multimodal input and output format as segmentation mask. This paper propose a neat way to tokenize point features and convert the text prediction to a condition for affordance discovery.", "weaknesses": "1. While this framework connects the LLM with point encoding and mask decoding and could handle complex text instructions, my major concern is about the way of combining LLM with affordance prediction. In the IRAS fine-tuning stage, the output of the LLM is directly supervised by text targets $y_{txt}$, combined with the fact that the point backbone and mask decoder are pre-trained on text-conditioned part segmentation task, it seems that the current framework is using LLM as just a part classifier and use decoder as a text-conditioned part segmentation network. The visualization also shows that this system tend to predict the whole part as the affordance mask. This could limit the method's application on more fine-grained affordance grounding task, where only a small region of part should be activate. Besides, in this way, the benefit of using LLM is just handling complex instruction and convert it into segmentation text instead of providing detailed prior on affordance (e.g., which region provides a firm grasping). I believe there could be more integrated way of combining LLM to fully leverage the priors from it.\n2. The design of the architecture also shows that the current function of LLM is just handling complex text and output text condition for segmentation. For example, in Figure 3, the text condition of the Mask Decoder is the name of part during pre-training, whereas in IRAS fine-tuning it is replace with the output of LLM.\n3. The figure 2 and Figure 3 have misalignment on the use of two point cloud networks. I seems that there is a typo in Figure 2 since the Point Backbone is \"specifically tailored for segmentation tasks\".\n4. Minor: The quality of Figure 2 is not good. The fonts of some text seems wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YhV61R8N18", "reviewer": "Reviewer_ovue", "summary": "The paper’s key advantage lies in its integration of large language models (LLMs) into affordance segmentation, leveraging their semantic richness to enhance understanding and address limitations of fixed affordance labels in prior methods, thus increasing adaptability in open-world settings. The proposed task, Instruction Reasoning Affordance Segmentation (IRAS), combines LLM-encoded text features with point features, with a pretraining step on part segmentation to address data scarcity.\n\nHowever, IRAS closely resembles prior affordance tasks, particularly Language-guided Affordance Segmentation on 3D Objects (LASO), which raises questions about its novelty. Additionally, the methodology shows minimal distinction from existing work, with pretraining as the main difference, which does not provide strong novelty. Another limitation is the lack of rigorous comparative experiments to demonstrate clear advantages over previous approaches, leaving key aspects, such as the contributions of LLM features and pretraining, unclarified.\n\nThe paper also suffers from poor organization and inconsistent notation, which hinder readability. While additional experiments could strengthen validation, the limited novelty and unclear contributions reduce the impact of the work. Therefore, I recommend rejection.", "questions": "- **Q1**: In **Figure 1**, the example question is ambiguous. Given a prompt like “something I can use to drink water” for a mug, most people would interpret this as referring to the whole mug, not specifically the handle or the body. In **Figure 2**, the questions are much clearer and directly target specific parts. Can you explain how your model infers that the handle of a mug is the intended affordance in such an ambiguous scenario in **Figure 1**? Is any specific design in your method to handle ambiguity or resolve cases with multiple possible interpretations? You could provide a range of example queries, from ambiguous to specific, to illustrate how the model addresses each type of query and manages interpretational differences.\n- **Q2**: In **Line 210**, the explanation and organization of the point encoder section are unclear and inconsistent. Why is the subscript for $f_{pe}$ lowercase, while $f_{PB}$ is uppercase? According to the projector section, the feature X is fed into the LLM. However, in **Figure 2**, it appears that feature X originates from the \"point backbone\" rather than the \"point encoder,\" as mentioned in **Line 210**. Could you clarify this inconsistency?\n- **Q3**: In **Lines 289-297**, the explanation is unclear, and the notations are inconsistent. The symbols $\\textbf{f}$, $f_{pb}$, and $\\textbf{p}_{clouds}$ do not match the earlier sections or the figures. Could you provide a clearer and more cohesive explanation?\n- **Q4**: In **Figures 2 and 3**, the \"b\" stage is inconsistent. In **Figure 2**, the point cloud feature input into the LLM is encoded by a \"trainable point backbone,\" while in **Figure 3**, it is processed by a \"frozen point encoder.\" Could you clarify this pipeline, ensuring consistent notations and descriptions?\n- **Q5**: In **Line 341**, how is the ground truth text obtained?\n- **Q6**: In **Line 353**, the formula for $L_{mask}$ appears incorrect. The summation should be outside the formula. It seems that your query already includes the affordance type, meaning the affordance type is known beforehand. Therefore, this task can be treated as a binary classification problem, as opposed to a multi-class setting. In OpenAD, they maximize the cosine similarity between points and the target affordance type, making it a multi-class classification problem. Given this setup, I’m curious about the role of the affordance class weights in your binary classification loss. Could you clarify how these affordance class weights are used in this binary classification context?\n- **Q7**: In **Table 1**, you claim the experiment is zero-shot and open-vocabulary and you seem to reference results from TZSLPC, 3DGenZ, and ZSLPC in OpenAD[2]’s **Table 1**. If your setup is identical to OpenAD’s **Table 1**, which evaluates 37 classes, why do you mention evaluating only 18 valid classes in **Line 402**? Could you clarify your experimental setup?\n- **Q8**: In **Table 1**, how is the accuracy (Acc) computed? In OpenAD’s **Table 1**, all reported accuracies are over 40%, yet here, your best result is only 29%, and the two OpenAD variants show values around 3.9%. Given that your reported mIoU and mAcc for the two OpenAD variants are closer to those in OpenAD, one would expect similar Acc values. Where do these reported accuracy results for the two OpenAD variants come from, and why aren’t you referencing the accuracies for TZSLPC, 3DGenZ, and ZSLPC, even though you reference other metrics for these models?\n- **Q9**: In **Table 1**, could you specify what “full view” and “partial view” refer to in your experimental setup?\n- **Q10**: In **Table 2**, could you further clarify the difference between the mAcc in **Table 2** and the mAcc in **Table 1**? It’s somewhat understandable that \"over all classes\" and \"over all instances\" might differ, but why does, for example, the mAcc for OpenAD-PointNet++ differ so significantly from 16.4 in **Table 1** to 74.59 in **Table 2**?\n- **Q11**: In **Tables 2 and 3**, why is the performance on out-of-distribution (OOD) unseen object data higher than on the test set of your primary dataset in **Table 2**? Adding visual examples could help clarify this discrepancy.\n- **Q12**: In **Tables 4 and 6**, the experiment setting is highly unclear, on what dataset were the experiments conducted, and did you use the full view or partial view setting?\n\n[2]Toan Nguyen, Minh Nhat Vu, An Vuong, Dzung Nguyen, Thieu Vo, Ngan Le, and Anh Nguyen.Open-vocabulary affordance detection in 3d point clouds. IROS 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Motivation**: the strength of this paper is its motivation to leverage large language models (LLMs) for enhancing semantic comprehension in 3D affordance detection. Unlike previous methods that are limited by a fixed set of affordance labels, this approach uses an LLM to interpret complex, instruction-based queries, enabling the model to recognize affordances in a more flexible and context-aware manner. By incorporating LLMs, the model can process nuanced instructions, overcoming the constraints of rigid label sets and allowing it to generalize affordances in open-world settings.\n\n2.  **Pretraining**: The authors address the challenge of limited data by introducing a pretraining stage on part segmentation, which enhances the model's ability to capture part-level features even with a smaller affordance dataset. This pretraining strategy not only compensates for data scarcity but also provides the model with a foundation to learn more transferable features, contributing to improved performance in affordance segmentation.", "weaknesses": "1. **Limited Novelty Regarding the Proposed Task**: The authors introduce a task, Instruction Reasoning Affordance Segmentation (IRAS), aiming to address perceived limitations in existing 3D affordance detection approaches. However, IRAS shares substantial overlap with previous work, particularly with the Language-guided Affordance Segmentation on 3D Objects (LASO) task[1]. Both IRAS and LASO involve generating a mask over functional parts of a 3D object point cloud based on a question or instruction. Specifically, each query in both tasks consists of an object type and an affordance type, with the objective of inferring the functional part relevant to the query. To clarify the distinctiveness of IRAS, the authors should elaborate on how their queries are generated, detailing the heuristics and rationale behind their construction, and the specific differences between IRAS and LASO, and any innovations that set IRAS apart as a unique contribution to the field. Without this, the similarities between IRAS and LASO bring into question the novelty of IRAS as a distinct task.\n2. **Limited Innovation in Methodology**: To address these challenges, the authors integrate a large language model (LLM) into a point cloud segmentation framework, using the LLM to process instructions and produce text features along with point cloud features for affordance mask prediction. However, the design involving two separate point encoders is confusing. The authors may claim that incorporating multimodal text and point cloud inputs allows the LLM to capture semantic information better, potentially enhancing segmentation performance. However, given the limited dataset size, it is unclear if sufficient point-text alignment can realistically be achieved. Furthermore, this design lacks supporting experiments to validate its effectiveness. Aside from a part segmentation pretraining step for extracting part-based features, the approach doesn't show a significant difference from LASO, which uses text features from LLM and point cloud features from a point backbone to decode affordance masks. This makes the contribution of pretraining alone insufficient as a distinguishing factor for novelty. The author needs to explain the core difference between your method and LASO.\n3. **Lack of Rigorous Experimental Validation**: though the proposed method shows performance gains over previous approaches, the effectiveness of the proposed design is not convincingly demonstrated due to concerns about the experimental setup and baseline data, as noted in the questions. A key issue is the lack of ablations to demonstrate the extent to which the improvements stem from the pretraining stage, the large-scale pretrained backbone, or the method itself. The following baseline experiments could strengthen the validation. The authors are encouraged to prioritize experiments based on their assessment of each module's importance, conducting analyses to substantiate the effectiveness of different components and explaining which aspects they consider most critical, and why.\n    - **Basic Baseline A**: Use the LLM to encode only text features, inputting them into the decoder along with point features encoded by a standard backbone, like PointNet++. This approach would closely follow LASO’s baseline method.\n    - **Pretraining Variant B**: Use the proposed pretraining task, ROPS, train variant **A** and then fine-tune **A** on the IRAS task. This would help isolate the contribution of pretraining compared to **A** and show its impact on performance. In your current experiments, you simply remove the pretraining step (as shown in **Table** 6). This makes it difficult to determine whether the improvements over LASO are due to the proposed `AFF` token or the use of the Point Transformer segmentation backbone. The Without additional experiments, it’s challenging to draw convincing conclusions about the source of the performance gains.\n    - **Backbone Variant C**: Replace the point encoder with the same point segmentation backbone (Point Transformer) used in this work, to determine the extent to which the performance improvement is due to the backbone choice rather than the proposed method.\n    - **Dual Encoder Variant D**: Introduce a second point encoder, initially using PointNet++, which outputs point features to the LLM along with text tokens. The LLM’s output is then fed into the decoder from Variant **C**. This setup would reveal how much the proposed `AFF` token contributes to performance.\n    - **Current Approach E**: Test the full model with the proposed `AFF` token integration and two encoders to measure the combined effectiveness of all components.\n    - **Comparison with previous approaches F**: The fairest comparison to the previous approaches is to remove the pretrain stage and replace your segmentation backbone with PointNet++.\n4. **Inadequate Clarity and Organization in the Paper**: The methodology section lacks clear logical flow, and inconsistent notation makes it challenging to follow. The experimental setup is insufficiently detailed, and numerous errors in descriptions further reduce clarity. Improvements in notation consistency, methodology organization, and experimental description are needed to enhance the overall clarity and coherence of the proposed approach.\n\n[1]Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language-\nguided affordance segmentation on 3d object. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CU9poWm2wz", "reviewer": "Reviewer_UMiE", "summary": "The authors propose a method to enable open-vocabulary affordance detection in the 3D world via utilising the knowledge from Large Language Models. A new framework named 3D-AffordanceLLM is proposed. The authors propose a multi-stage training strategy to overcome the problem of the scarcity of 3D affordance datasets. A new pre-training task is therefore designed, named Referring Object Part Segmentation and is to train the model to have the capability of general recognition and segmentation. Subsequently, the model is fine-tuned with Instruction Reasoning Affordance Segmentation. Experiments show the effectiveness of the proposed method.", "questions": "I would suggest the authors to address the questions raised in the weakness section in the discussion period.\n\n\n*** Post - discussion\n\nThe authors' comments have addressed some of my concerns, and I therefore decide to change raise my score to be 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I like the idea of using the knowledge from Large Language Models for open-vocabulary detection. It is sensible as LLM may contain knowledge about the world and can have good generalisation in terms of open-vocabulary. \n\n2. The design of the architecture is novel. An <AFF> token is added into the language model, which connects to the affordance prediction. It is a natural design.", "weaknesses": "1. For organisation of the paper, it would be better if the authors can focus more on the current work rather than using several paragraphs to describe the development of previous works some of which are not closely related.\n\n2. For experiments, it would be better if the authors can explain more about the numbers. For example, in the last row of Table 2, it is a bit strange why mAcc is so low compared with the rest, while the other numbers are the highest. Are these metrics consistent?\n\n3. Both the pre-training and evaluation of the model are conducted on synthetic datasets, but it is an important problem how the model can work in the real world as there can be a large domain gap between the real world scenarios and the synthetic datasets. I am not fully convinced unless the authors can show their training on synthetic datasets can have a good zero-shot capability in real world scenarios. It would be better if the authors can provide some results in real world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)