# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 4 Experiments and Evaluation
### Comparison Results (Rq1)
BPN [14]: BPN considers the potential relationship between object categories and object affordances and proposes a boundary-preserving network. It is an image-only method for affordance detection. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers. 6. GSE [15]: GSE utilizes a repeated multi-scale feature-map-fusion network to produce category-relevant feature maps. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers (the backbone with ResNet50). 7. ADOSMNet [16]: This state-of-the-art affordance detection method operates without language instructions. We considered it an upper bound in our comparisons since it eliminates the need to interpret language and achieve state-of-the-art results on the two datasets (IIT-AFF and UMD). Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers (the backbone with ResNet50). Unfortunately, ADOSMNet's performance on the UMD VL dataset could not be compared because the dataset selected is specific to our study (details in Section 4.1). 8. Cascaded Method: Our custom-designed cascaded method combines object detection and affordance segmentation. It begins by localizing the object mentioned in the instructions using Faster-RCNN [40]. The target object is the region with the highest prediction probability, and affordance segmentation is performed within the target region with DeepLabV3+ [74]). This approach provides insights into the impact of object grounding on the whole process. BKINet [56]: BKINet is a state-of-the-art method for referring image segmentation that involves uniquely segmenting a reference object directly based on the input text. We trained this model on our dataset to evaluate its performance under the same conditions as though in our proposed method. * TransVG [44]: TransVG is a classical visual grounding method that predict object localization with language. We trained this model on our dataset to provide the object-level grounding results, which is easier than our part-level task. In Tables 3 and 4, we present the accuracy values of the evaluated methods across the two datasets. The clear background and minimal environmental interference of the UMD dataset likely contribute to its generally higher performance than that of IIT-AFF. This observation has shown the importance of dataset characteristics in influencing the model accuracy. Among the three categories of methods, image-only methods, by concentrating solely on visual cues, avoid the complexities and potential inaccuracies introduced by language data integration, which explains their superior performance. However, when object grounding is integrated with the image-only method in the cascaded approach, there is a notable performance decrease. For instance, compared with that of DeepLabV3+, the cascaded method shows a decrease in performance by \(-1.41\%\), \(-1.69\%\), \(-8.59\%\), and \(-43.07\%\) on the two datasets. The cascaded method's performance drop, as indicated by the comparison with DeepLabV3+, suggests that the sequential process of object grounding followed by affordance segmentation may introduce compounded errors, leading to lower overall accuracy. Considering the image characteristics of UMD and IIT-AFF, we find that the more complex the image content is, the larger the performance decreases for the cascaded method. We also evaluated the accuracy of object localization with the visual grounding method TransVG [44], which has average accuracy values of approximately 87.28 on the IIT-AFF dataset and 99.78 on the UMD dataset. This result highlights a challenge for both cascaded and multimodal methods, which process both image and language instructions, as they tend to underperform compared to image-centric approaches. Our IAG-Net notably outperforms the cascaded and multimodal RIS methods across most categories, demonstrating its robustness and effectiveness.