# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 5 Conclusions  
In this work, we address the problem of affordance grounding by introducing the instruction-following affordance-grounding network (IAG-Net), representing a significant advancement in affordance segmentation. This is particularly notable due to its ability to directly interpret natural language instructions and bypass traditional object detection stages. Our key findings emphasize the importance of both sentence-level and task-specific contexts in visual language tasks, especially in achieving precise part-level affordance grounding. We contribute to the field by creating two multimodal datasets, namely, IIT-AFF VL and UMD VL, which are built upon the existing image-only datasets IIT-AFF and UMD. By augmenting images with affordance annotation with text instructions, we generate instruction-following affordance masks. The creation of the IIT-AFF VL and UMD VL datasets represents a notable contribution to the field, providing comprehensive resources for future research in fine-grained vision-language affordance understanding. These multimodal datasets not only extend existing image-only resources but also offer rich opportunities for exploring advanced multimodal interactions. While our comprehensive \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **TaskInf** & **MH-VSMA** & **Overall Acc** & **Mean Acc** & **Freq Acc** & **mIoU** & \(F_{B}^{\omega}\) \\ \hline Base & & & 94.8 & 70.5 & 90.5 & 74.9 & 43.0 \\ Base+TaskInf & ✓ & & **95.4** & 71.7 & **91.4** & 75.8 & 43.9 \\ IAGNet & ✓ & ✓ & 94.0 & **85.0** & 89.0 & **79.3** & **46.0** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results of the ablation study for each component. The symbol \(\checkmark\) indicates the addition of the corresponding component. Bold text is utilized to emphasize the best values achieved for each metric. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Head Number** & **Mean Acc** & **mIoU** & \(F_{B}^{\omega}\) & **FPS** \\ \hline
1-VSMA & 74.63 & 74.08 & 41.18 & **4.10** \\
2-VSMA & 82.85 & 76.39 & 44.10 & 4.02 \\
3-VSMA & 81.99 & 76.31 & 44.30 & 3.82 \\
4-VSMA & 83.94 & 76.68 & 45.86 & 3.61 \\
5-VSMA & **84.99** & **79.32** & **45.99** & 3.56 \\
6-VSMA & 81.41 & 75.48 & 41.77 & 3.55 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Hyperparameters for MH-VSMA, with different head numbers of VSMA. The bold text is used to highlight the best value for each metric. evaluations demonstrate the effectiveness and generalizability of IAG-Net, it is essential to acknowledge the limitations and challenges encountered. Since existing affordance datasets lack variety, performance in highly cluttered scenes or with objects exhibiting complex or subtle affordances may be affected. In summary, our work represents a significant step forward in vision-language affordance grounding. There remain opportunities for further exploration and refinement. Further enhancements to IAG-Net will involve capitalizing on large-scale multimodal models and advancing robustness in challenging contexts. Another avenue for improvement involves integrating additional knowledge sources, such as knowledge graphs, to enhance the understanding of affordance grounding. Furthermore, we aim to integrate richer knowledge sources and explore expansive applications of our methodology, particularly in interactive robotics and complex task execution. Conceptualization, W.Q. ; methodology, W.Q., L.G. and J.C.; software, J.C., L.G. and W.Q. ; validation, X.J. ; writing--original draft preparation, L.G. and X.J. ; writing--review and editing, W.Q. ; visualization, L.G. and W.Q. ; supervision, W.Q. All authors have read and agreed to the published version of the manuscript. This research received no external funding. Data Availability Statement: Data is available at [[https://github.com/WenQu-NEU/Affordance-Grounding-Dataset](https://github.com/WenQu-NEU/Affordance-Grounding-Dataset)]([https://github.com/WenQu-NEU/Affordance-Grounding-Dataset](https://github.com/WenQu-NEU/Affordance-Grounding-Dataset)) (accessed on 30 March 2024).