# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 4 Experiments and Evaluation
### Comparison Results (Rq1)
BKINet's underperformance in categories such as \"cut\", \"hit\", \"pound\", and \"support\" could be attributed to the limited sample size in these affordance categories within the IIT-AFF VL dataset. Our method, IAG-Net, demonstrates competitive results comparable to those of image-only methods and appears to be less influenced by dataset variability. In addition to the above methods, we also tested the large multimodal model NExT-Chat [78] for this task using the provided demo ([[https://next-chatv.github.io/](https://next-chatv.github.io/)]([https://next-chatv.github.io/](https://next-chatv.github.io/))). NExT-Chat is a large multimodal model for detection and segmentation that can handle multiple tasks, such as visual grounding, region captioning, and grounded reasoning. While NExt-Chat is capable of bounding boxes or segmentation for entire objects, it is unable to achieve part-level visual understanding. Enhancing large multimodal models such as NExT-Chat for more granular tasks points to an area for future exploration and could be a valuable direction for subsequent research. **Visualization:** To further illustrate the subjective performance of IAG-Net, we present a series of visualizations that compare our model's affordance-grounding results with the ground truth. Each set in our visual representation, as displayed in Figure 6, comprises four parts. The first column displays the original image paired with its corresponding language instruction. The second column shows the overlap of the predicted affordance mask on the original image. The third and fourth columns provide a side-by-side comparison of the affordance prediction maps of IAG-Net against the ground-truth annotations. These visual examples demonstrate that IAG-Net is capable of effectively interpreting language instructions and aligning closely with ground-truth affordance maps. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{6}{c}{**Image Only**} & \multicolumn{6}{c}{**Multimodal-Based**} \\ \cline{2-13}  & **DeepLabV3+** & **OCRNet** & **AffordanceNet** & **RelaNetGSE** & **BPN** & **ADOSMNet** & **Cascaded** & **BKINet** & **LAG-Net (Ours)** \\ \cline{2-13}  & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) & \(F_{P}^{\omega}\) & \(F_{P}^{\omega}\) & \(F_{P}^{\omega}\) & \(\mathbf{\Pi_{P}^{\omega}}\) & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) \\ \hline contain & 85.68 & 63.98 & 88.35 & 70.85 & 79.61 & 80.20 & 87.92 & 80.62 & 87.77 & 73.18 & 66.11 & 81.79 & 65.90 & 87.72 & 70.25 \\ cut & 69.29 & 61.58 & 79.20 & 61.94 & 75.68 & 78.04 & 65.34 & 79.23 & 84.68 & 67.45 & 58.68 & 63.37 & 43.01 & 73.29 & 65.11 \\ display & 87.03 & 39.30 & 89.46 & 41.22 & 77.81 & 79.14 & 91.90 & 80.55 & 84.26 & 74.37 & 39.18 & 96.68 & 77.17 & 95.15 & 42.84 \\ engine & 75.60 & 73.99 & 79.82 & 73.96 & 77.50 & 81.22 & 89.11 & 81.49 & 84.37 & 58.94 & 51.53 & 78.52 & 75.87 & 85.39 & 79.58 \\ grasp & 75.95 & 29.28 & 78.59 & 31.61 & 68.48 & 71.99 & 79.76 & 72.96 & 83.07 & 70.60 & 29.42 & 69.12 & 99.31 & 84.99 & 33.68 \\ hit & 85.08 & 40.07 & 94.97 & 39.12 & 70.75 & 88.52 & 90.51 & 88.84 & 84.64 & 89.96 & 35.53 & 64.25 & 75.00 & 92.55 & 43.19 \\ ground & 79.47 & 54.82 & 79.21 & 53.69 & 67.91 & 79.59 & 77.59 & 82.68 & 72.66 & 46.09 & 35.56 & 51.28 & 75.43 & 55.21 \\ support & 79.60 & 67.88 & 81.36 & 61.09 & 69.81 & 80.12 & 78.41 & 80.96 & 83.74 & 76.98 & 56.59 & 71.44 & 68.63 & 77.99 & 51.31 \\ w-grasp & 88.58 & 70.08 & 82.13 & 73.20 & 70.98 & 74.56 & 89.43 & 74.56 & 83.35 & 67.56 & 64.06 & 52.43 & 76.06 & 90.29 & 70.31 \\ Average & 81.03 & 55.66 & 83.68 & 56.26 & 73.35 & 78.92 & 82.33 & 79.64 & 84.28 & 72.41 & 49.73 & 68.13 & **65.83** & **84.19** & 56.83 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance on the IIT vision language dataset. Bold text is utilized to highlight the best values achieved for mIoU and F-score among multimodal-based methods. Figure 6: Visual examples of our methods.