# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 4 Experiments and Evaluation
### Comparison Results (Rq1)
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Affordance**} & \multicolumn{6}{c}{**Image Only**} & \multicolumn{6}{c}{**Multimodal-Based**} \\ \cline{2-13}  & **DeepLabV3+** & **OCRNet** & **AffordanceNet** & **Cascaded** & **BKINet** & **LAG-Net (Ours)** \\ \cline{2-13}  & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) & **mIoU** & \(F_{P}^{\omega}\) \\ \hline contain & 85.80 & 81.27 & 85.40 & 81.72 & 59.75 & 86.11 & 83.38 & 79.03 & 47.81 & 85.24 & 82.02 \\ cut & 82.3 & 78.6 & 82.76 & 79.42 & 64.26 & 82.07 & 80.62 & 42.32 & 43.65 & 78.34 & 79.89 \\ grasp & 69.5 & 80.03 & 71.02 & 80.54 & 65.97 & 71.39 & 78.70 & 75.98 & 51.80 & 76.24 & 81.08 \\ pound & 85.42 & 81.29 & 85.61 & 81.70 & 76.73 & 85.20 & 82.08 & 61.45 & 67.17 & 85.13 & 83.14 \\ scoop & 85.57 & 79.74 & 85.92 & 80.65 & 77.0 & 85.79 & 81.65 & 70.84 & 86.83 & 86.34 & 81.22 \\ support & 83.77 & 81.11 & 85.03 & 82.04 & 79.45 & 85.86 & 81.34 & 89.37 & 90.47 & 85.20 & 81.41 \\ w-grasp & 80.80 & 76.04 & 81.01 & 77.06 & 78.65 & 78.97 & 73.00 & 86.46 & 88.08 & 81.81 & 75.45 \\ Average & 81.88 & 79.73 & 82.39 & 80.45 & 71.6 & 82.19 & 80.11 & 72.20 & 67.97 & **82.61** & **80.60** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance on the UMD vision language dataset. Bold text is utilized to highlight the best values achieved for mIoU and F-score among multimodal-based methods. **Statistical Analysis:** To evaluate the statistical significance of the performance disparities among multimodal-based approaches, we undertook a statistical analysis focusing on the mean IoU and F-score values across two datasets. The results revealed a statistically significant difference in segmentation performance among the three methods. In Figure 7, we present a visualization depicting the mean IoU and F-score values alongside their respective confidence intervals for two datasets. These confidence intervals were calculated using the t-distribution, assuming a 95% confidence level and considering the sample size. The inclusion of confidence intervals in our visualization effectively displays the uncertainty associated with these performance estimates. Notably, our method is distinguished by its tighter confidence intervals for the mean IoU across both datasets, indicating a more uniform performance across various affordance categories and underscoring the robustness of our approach. Nevertheless it is important to note that there remains room for improvement in terms of the F-score performance of our method. While our method shows promising results in terms of mean IoU, it requires further refinement to improve F-score outcome. **Model Efficiency Analysis:** In terms of model efficiency, IAG-Net has a more streamlined architecture with fewer parameters than the cascaded method, as illustrated in Table 5. This efficiency stems from IAG-Net's integrated approach, in contrast to the cascaded method's separate training for visual grounding and affordance segmentation, which results in a larger parameter set and lower FPS. Compared to image-only methods, IAG-Net achieves a similar balance of FPS and parameters, indicating its efficiency in handling both visual and language data without significant resource overhead. BKINet achieves the highest FPS, partly because of the parallel computing of the architecture. **Failure Cases:** Our analysis of failure cases (as shown in Figure 8), particularly in complex scenes from the IIT-AFF dataset, reveals that IAG-Net sometimes struggles with images containing multiple objects matching the instructions, leading to partial target recognition. Occlusions present another challenge, affecting the model's performance.