# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 4 Experiments and Evaluation
### Dataset Construction  
This task necessitates paired data, \(<T,I,M>\), comprising natural language instructions, \(T\), corresponding RGB images, \(I\), and pixel-level part affordance masks, \(M\). As no existing datasets fulfill these requirements, we constructed a new affordance-grounding dataset ([[https://github.com/WenQu-NEU/Affordance-Grounding-Dataset](https://github.com/WenQu-NEU/Affordance-Grounding-Dataset)]([https://github.com/WenQu-NEU/Affordance-Grounding-Dataset](https://github.com/WenQu-NEU/Affordance-Grounding-Dataset)) accessed on 30 March 2024) with the following methodology:  
1. Image Selection: The first step involves selecting an image, \(I\), with existing affordance annotations from two widely recognized computer vision datasets: the IIT-AFF dataset ([PERSON] et al., 2018) and the UMD dataset ([PERSON] et al., 2018). We detail the specific characteristics and selection criteria for each dataset below. 2. Language Instruction Labeling: After image selection, we manually added instructions, \(T\), for each image, \(I\). 3. Affordance Mask Generation: We generated the affordance mask, \(M\), based on the image annotations corresponding to the given instructions, \(T\). **Image Selection:** In computer vision, there are two widely used datasets with pixel ground-truth affordance labels for objects, i.e., the IIT-AFF dataset ([PERSON] et al., 2018) and the UMD dataset ([PERSON] et al., 2018). The IIT-AFF dataset ([[https://sites.google.com/site/iitaffdataset/](https://sites.google.com/site/iitaffdataset/)]([https://sites.google.com/site/iitaffdataset/](https://sites.google.com/site/iitaffdataset/))) has 8835 real-world images with cluttered scenes. This dataset contains 10 object categories: bowl, tv, pan, hammer, knife, cup, drill, racket, spatula, and bottle, and 9 affordance classes: contain, cut, display, engine, grasp, hit, pound, support, and w-grasp. In contrast, the images in the UMD dataset ([[https://users.umiacs.umid.edu/~fer/affordance/part](https://users.umiacs.umid.edu/~fer/affordance/part)]([https://users.umiacs.umid.edu/~fer/affordance/part](https://users.umiacs.umid.edu/~fer/affordance/part))affordance-dataset/index.html) are objects on a rotating table with a clutter-free setup. These images were captured by a Kinect camera. We utilized the IIT-AFF and UMD datasets for their extensive annotations and the affordance categories they offer, which align with our objective to examine object affordances in diverse scenarios. To ensure the UMD dataset's variety and to avoid redundancy, we implemented a filtering process to remove highly similar images. By computing pairwise image similarity, any pair with a similarity score exceeding the threshold of 0.9 had one image randomly removed. This curation resulted in a refined UMD dataset consisting of 7070 unique images for our experiments. Consequently, our enhanced dataset now comprised 14,642 object bounding boxes and 24,677 pixel-level affordance regions, making it a robust and comprehensive resource for affordance analysis. **Language Instruction Labeling:** To create a semantic link between functional areas and language instructions, we manually crafted language instructions for each image, adhering to standardized guidelines to ensure consistency, relevance, and diversity across annotations. Two volunteers were tasked with generating instructions based on the content of each image. Each instruction was meticulously designed to encompass three critical elements: the object, the action to be performed, and the affordance related to the action, such as \"Use the cup to contain\". To ensure accuracy and consistency, each volunteer worked independently, followed by a review process where any discrepancies between their instructions were discussed and resolved. This process resulted in 8835 and 7070 expressions for the IIT-AFF and UMD datasets, respectively. To enrich the diversity of our language instructions, we specifically focused on varying the expressions used for objects with multiple affordances.