# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 1 Introduction  
The ability to comprehend and execute human instructions in natural language is important for the functionality of embodied agents. Once agents gain this ability, they can engage in natural and intuitive communication with humans. This ability necessitates an in-depth understanding of both natural language processing and visual content analysis. To address this challenge, a significant body of research has emerged, encompassing areas such as vision and language navigation [1, 2, 3], instruction-following manipulation [4], and visual grounding [5]. Despite considerable progress in various related research areas, formidable challenges, particularly in complex, real-world scenarios, remain in the domain of instruction-following manipulation. Most existing methods predominantly focus on object-level vision-language comprehension, which limits their ability to achieve fine-grained, part-level comprehension. Object-level grounding enables agents to identify the target object, such as \"the glass on the right\". However, agents fail to consider the specific functional parts of the object that are necessary to complete the task. For example, the instruction \"Hand over the glass on the right\", as depicted in Figure 1, requires the agent to understand not only the object itself (the glass on the right) but also the specific interaction (\"hand over\" involves the body of the glass in the green area of Figure 1). Conversely, for the instruction \"Please pour some water into the right glass\", the agent needs to associate the mouth area of the right glass (the pink area in Figure 1) with the task of \"pouring water into\". Object-levelcomprehension methods [6; 7; 8; 9; 10; 11] excel at locating objects but struggle to discern part-level differences for the same object. While object-level grounding is sufficient for basic manipulation tasks such as \"pick up\", \"hand over\", and \"move\", it is inadequate for more complex actions like 'use the knife to cut\". These actions require semantic grasping [12]--a nuanced comprehension of the object's different parts and their associated functionalities or \"_affordance_\", a term coined by psychologist [PERSON] [13]. In robotic manipulation, understanding affordances at a granular level is crucial for determining the exact location for task-specific actions within a real-world context. Interpreting directives such as \"Hand over the glass\" or \"Pour water into the glass\" requires an advancement beyond traditional object-level grounding techniques. Existing research typically concentrates on either tracking bounding boxes with language cues [6; 7; 8; 9; 10; 11] or segmenting object areas without considering the semantic context of tasks [14; 15; 16], as depicted at the bottom of Figure 1. Such methods, while effective for object detection and instance segmentation, fall short in their granularity, failing to meet the demands of manipulation-oriented robotic interactions. Current affordance detection methods can identify parts of objects, but they are confined to simplistic, pre-defined affordance categories and do not factor in the task semantics. This necessitates additional pre-processing steps for language interpretation and object localization prior to affordance segmentation. Our work seeks to fill this gap, providing a direct and nuanced understanding of instructions for fine-grained affordance segmentation that aligns with the complexities of real-world applications. To overcome this challenge, we explore the problem of part-level affordance grounding based on natural language instructions. Specifically, we address the problem of segmenting the relevant functional region of a referenced object when given an image and an expression describing a manipulation instruction.