# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 1 Introduction
In this paper, we introduce the Instruction-following Affordance Grounding Network (IAG-Net), targeting the nuanced task of part-level affordance grounding based on natural language instructions, as illustrated in the upper right of Figure 1. Unlike previous methods that rely on coarse object-level prediction and fixed task representations, our approach represents a significant advancement, achieving an end-to-end understanding of manipulation instructions for precise, fine-grained affordance grounding within images. We introduce a varied-scale multimodal attention mechanism for  
Figure 1: Comparison of object-level grounding and part-level affordance grounding. The red boxes indicates the detected object bounding boxes. Additionally, the pink and green colors in the masks represent two affordance categories: contain and w-grasp. efficient feature fusion and extract task-specific contexts for precise affordance prediction. Our approach bridges the gap between language instructions and visual cues, facilitating a more nuanced interaction between humans and robots--beyond the capabilities of current object recognition systems. In addition, different from several works using simulated or fixed environment settings [8; 11], we establish an instruction image dataset with realistic images and semantic grasping tasks. Affordance grounding from natural language expressions has broad applications, including human-robot communication and natural language-facilitated human-robot interaction [17]. For instance, when faced with different instructions for the same object, such as \"Help me move the table\" and \"Help me clean the table\", a robot should distinguish the relevant object regions and act appropriately. The part-level affordance grounding can enhance human-robot interaction capabilities and bolster the autonomy of robots in decision-making processes. In the context of dialog response generation, natural language-based affordance grounding can provide specific cues for predicting future human actions and understanding the semantics of manipulation tasks. Compared to one-shot manipulation task representation, natural language offers flexibility for various tasks, such as \"Pour the milk\" and \"Hand over the milk box\". Therefore, this research has the potential to advance the field of robotics by empowering robots with enhanced interaction and decision-making abilities, thus allowing them to operate more autonomously and effectively [18; 19] in dynamic environments or under remote control. Moreover, by integrating robust security measures and defenses against potential cyberattacks [20; 21], this work contributes to enhancing reliability and trustworthiness in practical applications. In a brief overview, our contributions consist of the following:  
1. We introduce an end-to-end framework for instruction-following part-level affordance grounding that streamlines the process and enables the development of more efficient robotics applications. 2. A realistic instruction dataset for instruction-following affordance grounding is used. Our multimodal dataset, featuring realistic images paired with manipulation instruction expressions, offers a more comprehensive and practical resource. 3. We introduce two language-vision feature fusion strategies. By leveraging both sentence-level and task-specific textual features, our approach incorporates a soft event detection strategy in the task-information extraction module, enhancing semantic information guidance for part-level affordance grounding. The remainder of this paper is structured as follows. Section 2 presents existing works related to our work. In Section 3, we delineate the architecture and three components of our IAG-Net. Section 4 discusses the experimental results and analysis conducted on two datasets. We conclude the paper and discuss potential applications in Section 5.