# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 2 Related Work
### Visual Grounding  
As a fundamental multimodal task, visual grounding mainly involves vision and language comprehension and aims to localize the referent object in an image according to the given natural language. This task has gained much attention in both the natural language and computer vision community. Typical visual grounding tasks include referring expression comprehension (REC) and referring image segmentation (RIS).  
REC focuses on locating the most related objects in an image based on a given referring expression [36]. In this task, it is assumed that the referring expression unambiguously describes an object. The typical two-stage methods [37; 38; 39] first generate several candidate regions via a pretraining proposal framework, e.g., Faster-RCNN [40] or Mask-RCNN [41]. Afterward, the candidate regions and expression map are embedded in a common space, and joint embedding is used to maximize the match score of region-expression pairs. However, whole-sentence embedding may fail to consider fine-grained details that are crucial for accurate localization. Compositional modular networks [37] first identify the entities and their relationships in the input expression and then implement end-to-end visual reasoning. MAttNet [38] uses Bi-LSTM to parse an expression into subject, location, and relationship representations and computes individual match scores. However, these two-stage methods tend to be restricted by the generated proposal quality in the first stage. More recently, one-stage methods [42; 43] eliminate the offline object detector and directly output the target bounding boxes of the referring expression. Following one-stage methods, several researchers [44; 45; 46] have explored end-to-end frameworks based on transformers.  
Similar to the REC task, the RIS task aims to segment the referring object in an image that matches the corresponding referring expression [47]. In early works [48; 49], textual features and image features were usually fused together and then the label for each pixel was predicted following classical image segmentation methods. To improve the performance of feature fusion, many methods have utilized attention mechanisms in image segmentation. [PERSON] et al. [50] proposed a key-word-aware network to evaluate the importance of words according to their contribution to identifying an object. [PERSON] et al. [51] considered the long-range correlations between text and vision modalities. Since most approaches usually perform multimodal fusion between language and vision at the decoding side, the EFN [52] employs a coattention mechanism to achieve simultaneous updates of multimodal features. Due to the excellent performance of transformers [53] based on self-attention mechanisms, many researchers [54; 55; 56] have extracted and fused vision and language features with self-attention mechanisms.  
Although the above methods also input a picture and an expression and output the target object guided by the expression, they are different from our method in two aspects. First, the referring expression and user instructions are different in content. While referring expressions describe the referent object within an image (e.g., \"the man on the left of the tree\"), manipulation instructions consist of objects and tasks (e.g., \"Use the knife to cut\"). On the other hand, the above methods achieve object-level location or segmentation. These methods do not adequately address the part-level granularity required for complex manipulation tasks.