# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 3 Method
### Task-Specific Context Extraction  
Task information, exemplified by actions and objects such as \"cup\" and \"contain\", plays a crucial role in grounding object parts for specific tasks. For instance, given two similar language instructions, \"Grasp the cup\" and \"Grasp the knife\", distinguishing the target object in the image is essential. Similarly, for instructions like \"Use the cup to contain\" and \"Grasp the cup\", the different actions lead to distinct affordance areas. Both action and object information are necessary for part-level affordance grounding. Therefore, this module is designed to extract a task-related textual context, which we denote as the task-specific context for part-level affordance grounding. The target object and responding action/verb are two important elements for the task, such as \"Use the knife to cut\" and \"Hand over the cup\". We conceptualize the task context as a pair of key elements: <object, action>. Drawing inspiration from event extraction (EE) [71] in natural language processing (NLP), we treat the task-specific context extraction problem akin to the event extraction problem, where the action serves as the event trigger and the object serves as the argument. Since there is no annotation for the trigger or the argument in the datasets, we use a soft classifier to predict the probability that each word belongs to one of three types (i.e., trigger, argument, or unnecessary words). Then, the task-specific context of the entire expression can be calculated from the probability of each word. **Word Feature Embedding:** By capturing the syntactic and semantic essence of each word, we can map words in the instruction to individual vectors using pre-trained BERT, as outlined in Section 3.2. After this step, the instruction is represented as a sequence of vectors, \(\{f_{1},f_{2},\ldots,f_{t}\}\). These word representations are then augmented with position embedding, \(p_{i}\), resulting in a composite feature for each word, \(w_{i}\), denoted by \([f_{i},p_{i}]\). **Graph-Based Feature Encoding:** Following word embedding, we transform the dependency tree of the instructions into an undirected syntactic graph. This graph effectively links words to their contextual counterparts, enhancing relational understanding. For instance, in the instruction \"Use the knife on the right to cut\", the action word \"cut\" and its corresponding object \"the knife\" are closely linked in the graph, despite being distant in the sequential representation. In the sequential representation, the two words are 5 steps apart, whereas in the dependency tree they are 2 steps apart (as shown in Figure 4). Figure 4: An example of a dependency tree for “Use the knife on the right to cut”. In the generated graph \(G=\{V,E\}\), each node, \(v\in V\), represents a word in the given instruction, while each directed edge, \(e\in E\), corresponds to a syntactic arc in the dependency tree (see Figure 4). To consider and learn the syntactic context for each word, we employ a graph neural network (GNN) [72] to aggregate information from the connected neighbors. After passing through \(L\) layers, the GNN outputs a representation for each word, \(f_{i}^{L}\). **Task-Specific Context Computing:** Each word's feature from the graph, denoted as \(f_{i}^{L}\), is passed through a multi-layer perceptron (MLP), resulting a soft probability vector, \(s_{i}\in\mathbb{R}^{1\times 3}\). Each vector, \(s_{i}\), indicates the likelihood of each word, \(w_{i}\), belonging to one of three categories: action, object, or unnecessary word. The task-specific context of the instruction, \(T\), is then computed using a weighted combination of all the words, as shown in Equation (7). \[f_{c}=\sum_{i=1}^{t}{(s_{i}^{1}\times h_{i}^{L}+s_{i}^{2}\times h_{i}^{L})}, \tag{7}\]  
where \(s_{i}^{1}\) and \(s_{i}^{2}\) represent the probabilities of the action and the object for each word, \(w_{i}\), respectively. \(h_{i}^{L}\) is the word embedding for the word, \(w_{i}\), which is encoded by GNN.