# Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding
## 3 Method
### Multiscale Vision Language Encoder  
A multimodal learning study [68] demonstrated the significance of high-level visual features, especially when language terms are directly related to objects such as \"Hand over the cup on the right of the desk\". This insight serves as a guiding principle for our approach, where we harness high-level visual features and sentence-level textual features to capture object-level vision and language attributes effectively. The multiscale visual language encoder is designed to extract object-level multimodal features for affordance grounding. We obtain the global sentence representation by concatenating the representations of all words in the sentence, as expressed in the following equation:  
\[F_{gt}=[f_{1},\ldots,f_{t}], \tag{2}\]  
where \([,]\) represents the concatenation operation. Initially, we transform both the textual and visual features into a uniform feature dimension using two separate linear layers. Subsequently, these features are fused, as shown in Equation (3). \[F_{m}=[g(F_{v}W_{v}),g(F_{gt}W_{t})], \tag{3}\]  
where \(W_{v}\) and \(W_{t}\) are two learnable parameter matrices and \(g\) denotes the activation function Relu. The concatenation operation, \([,]\), is employed to combine features from two different modalities. The visual feature, \(F_{v}\), is extracted from the visual backbone, specifically using the visual feature \(Res_{4}\). The textual feature, \(F_{gt}\), is computed according to Equation (2). The multimodal feature, \(F_{nn}\), results from fusing the sentence-level semantic and high-level visual feature, providing a global multimodal representation. In order to enhance the fusion of visual-linguistic features, we propose a varied-scale multimodal attention (VSMA) mechanism designed to effectively capture cross-modal features. The incorporation of multiscale visual features has demonstrated efficacy in image segmentation, particularly in handling objects of varied scales. Additionally, cross-modal attention mechanisms guide the network to focus on image regions relevant to the language context, thereby enhancing performance in vision-language tasks. Motivated by these insights, we introduce the VSMA mechanism, which integrate a multiscale structure withcross-modal attention to refine the features and concentrate on the essential multimodal features of the target object. The structure of the VSMA mechanism is depicted in Figure 3. Specifically, VSMA comprises channel attention, spatial attention at various scales, and multimodal attention. For channel attention, we compute the attention weights from the average pooling and max pooling of the multimodal feature, \(F_{m}\), following channel attention in the CBAM [69]. Since various scales have been efficient for various image processing tasks in previous works such as ASPP [70], we introduce varied-scale spatial attention to improve the robustness of the affordance parts in the images. For the feature of modality, \(m\), we first compute the max pooling and average pooling features as \(F_{max}^{m}\) and \(F_{avg}^{m}\), respectively. Next, atrous convolutional layers [70] with different sampling rates, \(Acon_{s}\), are employed to capture varied-scale spatial information, the results of which are concatenated. Then, a convolution operation is applied to obtain the varied-scale spatial attention weights. \[VSA(F_{m})=Conv(Acon_{s}([F_{max}^{m},F_{avg}^{m}])F_{m}, \tag{4}\]  
where \(Acon_{s}\) is the atrous convolutional layer for scale \(s\), as in ASPP, and \(Conv\) is the convolution layer. In addition to spatial attention, multimodal attention is useful for determining the role of each modality in the affordance-grounding tasks.