# Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog
## 3 Methodology
### Mgscrn
#### 3.2.3 VAAA  
Given the output representations of the QAAR module, \(Q_{t}^{his,s}\) and \(Q_{t}^{his,token}\), the VAAA module collaboratively aligns them with the visual features. Here, the initial features are extracted by employing the Faster-CNN [25], pretrained using Visual Genome [26], denotedas \(V^{0}\in\mathcal{R}^{m\times d_{v}}\), where \(m\) and \(d_{v}\) represent the overall number of visual objects in image \(I\) and the dimension size of each visual object, respectively. We use a linear projection to map the dimensions of the object embeddings to those of the token. Then, the visual features are denoted as \(V\in\mathcal{R}^{m\times d}\). We first establish the latent connections among visual objects in image \(I\) via the \(AM(\cdot)\):  
\[V^{self}=AM(V,V,V)\in\mathcal{R}^{m\times d}. \tag{14}\]  
Then, we apply the visual-aware attention mechanism to align visual objects with the visually related tokens from the question. The visual representation, \(V^{self}\), queries the token-granularity semantic representation, \(Q_{t}^{his,token}\), to obtain a new visual representation, \(V^{Q,token}\), that fuses the attentive token-granularity question features. We utilize cross-attention to compute \(V^{Q,token}\) as follows:  
\[V^{Q,token}=AM\Big{(}V^{self},Q_{t}^{his,token},Q_{t}^{his,token}\Big{)}\in \mathcal{R}^{m\times d}. \tag{15}\]  
However, the visual features, \(V^{Q,token}\), are obtained by attending to the most relevant question-token features, which are not target visual features for adequately answering \(Q_{t}\), then, an object-difference attention mechanism is used to locate the question-relevant visual objects in the corresponding image, formulated as:  
\[P_{V}^{Q,token}=Softmax\Big{(}\Big{[}V^{Q,token}\odot\Big{(}V^{Q,token}-V^{ self}\Big{)}\Big{]}W_{V}^{token}\Big{)}^{T}\in\mathcal{R}^{g\times m}, \tag{16}\]  
where \(V^{Q,token}\odot\Big{(}V^{Q,token}-V^{self}\Big{)}\in\mathcal{R}^{m\times d}\) is the difference between \(V^{Q,token}\) and \(V^{self}\) representations as guided by \(V^{Q,token}\). \(W_{V}^{token}\in\mathcal{R}^{d\times g}\) is the parameter matrix to be learned with \(g\) glimpses. The SoftMax function transforms \(g\) glimpse results into an attention weight over all visual objects. The final distribution on all of the visual objects is \(P_{V}^{token}=1/g\sum_{i=1}^{g}P_{V}^{token}\).  
\(P_{V}^{token}\) is obtained by finding clues from question tokens and comparing visual object differences. Note that it may inaccurately identify relevant visual objects, depending on the token semantically granular representation of the question. Therefore, we also use the sentence-granularity semantic representation of \(Q_{t}^{his,s}\) to align it with \(V^{Q,token}\), which is implemented by the question-aware attention:  
\[P_{V}^{s}=Softmax\left(\frac{Q_{t}^{his,s}W_{k}^{Q_{t}^{his,s}}\Big{(}V^{Q,token }W_{k}^{V}\Big{)}^{T}}{\sqrt{d}}\right). \tag{17}\]  
The obtained \(P_{V}^{token}\) and \(P_{V}^{s}\) jointly decide an appropriate distribution, \(P_{V}^{token+s}\), over the visual objects as follows:  
\[P_{V}^{token+s}=Norm\Big{(}P_{V}^{token}+P_{V}^{s}\Big{)}. \tag{18}\]  
Based on distribution \(P_{V}^{token+s}\), the weighted visual object representation is formulated as follows:  
\[v^{Q,token}=\sum_{i=1}^{m}P_{V,i}^{token+s}V_{i}^{Q,token}. \tag{19}\]  
Finally, we fuse the filtered visual feature, \(v^{Q,token}\), with \(Q_{t}^{his,s}\) (expressing the complete semantic intent of the question) as the final multimodal fused representation:  
\[C_{t}=tanh\Big{(}W_{f}\Big{[}v^{Q,token};Q_{t}^{his,s}\Big{]}\Big{)}, \tag{20}\]where \([j]\) denotes the concatenation, yielding context representation \(C_{t}\), which is then fed to the discriminative decoder to predict the answer to the current question, \(Q_{t}\).