# Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog
## 3 Methodology
### Mgscrn
#### 3.2.2 Qaar  
Differing from most of the extant works [4; 6; 17; 19] that rely only on single-granularity representations to construct relationships between the question and dialog history, QAAR uses multi-granularity representations to collaboratively capture relevant historical information. To achieve this goal, the question guides the dialog history using sentence- and token-granularity semantic representations to fetch the history features relevant to the question. Two attention-weight distributions over the dialog history are computed simultaneously: all tokens and all rounds. Then, the distributions correct each other to accurately align the features of the question and dialog history items. To this end, we formally describe the QAAR module, which adopts \(AM(\cdot)\) to learn the correlated history features while considering multi-granularity information. The equations are as follows:  
\[Q^{his,s}_{t}=AM\Big{(}Q^{self,s}_{t},H^{self,s}_{t},H^{self,s}_{t}\Big{)}, \tag{6}\]  
\[Q^{his,token}_{t}=AM\Big{(}Q^{self,token}_{t},H^{self,token}_{t},H^{self,token} _{t}\Big{)}, \tag{7}\]  
where \(Q^{his,s}_{t}\) and \(Q^{his,token}_{t}\), respectively, represent the question with fused sentence- and token-granularity. When learning the representations of \(Q^{his,s}_{t}\) and \(Q^{his,token}_{t}\), the Soft-Max function existing in \(AM(\cdot)\) (i.e., Equations (6) and (7)) produce two types of distributions over the dialog history. Here, we take these out and denote them as \(P^{s}_{his}=\left[ \begin{subarray}{c}p^{s}_{0},p^{s}_{1},\ldots,p^{s}_{t-1} \end{subarray} \right]\in\mathcal{R}^{t}\) and \(P^{token}_{his}=\left\{\left[ \begin{subarray}{c}p^{i,token}_{0,1},\ldots,p^{ i,token}_{0,n_{H}},p^{i,token}_{1,1},\ldots,p^{i,token}_{1,n_{H}},\ldots,p^{i,token}_{t-1,1}, \ldots,\\ p^{i,token}_{t-1,n_{H}}\end{subarray} \right]\right\}_{i=1}^{n_{Q}}\in\mathcal{ R}^{n_{Q}\times(t\times n_{H})}\), respectively:  
\[P^{s}_{his}=Softmax\left(\frac{Q^{self,s}_{t}W^{Q^{s}}_{k}\left(H^{self,s}_{t} W^{Hs}_{k}\right)^{T}}{\sqrt{d}}\right), \tag{8}\]\[P_{his}^{token}=Softmax\left(\frac{Q_{t}^{his,token}W_{k}^{Q^{token}}\left(H_{t}^{ self,token}W_{k}^{H^{token}}\right)^{T}}{\sqrt{d}}\right), \tag{9}\]  
where \(P_{his}^{s}\) represents the attention weights on \(t\) rounds of dialog history, and the \(i^{th}\) row of \(P_{his}^{token}\) represents the attention weights on all of the tokens that possibly have dependencies to the \(i^{th}\) token in the question. To better explore the question-related tokens and sentences in the dialog history, we dynamically modify and update \(P_{his}^{s}\) and \(P_{his}^{token}\) under their mutual guidance during training. To make the deterministic distribution, \(P_{his}^{s}\), more accurate, we first deal with the \(i^{th}\) row of \(P_{his}^{token}\) by separately adding the weight values of tokens in the different rounds of the dialog history, which is defined as:  
\[p_{his}^{i,token}=\left\{\sum_{k=1}^{n_{H}}p_{j,k}^{i,token}\right\}_{j=0}^{t- 1}\in\mathcal{R}^{t}, \tag{10}\]  
where \(p_{his}^{i,token}\) represents the different contributions of \(t\) rounds of dialog history to the \(i^{th}\) token in the question. Thus, \(P_{his}^{Q,token}=\left[p_{his}^{1,token},p_{his}^{2,token},\ldots,p_{his}^{ n_{Q,token}}\right]^{T}\in\mathcal{R}^{n_{Q}\cdot t}\) indicates the different contributions of \(t\) rounds of dialog history to each token in the question. Then, we add the weight values of each column of \(P_{his}^{Q,token}\) to obtain \(P_{his}^{Q}\), followed by normalization:  
\[P_{his}^{Q}=Norm\left(\sum_{k=1}^{n_{Q}}P_{his,k}^{Q,token}\right)\in\mathcal{ R}^{t}, \tag{11}\]  
where \(P_{his}^{Q}\) represents the contribution of \(t\) rounds of dialog history to the whole question from the perspective of aggregating token-granularity semantic information. Next, as shown in Equation (12), we modify \(P_{his}^{s}\) with the help of \(P_{his}^{Q}\):  
\[P^{s}=Norm\left(P_{his}^{s}\odot P_{his}^{Q}\right)\in\mathcal{R}^{t} \tag{12}\]  
where \(\odot\) denotes the elementwise product.