# Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog
## 3 Methodology
### Mgscrn
#### 3.2.1 Mgtr  
To collaboratively reason through the question and dialog history representations with different levels of granularity, the MGTR module adopts transformer-based attention mechanisms [22] and convolutional neural networks (CNN) to obtain the initial token- and sentence-granularity semantic representations. We first embed each token in the current question, \(Q_{t}\), as \(Q_{t}^{G}=\left[q_{t,1}^{G},q_{t,2}^{G},\ldots,q_{t,n_{Q}}^{G}\right]\), using a pretrained GloVe-embedding layer [23], where \(n_{Q}\) denotes the number of tokens in \(Q_{t}\). We then use a long short-term memory (LSTM) to encode \(Q_{t}^{G}\) into a token sequence, \(Q_{t}^{lstm}=\left[q_{t,1}^{lstm},q_{t,2}^{lstm},\ldots,q_{t,n_{Q}}^{lstm}\right]\). Next, we feed \(Q_{t}^{lstm}\) into a standard transformer attentive module (i.e., \(AM(\cdot)\)), which is carried out by a multi-head attention layer (i.e., \(MultiHead(\cdot)\)) and a position-wise fully connected feed-forward layer (i.e., \(FN(\cdot)\)). Taking query matrix \(Q\), key matrix \(K\), and value matrix \(V\) as input, \(AM(\cdot)\) is defined as follows:  
\[AM(Q,K,V)=FN(MultiHead(Q,K,V)), \tag{1}\]  
\[MultiHead(Q,K,V)=Concat(head_{1},head_{2},\ldots,head_{N_{dual}})W^{C}, \tag{2}\]  
\[head_{k}=Softmax\left(\frac{QW_{k}^{Q}(KW_{k}^{K})^{T}}{\sqrt{d}}\right)VW_{k }^{V}, \tag{3}\]  
Figure 3: Model architecture of the multi-granularity semantic collaborative reasoning network (MGSCRN) for visual dialog task. CNN = Convolutional neural network; LSTM = Long short-term memory; MGTR = Multi-granularity textual representation; QAAR = Question-aware attention refer; VAAA = Visual-aware attention alignment.  
where \(W^{C}\) and \(W^{*}_{t}\) are learned parameter matrices. By setting inputs \(Q\), \(K\), and \(V\) as \(Q^{lstm}_{t}\) in \(AM(\cdot)\), we obtain the token-granularity semantic representation of the question, \(Q^{self,token}_{t}=\left[\begin{subarray}{c}self,q^{self}_{t,1},q^{self}_{t,2},\ldots,q^{self}_{t,n_{Q}}\end{subarray}\right]\), after stacking \(L\) transformer blocks:  
\[Q^{self,token}_{t}=AM\Big{(}Q^{lstm}_{t},Q^{lstm}_{t},Q^{lstm}_{t}\Big{)}. \tag{4}\]  
To more accurately obtain a sentence-granularity semantic representation of \(Q^{self,s}_{t}\), we employ a CNN instead of average pooling the transformer or bidirectional encoder representations from transformers (BERT) token-embedding outputs, following the approach of [24]. We denote \(Q^{self,s}_{t}\) as follows:  
\[Q^{self,s}_{t}=CNN\Big{(}Q^{self,token}_{t}\Big{)}. \tag{5}\]  
Similarly, for the dialog history, \(H_{t}\), we first concatenate the image caption and \(t-1\) question-answer pairs, and then initialize them with the same embedding layer as in the question. Next, we utilize another LSTM and \(AM(\cdot)\) to acquire a token-granularity semantic representation of the dialog history, \(H^{self,token}_{t}=\left[\begin{subarray}{c}self,\ldots,h^{self}_{0,n_{H}},h^{ self}_{1,1},\ldots,h^{self}_{1,n_{H}},\ldots,h^{self}_{t-1,1},\\ \ldots,h^{self}_{t-1,n_{H}}\end{subarray}\right]\), where \(n_{H}\) denotes the number of tokens in each turn of the dialog history, \(H_{t}\). Likewise, the sentence-granularity semantic representation of the dialog history, \(H^{self,s}_{t}=\left[\begin{subarray}{c}self,s\\ 0\end{subarray},h^{self,s}_{1},\ldots,h^{self,s}_{t-1}\right]\), is obtained by utilizing Equation (5).