# Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog
## 5 Experimental Results and Analysis
### Qualitative Analysis  
To further demonstrate the good interpretability of our model, we visualized the attention weights generated by QAAR and VAAA modules through the examples illustrated in Figure 4. For the QAAR module, two types of attention weights were shown. The first was attention weights \(P^{s}\) for \(t\) rounds of dialog history, obtained by modifying \(P^{s}_{his}\) with the assistance of \(P^{Q}_{his}\) (as in Equation (12)). Doing so revealed the relevance of different dialog history rounds. The second was \(P^{token}_{i,j}\) over all tokens in the \(j^{th}\) (selected by \(P^{s}\)) round of dialog history based on their relationship to the current question, where \(P^{token}_{i,j}=Norm\big{(}P^{token}[i,jn_{H}:(j+1)n_{H}]\big{)}\). That is, \(P^{token}_{i,j}\) reveals the relevance of different tokens in the \(j^{th}\) round of dialog history to the \(i^{th}\) token in the current question. Importantly, for both types of attention, we fetched the average attention weight over all heads computed by the topmost stacked transformer block. For the VAAA module, we first visualized the object-difference attention (\(P^{token}_{V}\) in Equation (16)) and question-aware attention (\(P^{s}_{V}\) in Equation (17)), and visualized the collaborative reasoning attention (\(P^{token+s}_{V}\) in Equation (18)) computed by CRVO. Here, we display the top three ranked visual objects for each attention in the rightmost column of Figure 4.  
Two QAAR visualization examples are presented, for which our model accurately captured question-aware historical information at both sentence- and token-granularities. In the top example, when answering the current question, \"Is he wearing shorts?\", to determine what \"he\" refers to, the model assigned a maximum weight to the \"H1\" round of dialog history (i.e., \"Is the young boy a toddler? No.\"), in which the token, \"boy\", gained a relatively large weight by visualizing the weight of each token in \"H1\", given by the token \"he\" in the question. This observation validates that our model accurately obtains clues from historical information, thus clarifying the semantics of the question. In the VAAA module's visualization, we observed that the question-related visual objects were located more accurately, benefiting from collaboratively reasoning the distribution over all of the visual objects using multi-granularity semantic representations of the question. For instance, when answering \"Is there anything around the laptop?\", the object-difference attention of the VAAA module grounded \"cat,\" \"screen\", and \"pens\" as the top three target objects. However, \"cat\" and \"screen\" should not have been considered based on the complete semantics of the question. However, VAAA's question-aware attention located all of the visual objects around the laptop, including \"table lamp\", \"keyboard\", and \"pens.\" Although the OdA did not produce an accurate distribution over the visual objects, the CRVO method that combined OdA and QaA determined a suitable distribution that still ranked \"table lamp\", \"keyboard\", and \"pens\" as the top three target objects. Similar phenomena were verified in the top example. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **NDCG\(\uparrow\)** & **MRR\(\uparrow\)** & **R@1\(\uparrow\)** & **R@5\(\uparrow\)** & **R@10\(\uparrow\)** & **Mean\(\downarrow\)** \\ \hline SgSR & 59.85 & 63.65 & 50.88 & 80.76 & 89.20 & 4.37 \\ TgSR & 60.49 & 64.22 & 51.59 & 81.46 & 90.01 & 4.19 \\ TgSR w/o OdA & 59.58 & 63.45 & 50.93 & 80.80 & 89.38 & 4.32 \\ SgSR+TgSR & 61.25 & 64.98 & 52.68 & 81.92 & 90.46 & 4.01 \\ \hline
**MGSCRN** & **62.28** & **66.44** & **53.42** & **82.80** & **91.21** & **3.83** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Ablation study of MGSCRN using the VisDial v1.0 validation split. Better performance is indicated by “\(\uparrow\)” when the value is higher and by “\(\downarrow\)” when the value is lower. The best results are shown in bold.