# Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog
## 5 Experimental Results and Analysis
### Ablation Study  
To evaluate the influence of our model's key components, we performed an ablation study using the VisDial v1.0 validation split and the same discriminative decoder for all variations:  
* Sentence-granularity semantic representations (SgSR): This model used only sentence-granularity semantic representations of the question and dialog history to acquire clues via the attentive module and CNN (Equations (4)-(6)). Then, it located question-related visual objects from \(V^{self}\) using the question-aware attention (Equation (17));
* Token-granularity semantic representations (TgSR): This model used only token-granularity semantic representations of the question and dialog history to acquire clues via the attentive module (Equations (4) and (7)). Then, it located question-related visual objects from \(V^{Q,token}\): we first use the object-difference attention mechanism (Equation (16)) to obtain distribution on all of the visual objects, \(P_{V}^{token}\); second, the final question-relevant visual representation is computed by \(v^{Q,token}=\sum\limits_{i=1}^{m}P_{V,i}^{token}V_{i}^{Q,token}\). The \(v^{Q,token}\)-fused question features is fed to the discriminative decoder to predict answers;
* TgSR without object-difference attention (TgSR w/o OdA): In this model, we performed self-attention instead of object-difference attention on \(V^{Q,token}\) to locate question-related visual objects. The other settings matched those of TgSR;
* SgSR+TgSR: This model further combined SgSR and TgSR. Specifically, in the discriminative decoder, two probability distributions over answer candidates from SgSR and TgSR were added after normalization to produce the final distribution. It notably did not include collaborative reasoning historical information (CRHI) and collaborative reasoning visual object (CRVO) operations;
* MGSCRN: This is our full semantic model, which combined SgSR and TgSR with CRHI and CRVO operations. Table 5 shows that both SgSR and TgSR considered only single-granularity semantic representations of the question and dialog history to reason the relational information among the question, dialog history, and visual objects. They achieved lower performance on all of the evaluation metrics compared with SgSR+TgSR. This illustrates the effectiveness of the joint decision-making that uses multi-granularity semantic information. Due to the introduction of CRHI and CRVO operations, MGSCRN bridged the multi-granularity semantic information to collaboratively reason the important information from the dialog \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **NDCG\(\uparrow\)** & **MRR\(\uparrow\)** & **R@1\(\uparrow\)** & **R@5\(\uparrow\)** & **R@10\(\uparrow\)** & **Mean\(\downarrow\)** \\ \hline
**VisPro dataset** & & & & & & \\
**MCA-I-HG** & 61.35 & 60.13 & 47.11 & 75.26 & 86.18 & 5.23 \\
**MCA-I-VGH** & 61.68 & 59.33 & 46.18 & 75.35 & 86.17 & 5.07 \\
**MCA-I-H** & 61.72 & 59.62 & 45.92 & 77.11 & 86.45 & 4.85 \\ \hline
**MGSCRN** & **62.57** & **61.56** & **48.73** & **78.06** & **88.11** & **4.70** \\ \hline
**VisDialConv dataset** & & & & & & \\
**MCA-I-HG** & 53.18 & **62.29** & 48.35 & 80.10 & 88.76 & 4.42 \\
**MCA-I-VGH** & 55.48 & 58.45 & 44.54 & 74.95 & 86.19 & 5.18 \\
**MCA-I-H** & 53.01 & 61.24 & 47.63 & 79.07 & 87.94 & 4.77 \\ \hline
**MGSCRN** & **57.13** & 61.87 & **48.84** & **80.44** & **88.98** & **4.22** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results comparison on VisPro and VisDialConv datasets. Better performance is indicated by “\(\uparrow^{\sigma}\) when the value is higher and by “\(\downarrow^{\sigma}\) when the value is lower. The best results are shown in bold. history and visual objects, and it performed best on all of the metrics. This validates the method's provision of more accurate clues by leveraging a collaborative interaction strategy with multi-granularity semantic information.