# Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
## 6 Ablation Study
### Similarity Aggregation  
We investigated the impacts of similarity aggregation using the average or the proposed weighted sum. We used dual-S and dual-I for this investigating experiment. The experimental results are shown in Table 4. There are improvements from the weighted sum at R@1, R@10, and MR in both of the dual-S and dual-I. Therefore, we confirmed the effectiveness of the proposed similarity aggregation module.  
Furthermore, we performed an analysis of the weights in the similarity aggregation. As described in Equation (12), the weights are flexibly determined according to the given sentence. In other words, the weight represents the importance of embedding spaces. We attempted to go further by analyzing the weights. For simplicity, we used the dual-S model in this analysis. Therefore, the analysis is on the importance of global and sequential visual features. We summarize the statistics of the weights in Table 5. The statistics show that the proposed method assigns larger weights to the global features.  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & & **R@1** & **R@5** & **R@10** & **MR** \\ \hline \multirow{2}{*}{dual-S} & average & 6.6 & 19.9 & 29.9 & 31 \\  & weighted & 7.1 & 19.9 & 31 & 30 \\ \hline \multirow{2}{*}{dual-I} & average & 6.7 & 20.2 & 30.4 & 30 \\  & weighted & 6.8 & 20.2 & 30.6 & 30 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison on similarity aggregation using the average or the proposed weighted sum.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **Global** & **Sequential** & **I3D** & **R@1** & **R@5** & **R@10** & **MR** \\ \hline single & \(\surd\) & & & 5.6 & 18.4 & 28.3 & 41 \\ dual-S & \(\surd\) & \(\surd\) & & 7.1 & 19.8 & 31 & 30 \\ dual-I & \(\surd\) & & \(\surd\) & 6.8 & 20.2 & 30.6 & 30 \\ triple & \(\surd\) & \(\surd\) & \(\surd\) & 6.7 & 21.2 & 32.4 & 29 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation on combinations of embedding spaces.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **R@1** & **R@5** & **R@10** & **MR** \\ \hline w/o attention & 5.8 & 19.8 & 28.6 & 34 \\ w/ attention & 7.1 & 19.8 & 31 & 30 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Effectiveness of the spatial attention.  
We show the accumulative step histogram for the global weight in Figure 4. The ratio reached 0.75 at the weight 0.5. Thus, 0.75 total instances received larger weights for the global feature. In contrast, the weights of the sequential feature are larger only in the remaining 0.25 instances. Therefore, the global feature is more critical than the sequential feature. Thus, dual-S aggressively used global features.  
Figure 5 shows examples of videos and sentences with assigned weights to the global feature. Videos containing clear scenes tend to have larger weights on the global visual feature since objects in the videos are relatively easy for detection. On the other hand, videos containing unclear scenes tend to assign larger weights to the sequential visual feature.  
Figure 4: Accumulative step histogram of the weights of the global visual features used in similarity aggregation.  
Figure 5: Examples of video and sentence with assigned weight to the global feature. The numbers and the sentences represent the weight for the global feature and query sentences, respectively.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Average** & **Min** & **Max** \\ \hline Global & 0.52 & 0.399 & 0.61 \\ Sequential & 0.48 & 0.393 & 0.60 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Statistics of the weights in similarity aggregation for the global and sequential embedding spaces.