# Deep Reinforcement Learning for Query-Conditioned Video Summarization
## 4 Experiments
### Ablation Analysis  
#### 4.3.1 The Effect of the Proposed Mapping Mechanism  
We analyze the effect of the mapping mechanism by dropping the MapNet network in our method and compare the performance to the full model. Note that the MapNet is used to express the relatedness reward, and that dropping MapNet leads to removing the relatedness reward in our model. The _W/O MapNet_ model is therefore only trained with two rewards: representativeness and diversity rewards. The results of this comparison are shown in Table 2. We can see that the combined proposed model outperforms the model without MapNet (_W/O MapNet_) on all four videos and that the average performance decreases by 3.29% when removing the mapping mechanism of MapNet. This indicates that the mapping mechanism allows the model to incorporate the connection between the visual features and the text features by learning a transformation between the two modalities. It also shows that it is able to exploit this transformation to provide a good reward signal for the summarization model in order to learn query-conditioned video summaries. Despite of the fact that the ground-truth labels are included in the summarization model as part of the supervised loss and should enable the model to learn the relation between the visual and the query information, it appears that the proposed relatedness reward of MapNet enhances the robustness of the results.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**SeqDPP [30]**} & \multicolumn{3}{c}{**SH-DPP [12]**} & \multicolumn{3}{c}{**QC-DPP [13]**} & \multicolumn{3}{c}{**QueryGAN [36]**} & \multicolumn{3}{c}{**Ours**} \\ \cline{2-13}  & **Pre** & **Rec** & **F** & **Pre** & **Rec** & **F** & **Pre** & **Rec** & **F** & **Pre** & **Rec** & **F** & **Pre** & **Rec** & **F** \\ \hline Video1 & 53.43 & 29.81 & 36.59 & 50.56 & 29.64 & 35.67 & 49.86 & 53.38 & 48.68 & 49.66 & 50.91 & 48.74 & 50.81 & 51.57 & **48.77** \\ Video2 & 44.05 & 46.65 & 43.67 & 42.13 & 46.81 & 42.72 & 33.71 & 62.09 & 41.66 & 43.02 & 48.73 & 45.30 & 39.99 & 58.51 & **46.65** \\ Video3 & 49.25 & 17.44 & 25.26 & 51.92 & 29.24 & 36.51 & 55.16 & 62.40 & 56.47 & 58.73 & 56.49 & 56.51 & 63.48 & 54.91 & **57.01** \\ Video4 & 11.14 & 63.49 & 18.15 & 11.51 & 62.88 & 18.62 & 21.39 & 63.12 & 29.96 & 36.70 & 35.96 & 33.64 & 39.03 & 38.37 & **36.37** \\ \hline Avg. & 39.47 & 39.35 & 30.92 & 39.03 & 42.14 & 33.38 & 40.03 & 60.25 & 44.19 & 47.03 & 48.02 & 46.05 & 48.33 & 50.84 & **47.20** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of our proposed algorithm to the state-of-the-art methods for query-conditioned video summarization in terms of Precision (Pre), Recall (Rec) and F-measure (F). The best results (F-measure) are highlighted in bold.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**W/O MapNet**} & \multicolumn{3}{c}{**W/O \(\mathcal{L}_{summ}\)**} & \multicolumn{3}{c}{**Ours**} \\ \cline{2-13}  & **Pre** & **Rec** & **F** & **Pre** & **Rec** & **F** & **Pre** & **Rec** & **F** \\ \hline Video1 & 49.88 & 44.26 & 43.37 & 30.45 & 64.36 & 40.20 & 50.81 & 51.57 & **48.77** \\ Video2 & 44.17 & 45.49 & 43.54 & 42.01 & 37.78 & 39.50 & 39.99 & 58.51 & **46.65** \\ Video3 & 61.89 & 52.95 & 55.45 & 39.72 & 74.92 & 51.63 & 63.48 & 54.91 & **57.01** \\ Video4 & 28.19 & 48.13 & 33.28 & 36.28 & 29.40 & 32.12 & 39.03 & 38.37 & **36.37** \\ \hline Avg. & 46.03 & 47.71 & 43.91 & 37.12 & 51.62 & 40.86 & 48.33 & 50.84 & **47.20** \\ \hline \hline \end{tabular}
\end{table}
Table 2: We compare the proposed model to two models, one that does not make use of MapNet, and one that does not make use of the ground-truth loss \(\mathcal{L}_{summ}\), but that are identical otherwise. The best results (F-measure) are highlighted in bold.