## 6 Video Saliency
### Evaluation of Video Saliency  
The analyzed methods for video saliency estimation introduce two domain-specific evaluation measures, namely the Temporal stability (\(\mathcal{T}\)) [125], and the Per-frame pixel error rate (\(\epsilon\)) [121]. Both are based on a salient object region ground truth. Temporal stability \(\mathcal{T}\) is computed as the distance between the descriptors of the segmentation boundaries between two successive frames, in terms of shape context descriptors [155]. Per-frame pixel error rate \(\epsilon\) is computed as:  
\[\epsilon=\frac{\text{XOR}(th(P),G)}{N} \tag{16}\]  
where \(th(P)\) is a binary (thresholded) version of the predicted saliency map, \(G\) is the reference ground truth, and \(N\) is the total number of frames in the input sequence.  
Other general-purpose measures often used to evaluate saliency estimation in the video domain include \(F_{\beta}\) ([126; 127; 128; 130; 131; 49]), the Precision/Recall curve ([126; 127; 128; 130; 131]), and MAE ([126; 127; 130; 131]).  
The landscape defined by the dataset/method matrix for video saliency estimation in Table 10 is particularly scattered. We report in Table 12 quantitative results for the frequently adopted SegTrack v2 dataset, and for the DAVIS dataset. These two datasets are comparable in terms of video length and type of annotations, with DAVIS being composed of about three times as many sequences, at a higher resolution.  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Method** & \multicolumn{2}{c}{**SegTrack v2**} & \multicolumn{2}{c}{**DAVIS**} \\ \hline  & \(\mathbf{max}F_{\beta^{\dagger}}\) & \(\mathbf{MAE}_{\text{i}}\) & \(\mathbf{max}F_{\beta^{\dagger}}\) & \(\mathbf{MAE}_{\text{i}}\) \\ \hline [PERSON] 2019 [49] & 0.801 & **0.0230** & - & - \\ [PERSON] 2019 [127] & - & - & **0.902** & **0.0220** \\ [PERSON] 2019 [128] & - & - & 0.859 & - \\ [PERSON] 2019-III [129] & - & - & 0.765 & 0.0588 \\ [PERSON] 2018 [130] & **0.899 * & 0.0807 * & 0.747 * & 0.0636 * \\ [PERSON] 2017-II [126] & 0.890 * & 0.0489 * & - & - \\ [PERSON] 2017 [131] & 0.810 * & - & - & - \\ \hline \hline \end{tabular}  
* Values inferred from graphs in the corresponding publication.  
\end{table}
Table 12: Quantitative comparison of selected methods for video saliency estimation on the SegTrack v2 and DAVIS datasets. Best results are highlighted in boldface.  
We did not include an analysis of FBMS due to the wider variability of versions (subsets of video sequences) used by different methods. Drawing any conclusions in the reported scenario is particularly challenging: on the SegTrack v2 dataset, the hand-crafted method by [PERSON] et al. [126] appears to be the most well-balanced solution according to \(F_{\beta}\) and MAE, while on DAVIS the best results are obtained by [PERSON] et al. [127], which is a deep-learning model. At the same time, the best-\(F_{\beta}\) method on SegTrack, developed by [PERSON] et al. [130], reports worse performance on other metrics and datasets. [PERSON] et al. [49], which is based on the recently-introduced concept of saliency shift, reaches the best result in terms of MAE, at the cost of penalizing \(F_{\beta}\)-based evaluation. It is therefore ultimately not clear whether one type of solution should be preferred against another, for saliency estimation in video sequences.