# Personalized Video Summarization: A Comprehensive Survey of Methods and Datasets
## 7 Evaluation
### Evaluation Metrics and Parameters
#### 7.3.1 Objective Metrics  
The evaluation of produced summaries, specifically the evaluation of the effectiveness of a video summarizer, poses a significant hurdle. This stems primarily from the absence of a singular qualitative and quantitative measure for assessment. The frequently used quantitative (objective) evaluation metrics are outlined below. Using different sets of quantitative evaluation metrics, the summaries produced by different methods are compared with the ground truth. A set of evaluation metrics proposed in [87] is the user summary error rate comparison (\(CUS_{E}\)) and the user summary accuracy comparison (\(CUS_{A}\)), defined as follows:  
\[CUS_{A}=\frac{n_{mAS}}{n_{US}} \tag{1}\]  
\[CUS_{E}=\frac{n_{m^{\prime}AS}}{n_{US}} \tag{2}\]  
The fundamental explanations of the variables utilized in the equations above are provided below:  
* \(n_{mAS}\): Denotes the number of matching keyframes in the automated summary (AS). * \(n_{m^{\prime}AS}\): Denotes the number of non-matching keyframes in the automated summary. * \(n_{US}\): Denotes the number of keyframes in the user manual summary (US). The values of \(CUS_{A}\) vary between 0 (representing the worst scenario, where none of the AS keyframes align with the keyframes from the US, or vice versa) and 1 (indicating the best scenario, where all keyframes from the US align with those from AS). It is essential to understand that a \(CUS_{A}\) value of 1 does not necessarily imply that all AS and US keyframes are matched. For instance, if \(n_{US}<n_{AS}\) (where \(n_{AS}\) is the number of AS keyframes) and \(CUS_{A}=1\), it indicates that some AS keyframes did not find a match. Concerning \(CUS_{E}\), the  
\begin{table}
\begin{tabular}{l c  
values range from 0 (the optimal scenario, where all AS keyframes match US keyframes) to \(n_{AS}/n_{US}\) (the least favorable scenario, where none of the AS keyframes align with US keyframes, or vice versa). This means that the metrics \(CUS_{A}\) and \(CUS_{E}\) are complementary, with the highest summary quality achieved when \(CUS_{A}=1\) and \(CUS_{E}=0\), indicating a precise match between all AS and US keyframes [87]. A set of frequently used evaluation metrics for the video summarization problem is recall, precision, and F-score [23]. Recall is defined as the quotient of retrieved and relevant frames divided by all relevant frames. Precision is defined as the quotient of relevant and retrieved frames divided by all retrieved frames. F-score is the most commonly used metric and is defined as the harmonic mean of precision and recall. Popular datasets such as TVSum and SumMe include reference summaries (human-annotated summaries). Evaluation can be performed by comparing some reference summaries from a dataset with the personalized video summary generated. Recall, precision, and F-score are defined as follows:  
\[REC=\frac{TP}{TP+FN} \tag{3}\]  
\[PR=\frac{TP}{TP+FP} \tag{4}\]  
\[F-score=2\cdot\frac{PR\cdot REC}{PR+REC} \tag{5}\]  
The fundamental explanations of the variables utilized in the equations above are provided below:  
* _TP (true positive):_ Denotes the number of keyframes extracted by the summarization method that exist in the ground truth. * _FP (false positive):_ Denotes the number of keyframes extracted by the summarization method that do not exist in the ground truth. * _FN (false negative):_ Denotes the number of keyframes existing in the ground truth that are not extracted by the summarization method. Precision and recall metrics vary between 0 and 1, with 0 indicating the lowest performance and 1 indicating the highest. These two metrics are interdependent and cannot be considered separately. An optimal automated video summary is characterized by a high F-score, indicating elevated values for both precision and recall [23]. Another set of evaluation metrics proposed in [59] is the True Positive (PTP) and False Positive (PFP) percentages. The percentages of True Positive and False Positive were determined in comparison to the generated keyframe(s).