# Personalized Video Summarization: A Comprehensive Survey of Methods and Datasets
## 7 Evaluation
### Evaluation Measures and Protocols
#### 7.4.1 Evaluating Personalized Video Storyboards  
Early personalized video summarization techniques using representative frames created a static personalized video summary. To evaluate the quality of the summaries, the first attempts used more human judges, which are subjective criteria. Judgment with human judges was based on criteria such as the percentage of interesting events in the videos produced [22], the provision of questionnaires to users to evaluate feedback [16; 46], or the relevance to the video content of each keyframe [16].  
Evaluations relying on subjective criteria have certain drawbacks, including the limitation that the outcomes of the evaluation may not be applicable to future comparisons and the fact that subjective judgments by human evaluators typically consume more time compared to objective evaluations. Consequently, recent research endeavors have shifted towards utilizing ground truth summaries and objective metrics for evaluating summaries, aiming to circumvent the limitations associated with subjective evaluation. Recent studies evaluate the effectiveness of their keyframe-based summaries using objective metrics and comparing them with ground truth summaries [79].  
In this context, ref. [23] estimates the performance of the proposed methodology using the RMSE, MAE, AAR, and Kappa statistics. Instead of the _RMSE_, _MAE_, _AAR_, and Kappa statistics, in [23], the evaluation is based on the known comparison of the user summary error rate (_CUSE_) and a comparison of the user summary accuracy (_CUSA_), and recall, precision, and the F-score. In addition to recall, precision, and F-score in [33], accuracy was used to evaluate the quality of the summary produced. Recall, precision, and F-score were also used in [59] to evaluate the Pipe-1 processing pipeline by investigating the performance of the units used to select the minimum number of keyframes by which the maximum number of objects was represented. In [59], the Pipe-2 processing pipeline was evaluated by measuring the ability to recognize frames included in the OoI using True Positive and False Positive percentages. Each video was tested with a specific OoI, and calculations were performed accordingly.  
In [58], a different method was proposed that evaluates the summary generated according to keyframes that received high scores, comparing the F-score of their method with the F-score of other approaches in the SumMe, TVSum, and query-focused video summarization datasets (QFVS). A five-fold cross-validation with all splits having an average F-score was used in [56] to quantitatively evaluate the quality of the summaries produced. The evaluation of the method's ability to produce videos in various manners is conducted using keyframes by calculating the harmonic mean F-score in [54].  
In [57], an evaluation was conducted using MAP to measure the degree of similarity of ground truth video summaries with predicted video summaries. In [57], also the mADmetric was proposed to measure the average normalized time distance between the closest portion of the predicted summary and the summary reporting portions of the ground truth. In [32], two new metrics were defined to evaluate the emotional video summary. In the first measurement, differences in the F-score were normalized with the baseline relative to videos whose number of impressions of the face was higher. The percentage of recall of frames displayed on faces in the exported summary was defined as the second metric.